Remote I/O egress controls and retryable transport failures by karthikeyann · Pull Request #1052 · rapidsai/kvikio

karthikeyann · 2026-08-30T17:30:31Z

PR summary

This PR adds the following features:

Additional knobs that affect the receive path and egress

These knobs are in the form of environment variables, read per process.

KVIKIO_REMOTE_IO_NONTEMPORAL_COPY: copy received data with AVX2 non-temporal stores, including into the pinned bounce buffers used for device reads. On by default. Falls back to memcpy without AVX2.
KVIKIO_REMOTE_IO_NONTEMPORAL_COPY_THRESHOLD: smallest range request that uses non-temporal copy. Default 1 MiB.
KVIKIO_REMOTE_IO_BUFFER_SIZE: libcurl receive buffer size (CURLOPT_BUFFERSIZE). Unset keeps libcurl's 16 KiB.
KVIKIO_REMOTE_IO_INTERFACE: bind connections to a NIC or source address (CURLOPT_INTERFACE), for hosts with several NICs on one subnet.
KVIKIO_REMOTE_IO_DISCARD_DATA: benchmark-only, drops received host data to measure the network path alone.

Additional retry candidates

CURLE_COULDNT_RESOLVE_HOST, CURLE_COULDNT_CONNECT, CURLE_SEND_ERROR, CURLE_RECV_ERROR, CURLE_PARTIAL_FILE and CURLE_GOT_NOTHING. Previously only timeouts were retried.

Performance result

Microbenchmark: Remote I/O egress controls and retryable transport failures #1052 (comment)
Presto GPU: Remote I/O egress controls and retryable transport failures #1052 (comment)

Original PR description by @karthikeyann

Draft. Two commits, both driven by measurements while reading a 3 TB TPCH dataset from S3 into GPU
memory on a g7e.48xlarge (4 ENA cards, 8 GPUs) through Velox/Presto.

1. `feat(remote): egress controls, streaming-store receive, and a retry-accounting fix`

Unchanged from karthikeyann:s3-max-bandwidth, rebased onto current main. Adds
KVIKIO_REMOTE_IO_INTERFACE (CURLOPT_INTERFACE, if!<name> / SO_BINDTODEVICE),
KVIKIO_REMOTE_IO_DNS_SHUFFLE, KVIKIO_REMOTE_IO_NT_COPY, KVIKIO_REMOTE_IO_BUFFER_SIZE,
KVIKIO_REMOTE_IO_KTLS, KVIKIO_REMOTE_IO_DISCARD, and fixes retry accounting so a partially
completed range transfer rolls the write-callback context back before being retried.

Independent confirmation of the egress control's value, measured end to end through Presto on
SF3000 TPCH: with all four cards on one subnet the kernel picks by route metric, so every
connection left through a single NIC.

	without `KVIKIO_REMOTE_IO_INTERFACE`	with per-NIC binding
per-NIC receive	`[95.8, 0, 0, 0]` Gbps	`[68.8, 45.4, 81.4, 51.0]` Gbps
peak aggregate	95.8 Gbps	579.6 Gbps

One deployment note that cost some time to find: the interface name must exist in the process's
network namespace, so a containerised reader needs host networking. Without it SO_BINDTODEVICE
fails and the failure surfaces from UrlParser::parse as a misleading
url.cpp: Unsupported URL scheme on the original s3:// URL, because
encode_special_chars_in_path() parses without CURLU_NON_SUPPORT_SCHEME and outside any
try/catch. Worth improving separately.

2. `fix(remote): retry transport-level curl failures (resets, DNS, partial body)`

HttpRetryPolicy::is_retryable() currently accepts only CURLE_OPERATION_TIMEDOUT, with a TODO
asking for more candidates. Every other transport failure is fatal, so a single dropped packet fails
an entire multi-TB table scan. Two failure modes hit repeatedly at these rates:

CURLE_COULDNT_RESOLVE_HOST — the remote path resolves per request with no shared DNS cache, so at
512 concurrent requests x 8 processes the query rate exceeds the AWS VPC resolver's 1024 packets/s
per-ENI allowance.
CURLE_RECV_ERROR — S3 recycles connections; observed 6-7 times per 22-query run.

Both were fatal, and in our stack the throwing worker then crashed, so one lost packet cascaded into
unrelated query failures. Measured impact: TPCH runs went from 17/22 and 14/22 queries passing (a
different set failing each time) to 22/22 in a single run.

This now also retries CURLE_COULDNT_CONNECT, CURLE_SEND_ERROR, CURLE_PARTIAL_FILE and
CURLE_GOT_NOTHING, and stops reporting Got HTTP code 0 for failures that never carried an HTTP
status. It is only safe on top of commit 1's retry-accounting fix: perform(on_retry) rolls the
callback context back, so a retried partial transfer cannot corrupt the destination buffer.

Not included

KVIKIO_REMOTE_IO_KTLS is retained from commit 1 but confirmed still inert: setting it left
/proc/net/tls_stat TlsRxSw unchanged, consistent with libcurl installing its own BIO while
OpenSSL requires the BIO under the SSL object to advertise kTLS on a real socket fd.

Open questions for reviewers

Should the retryable-transport-error set be configurable rather than hard-coded?
Should encode_special_chars_in_path() pass CURLU_NON_SUPPORT_SCHEME so endpoint-probe failures
stop masquerading as scheme errors?

PR dependency

Depends on #1082

…ccounting fix Reading S3 over HTTPS at multi-hundred-Gbps exposed several gaps in the remote path. Measured on a g7e.48xlarge, four ENA cards, against s3://rapids-tpch. Retry accounting (a correctness bug). CurlHandle::perform() retries curl_easy_perform() without rolling back what the write callback already accepted, so a transfer that times out part way through resumes with the destination offset still advanced, overruns the caller's buffer and returns CURLE_WRITE_ERROR, which is not retryable. A recoverable timeout therefore became a hard failure, reported as the misleading "maybe the server doesn't support file ranges?". With 512 MiB ranges under four-card load this killed two of four processes on every attempt. CurlHandle now takes an on-retry hook and RemoteHandle::read uses it to reset the callback context. Streaming-store receive, KVIKIO_REMOTE_IO_NT_COPY. libcurl delivers one buffer at a time, 16 KiB by default, so every memcpy in the write callback is far below glibc's non-temporal threshold and uses ordinary stores. An ordinary store to a line not in cache fetches it first, so with a destination far larger than last-level cache each delivered byte costs two DRAM accesses and the receive path becomes memory-bandwidth-bound rather than CPU-bound. Streaming stores skip the fetch: 669 -> 874 Gbps delivered into pinned host memory, +29%, reproduced. Off by default because it is the wrong choice for a small destination that stays in cache, which is exactly the device-memory bounce buffer. CURLOPT_INTERFACE, KVIKIO_REMOTE_IO_INTERFACE. The if!<name> form binds with SO_BINDTODEVICE, the only way to choose an egress NIC from inside the process when a host has several cards on one subnet. Without it the kernel picks by route metric and every connection leaves through one card. CURLOPT_DNS_SHUFFLE_ADDRESSES, KVIKIO_REMOTE_IO_DNS_SHUFFLE. S3 publishes many front-end addresses but libcurl connects to the first that answers, so a process otherwise piles every connection onto one endpoint. CURLOPT_BUFFERSIZE, KVIKIO_REMOTE_IO_BUFFER_SIZE, default unchanged. Raising it looks right and measures worse, because libcurl allocates this buffer per easy handle and there is one handle per in-flight sub-range: at 4096-way concurrency that is 64 MiB at 16 KiB against 4 GiB at 1 MiB, and the smaller total stays in L3. Measured 862 Gbps at 16 KiB, 745 at 256 KiB, 727 at 1 MiB. Exposed because the trade-off inverts at low concurrency. KVIKIO_REMOTE_IO_KTLS sets SSL_OP_ENABLE_KTLS. It does not currently engage through libcurl, which installs its own BIO while OpenSSL requires the BIO under the SSL object to advertise kTLS on a real socket fd; TlsRxSw stays at zero. Kept off by default and left in place for when libcurl gains support. KVIKIO_REMOTE_IO_DISCARD accounts for bytes without copying them, to separate transport cost from destination cost. Benchmarking only: the accounting stays honest but the destination holds garbage. It measured the destination write at 31% of achievable throughput, which is what motivated the streaming-store path. All options are read once into function-local statics, because a handle is constructed per sub-range transfer and getenv would otherwise sit on the hot path.

…l body)

copy-pr-bot · 2026-08-30T17:30:34Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

kingcrimsontianyu · 2026-09-25T19:55:55Z

/ok to test

kingcrimsontianyu · 2026-09-25T20:12:03Z

/ok to test

kingcrimsontianyu · 2026-09-28T14:32:01Z

Microbenchmark result

g7e.12xlarge with a network bandwidth of 400 Gbps (46.6 GiB/s)

s3-remote-io-egress-and-retry @ 00936140, libcurl 8.13, GiB/s.
MULTI_POLL, 48 reactors, SHARED_QUEUE. 64 MiB tasks, 1 GiB reads.
HTTP: 1024 connections.  HTTPS: 512 connections.  2 runs per cell, 30 s each.

+--------+-----------------------------------+----------+---------+---------+
| Into   | Receive option                    | Buffer   |  HTTP   |  HTTPS  |
+--------+-----------------------------------+----------+---------+---------+
| Host   | Discard (no copy: ceiling)        | 256 KiB  |  45.5   |  33.1   |
|        | Normal copy                       |  16 KiB  |  25.9   |  23.2   |
|        | Normal copy                       | 256 KiB  |  33.9   |  22.3   |
|        | Non-temporal copy                 |  16 KiB  |  30.9   |  27.7 * |
|        | Non-temporal copy                 | 256 KiB  |  45.4 * |  26.4   |
+--------+-----------------------------------+----------+---------+---------+
| Device | Normal copy                       |  16 KiB  |  23.7   |  22.3   |
|        | Normal copy                       | 256 KiB  |  28.4   |  23.0   |
|        | Non-temporal copy                 |  16 KiB  |  27.3   |  25.9   |
|        | Non-temporal copy                 | 256 KiB  |  39.7 * |  26.1 * |
+--------+-----------------------------------+----------+---------+---------+
* best for that destination and protocol

kingcrimsontianyu · 2026-09-30T16:52:23Z

Presto GPU TPC-H SF-1K

Q9 time excluded. Currently it fails due to a HashCSR error, and a fix has been provided for cuDF + velox, which will merge into main soon.

+---------------------+-----------+----------------------------+----------------------------+
| Receive option      | Buffer    |            HTTP            |           HTTPS            |
|                     |           |  it 1 / it 2  (it 1 range) |  it 1 / it 2  (it 1 range) |
+---------------------+-----------+----------------------------+----------------------------+
| Normal copy         |  16 KiB   |  104.8 / 91.7  (104-105)   |  111.1 / 97.5  (109-112)   |
| Normal copy         | 256 KiB   |  103.4 / 89.5  (102-104)   |  111.5 / 97.6  (110-115)   |
| Non-temporal copy   |  16 KiB   |  103.3 / 89.1  (102-104)   |  106.1 / 91.9  (105-108)   |
| Non-temporal copy   | 256 KiB   |  101.7 / 87.7  (101-102)   |  106.5 / 91.9  (106-107)   |
+---------------------+-----------+----------------------------+----------------------------+

With HTTPS, non-temporal copy is about 5% faster than normal copy.
With HTTP, non-temporal copy is about 1.5% faster than normal copy.
A 256 KiB buffer is 0 to 1.5% faster than 16 KiB.
HTTP is faster than HTTPS. The gap is about 6% with normal copy, and 3 to 4% with non-temporal copy.

kingcrimsontianyu · 2026-09-30T20:01:19Z

It seems the test test_http_io_under_shared_queue introduced in #1062 is flaky. This time it failed on:

conda-python-tests / 12.2.2, 3.11, amd64, rockylinux8, h100, earliest-driver, oldest-deps

and last time it failed on:

wheel-python-tests / 12.2.2, 3.11, arm64, ubuntu22.04, a100, latest-driver, latest-deps

kingcrimsontianyu · 2026-10-01T04:17:31Z

The CI fix from #1082 works.

madsbk

Nice work @kingcrimsontianyu

kingcrimsontianyu · 2026-10-06T14:25:25Z

/merge

karthikeyann added 2 commits August 29, 2026 22:24

fix(remote): retry transport-level curl failures (resets, DNS, partia…

79851f5

…l body)

madsbk assigned karthikeyann Sep 1, 2026

madsbk added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Sep 1, 2026

madsbk reviewed Sep 1, 2026

View reviewed changes

Comment thread cpp/src/detail/remote_callback.cpp

Merge branch 'main' into s3-remote-io-egress-and-retry

7e88bc3

This was referenced Sep 15, 2026

Improve remote I/O performance and expose several additional knobs #1059

Closed

KvikIO remote backend performance improvement #1065

Open

kingcrimsontianyu mentioned this pull request Sep 25, 2026

[Tracking] Upstream findings from the Presto GPU cloud spike #1079

Open

8 of 27 tasks

kingcrimsontianyu added 2 commits September 25, 2026 14:35

Merge branch 'main' into s3-remote-io-egress-and-retry

53fbef2

Bug fixes. Refine comments. Add unit test

de5ed56

Fix a unit test error

0093614

wence- requested changes Sep 28, 2026

View reviewed changes

Comment thread cpp/src/detail/remote_callback.cpp

Comment thread cpp/src/shim/libcurl.cpp Outdated

Comment thread docs/source/runtime_settings.rst

kingcrimsontianyu added 2 commits September 28, 2026 13:43

Add a threshold knob for non-temporal copy

2ab2fe1

Fix CURL_MAX_READ_SIZE

d8f9d9a

kingcrimsontianyu requested review from madsbk and wence- September 28, 2026 14:36

kingcrimsontianyu added 2 commits September 29, 2026 14:53

Merge branch 'main' into s3-remote-io-egress-and-retry

1919de4

Merge branch 'main' into s3-remote-io-egress-and-retry

25c3124

kingcrimsontianyu marked this pull request as ready for review September 30, 2026 17:50

kingcrimsontianyu requested review from a team as code owners September 30, 2026 17:50

kingcrimsontianyu requested a review from vuule September 30, 2026 17:51

kingcrimsontianyu added 2 commits October 1, 2026 03:19

Attempt to fix CI error

7bebd6e

Merge branch 'fix-http-hang' into s3-remote-io-egress-and-retry

680f7bc

kingcrimsontianyu added 2 commits October 1, 2026 12:51

Merge branch 'main' into s3-remote-io-egress-and-retry

16e04b6

Further doc clarification

542ad67

madsbk reviewed Oct 5, 2026

View reviewed changes

Comment thread docs/source/runtime_settings.rst

Comment thread cpp/src/shim/libcurl.cpp Outdated

Comment thread docs/source/runtime_settings.rst Outdated

Comment thread cpp/src/detail/remote_callback.cpp Outdated

Comment thread cpp/tests/test_remote_callback.cpp

kingcrimsontianyu added 4 commits October 5, 2026 12:39

Update comments

52ac36e

Update comment

2232d6e

Update

78bc594

Add unit test for partial file retry

759827d

kingcrimsontianyu requested a review from madsbk October 5, 2026 23:11

madsbk approved these changes Oct 6, 2026

View reviewed changes

wence- approved these changes Oct 6, 2026

View reviewed changes

rapids-bot Bot merged commit d3c6a69 into rapidsai:main Oct 6, 2026
58 checks passed

kingcrimsontianyu mentioned this pull request Oct 16, 2026

feat(cudf): Serve KvikIO reads through AsyncDataCache facebookincubator/velox#18941

Open

Conversation

karthikeyann commented Aug 30, 2026 • edited by kingcrimsontianyu Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR summary

Additional knobs that affect the receive path and egress

Additional retry candidates

Performance result

Original PR description by @karthikeyann

1. feat(remote): egress controls, streaming-store receive, and a retry-accounting fix

2. fix(remote): retry transport-level curl failures (resets, DNS, partial body)

Not included

Open questions for reviewers

PR dependency

Uh oh!

copy-pr-bot Bot commented Aug 30, 2026

Uh oh!

Uh oh!

kingcrimsontianyu commented Sep 25, 2026

Uh oh!

kingcrimsontianyu commented Sep 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kingcrimsontianyu commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Microbenchmark result

Uh oh!

kingcrimsontianyu commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Presto GPU TPC-H SF-1K

Uh oh!

kingcrimsontianyu commented Sep 30, 2026

Uh oh!

kingcrimsontianyu commented Oct 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

madsbk left a comment

Choose a reason for hiding this comment

Uh oh!

kingcrimsontianyu commented Oct 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

karthikeyann commented Aug 30, 2026 •

edited by kingcrimsontianyu

Loading

1. `feat(remote): egress controls, streaming-store receive, and a retry-accounting fix`

2. `fix(remote): retry transport-level curl failures (resets, DNS, partial body)`

kingcrimsontianyu commented Sep 28, 2026 •

edited

Loading

kingcrimsontianyu commented Sep 30, 2026 •

edited

Loading