Skip to content

perf(endpoint): keep the deadlines in a heap for the timer path - #15

Open
pylaterreur wants to merge 2 commits into
mp0rta:mainfrom
pylaterreur:timer-heap
Open

pylaterreur wants to merge 2 commits into
mp0rta:mainfrom
pylaterreur:timer-heap

Conversation

@pylaterreur

Copy link
Copy Markdown

Part of #12.

poll_timeout computed every connection's deadline on each call, and handle_timeout ran every connection's timer pass, so each event-loop wakeup cost O(N).

Each connection now caches its deadline: the per-connection minimum that poll_timeout computed, unchanged. A min-heap of (deadline, handle) with lazy deletion orders the cached deadlines.

  • Every endpoint call that reaches a connection marks it: a packet for it, a path call, connection(h), a turn of poll_transmit, a handle_timeout pass.
  • A marked connection's heap entry is dropped when it reaches the top. Its deadline is cached again once poll_transmit has nothing left to send or handle_timeout runs. Until then poll_timeout (still &self) computes it afresh, so it returns exactly what the full scan returned.
  • handle_timeout processes, in handle order, the connections whose deadline passed and those an endpoint call reached since their last pass (which every connection used to get). It frees only among them.

poll_transmit still visits every connection; the ready-set PR handles that.

The full scan stays as a test-only oracle (Endpoint::full_scan), where poll_timeout and handle_timeout look at every connection. diff_tests, the same as in the ready-set PR, checks the same packets in the same order, events, poll_timeout values, reads and call results.

Connections processed per handle_timeout call during a bulk transfer next to N idle MPTCP connections (scale_tests): main processes N+1 (2, 11, 101, 501, 2001 for N = 1, 10, 100, 500, 2000); this PR processes 1.0 for every N.

At N=1000, callgrind instructions for a whole run (setup plus a 1 MB transfer):

main this PR
poll_timeout 141.5M 0.13M
handle_timeout 40.3M 2.2M
handle_packet 177.7M 73.3M

poll_transmit's full walk still dominates, so the whole run is only 5% shorter at N=1000. At N=0 it is 0.7% longer. With both PRs the whole run is 96% shorter at N=1000, and within 0.1% of main at N=0.

Testing:

  • cargo test, clippy, fmt, MSRV 1.85, and the fuzz targets.
  • diff_tests on 40 small and 400 large scenarios.
  • The two-crate harness against main on 3000 more scenarios.
  • An instrumented build that runs main's full scan alongside and reports any difference in poll_timeout, any skipped timer pass that would have changed a connection, and any connection freed outside the processed set. No report over the whole test suite and 150 random scenarios.
  • tests/interop on a Linux 7.0 VM: m1 and the 23 server-role runs pass.

The two PRs are independent; whichever lands second needs a rebase. I have that rebase ready: on top of the ready set, a handle_timeout pass also makes its connection ready. With both PRs, the full interop suite passes too.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GwUvhmzgfkiZ6vo4uGyesN

poll_timeout computed every connection's deadline at each call and
handle_timeout ran every connection's timer pass, so an event loop paid
O(N) per wakeup whatever was due (issue mp0rta#12).

Each connection now caches its deadline (what poll_timeout computed for
it, unchanged), and a min-heap of (deadline, handle) with lazy deletion
orders the cached ones. Every endpoint call that reaches a connection
marks it: a packet for it, a path call, connection(h) (which hands out
&mut Connection), a turn of poll_transmit and a handle_timeout pass. A
marked connection's entry leaves the heap once it reaches the top, and
its deadline is cached again once poll_transmit has nothing left to send
or handle_timeout runs; until then poll_timeout computes it afresh, so
it returns what the full scan returned. handle_timeout processes, in
handle order, the connections whose deadline passed and those an
endpoint call reached since their last pass (which every connection
used to get), and frees only among them. apply_aborts looks only at the
connections the application reached.

poll_transmit still visits every connection. A turn that sends nothing
and leaves nothing for the next turn leaves the connection idle until
its wake time (its deadline or a pacing release): the turns until then
change nothing, so they need no new deadline.

The full scan stays as a test-only oracle (Endpoint::full_scan), and
diff_tests runs randomized two-endpoint scenarios both ways and checks
that the packets, in order, the events, poll_timeout values, reads and
call results are the same. With 2000 idle MPTCP connections
handle_timeout processes one connection per call during a bulk transfer
instead of 2001 (scale_tests). The rogue-plan tests change the probe
scheduler's plan from outside the connection, so they mark it
themselves (set_rogue).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwUvhmzgfkiZ6vo4uGyesN

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants