Repository navigation
Conversation
With checksums, a per-subflow hold buffer keeps each mapping's octets by SSN until the whole mapping has arrived and its DSS checksum verifies; only then do they reach RecvAssembler, which clips what another subflow already delivered (spec G6, G7). Held octets are TCP-ACKed but not DATA_ACKed. A folded DATA_FIN applies only from a verified mapping, and a DATA_FIN-only mapping only if its checksum verifies (G8). Completion credit lets a mapping that starts below the base edge raise the offered edge to its end (at most base + 65,535), and free() counts held octets. Each subflow holds at most recv_buffer + 65,535 octets plus the frontier mapping's; past that, other mappings' octets are refused. Interim: a failed checksum resets the subflow with MP_TCPRST 0x06 (Tasks 5-6 add MP_FAIL). Without checksums nothing changes (G9).
A peer could fragment the hold buffer without limit: every disjoint arrival became its own heap run, each insert re-sorted and rebuilt the list, and descending arrival copied quadratically. Held runs per subflow are now capped at 64 (as MAX_OOO_RANGES and range_cap): past it, a run that would start a new disjoint run is refused at the TCP level unless it starts at the frontier. A run merges in place into the largest touched run (VecDeque), copying only the octets outside it. The prune of a mapping wholly below the frontier is named as the only exception to "held octets are never dropped", with a test.
RST options widen to a Vec of at most 16 octets so a reset can carry MP_FAIL + MP_TCPRST. A failed checksum resets its subflow with MP_FAIL(M.dsn) + MP_TCPRST 0x06 and discards the subflow's held octets. ConnectionHooks gains infinite_ok (cleared when a join completes or a connection-level retransmission is sent); an MP_FAIL without RST while !infinite_ok resets the subflow with the received DSN. MP_FAIL after fallback, on a joining subflow, or malformed is ignored. Adds initial_offset_ok for the single-subflow path (M4b spec G10, G11, G13, G17, G18, G20).
A checksum failure on the one data-carrying subflow, with infinite_ok and the handshake's DSN-SSN offset in the failed and every held mapping, enters Waiting instead of resetting: MP_FAIL(M.dsn) rides every ACK on the subflow (after the DSS, before SACK blocks), a pure ACK with it is forced on a doubling schedule from the RTO, and the failed mapping's octets stay held (Hold keeps failed mappings until drained). While Waiting, joins are neither started nor answered, pending ones are reset, and reinjection, opportunistic retransmission and the recovery timer are suspended. The peer's MP_FAIL or an infinite mapping ends Waiting with fall_back + on_fallback, the held octets replayed. An MP_FAIL while Idle and infinite_ok makes us the echoer: echo MP_FAIL(recv.next), fall back, and let the echo ride every segment while our infinite mapping is due, with forced retries until it retires or 120 s. After fallback an MP_FAIL triggers nothing. 120 s in Waiting resets the subflow with MP_TCPRST 0x06. An MP_FAIL on the ACK that completes a join is ignored (M4b spec G10-G18, G21).
Waiting outlived a subflow the peer reset (RST + MP_FAIL, G17) or that timed out: for up to 120 s joins stayed frozen, so the no-subflow wait's rescue never ran, and the failed mapping stayed held. Every recovery evaluation and every timeout now drops an orphaned Waiting to Idle and discards that subflow's held octets (refreshing the held count); queued joins then start. The 120 s expiry drains the hold buffer whatever the TCB state (M4b spec G12, G16, G17).
ConnectConfig::default() now requires DSS checksums in both roles. Tests
that script or read the A=0 wire layout set checksums = false in their
fixtures; no regression was found among the failures. Adds
default_requires_checksums_{client,server}. README and lib.rs docs updated.
Two real endpoints, so nothing forces A=0 (spec A4). mptcp_net takes `checksums`; expected MP_CAPABLE-with-data length is 24 with it, 22 without, and the flags gain A. No library change was needed: all ten scenarios pass at A=1, including the length-24 lost-third-ACK paths.
Linux folds a DATA_FIN queued after part of the final mapping was sent into that mapping's later segments (mptcp_write_data_fin): same DSN, SSN and payload length, Data-Level Length + 1, DATA_FIN set and a checksum over the new length. Linux's receiver takes that as the identical mapping (get_mapping_status). With checksums, Hold::install required an equal HoldMap and reset the subflow, which in the csum-fail2 c2s interop run (Linux client, mptcp-rs server) killed the only subflow left. Treat the repeat as the same mapping and keep its DATA_FIN with the checksum covering it; a repeat without the DATA_FIN changes nothing.
mptcp-rs-tun gets --checksums on|off (default on, as the library) and --invert N; server.py and kclient.py get the same --invert N (shared in pattern.py). Every verifier prints "inverted at OFF". run.sh: INTEROP_CSUM=<ours>:<linux> sets the tun flag and Linux's net.mptcp.checksum_enabled; scenarios csum-fail1 (one subflow) and csum-fail2 (two, corruption on leg B), c2s and s2c, for both role pairings, with the Linux selftest's corruption (iptables nth mark, tc pedit invert + csum tcp, one packet per skb) and its pedit packet count required to be 1. Linux nstat counters are recorded for every run: DataCsumErr must be 0 outside csum-fail*, which pin the exact MP_FAIL / infinite mapping / MP_TCPRST counters from the first runs. check.py: csum-fail2 counts as two-subflow; MP_FAIL DSNs are read per option. csum-fail1 needs MP_FAIL each way, an infinite mapping, no RST, our FellBack, and exactly one inverted octet inside the mapping the receiver's MP_FAIL names. csum-fail2 needs exactly one reset with MP_FAIL + MP_TCPRST 6 on leg B (exempt from the FIN rule), no infinite mapping, no inverted octet; the only other RSTs allowed are the resetting end's bare answers to segments still in flight on that subflow, which both Linux and mptcp-rs send.
Collect them before the client/server exit-code gate, as a diagnosis aid.
A peer could send each in-order octet unmapped with a DSS mapping of the octet before it: the mapping took the unmapped path, so Hold never pruned it and the entries grew without bound (spec G7). Prune on every receipt advance, as RxMaps does.
While Waiting, a peer could send further failing mappings of the same in-window DSNs at each new frontier: each was exempt from the hold cap and kept as failed, so held octets grew past recv_buffer + 2 x 65,535 until the 120 s timeout (spec G7, G11). With a failed mapping held, the cap now binds every mapping; octets past it are refused at the TCP level and come back as plain TCP after the fallback.
The MP_FAIL block fell back and returned before the DSS was looked at: with checksums, MP_FAIL plus an infinite mapping without its checksum fell back instead of resetting with MP_TCPRST 0x01 (RFC 8684 L1331-1342, spec G3), and a valid infinite mapping there lost its own anchor to the handshake offset. The presence check now runs first, and the MP_FAIL fallback takes the infinite mapping's anchor.
Every input summed every subflow's hold buffer, also with checksums off on both ends, where nothing is ever held: a per-packet cost spec A1 forbids. The other recomputations run only while MP_FAIL waits, which needs checksums.
Pure ACKs never advance the receipt, and RxMaps takes a mapping inside one it keeps without a slot, so a peer could re-declare one-octet parts of a verified mapping and pile up hold entries (spec G7). A mapping wholly below the last pruned frontier is no longer kept, and past 1024 entries, as for RxMaps, a new one is invalid and resets the subflow.
A verified mapping went to RecvAssembler::accept and the result was ignored: past the range cap its octets, TCP-ACKed already, were dropped, and a later fallback left a hole (spec G7, G15). take_complete now hands each verified mapping to a deliver callback and keeps it held, verified, until the callback takes it; the fallback replays it like any held run.
… check The G10 offset check looked at the failing mapping and the held ones, after take_complete had already removed the mappings that verified in the same input: a valid mapping with another DSN-SSN offset completed alongside a corrupt one let us wait for MP_FAIL although contiguity was broken. Every mapping the input completed now counts (spec G10, G11).
A verified mapping the assembler's range cap refused was offered again only on its own subflow's next payload; once another subflow cleared the range pressure, a sender with nothing more to send there left it stranded, though it was TCP-ACKed (spec G7, G15). Hold counts such mappings, and after every input and on every read, with checksums, the connection offers them again on all subflows until none is taken.
The G10 check looked only at the failing mapping and those held or completed in the same input: a mapping with another DSN-SSN offset that had verified earlier escaped it, and we waited for MP_FAIL with contiguity broken. Every data mapping installed on a subflow now sets other_offset for good when its offset is not the handshake's, and Waiting requires it clear (spec G10, G11). It subsumes the per-input check, which is removed.
… held A verified mapping the assembler refused stays held, so valid mappings of one in-window DSN range at successive frontiers each took the frontier exemption and the subflow held past recv_buffer + 2 x 65,535 (spec G7). As for a failed mapping, none is exempt while one is held; held octets are kept and further ones are refused at the TCP level.
With a refused verified mapping held, no frontier mapping was exempt from the hold cap: once refused mappings filled it, a frontier mapping that would complete at or below recv.next (cross-subflow duplication) could never complete, and the subflow stalled (spec G7). Such a mapping stays exempt: its completion is taken or a duplicate, never refused by the range cap, so it leaves the hold and the bound holds. Hold::insert takes recv.next for it.
With checksums off on both ends, every data mapping still computed other_offset and every receipt pruned the hold buffer, which A1 does not allow (spec G9). Both are now gated on the negotiated csum.
The frontier exemption from the hold cap had special cases (no refused mapping held, or a start at recv.next); with a redundant sender over three subflows, a frontier mapping that would merge assembler ranges was refused at full holds and the connection hung (spec G7). One rule now: the frontier mapping is exempt iff no failed mapping is held and RecvAssembler::would_take says its completion would be taken now (it starts at or before recv.next, touches a stored range, or a range is free). A mapping it would refuse gains nothing from being held.
With the hold at its cap and a frontier mapping the assembler would refuse, the frontier octet could never be taken: the subflow hung (spec G7). Such a refusal now resets the subflow with MP_TCPRST 0x02 "lack of resources" and discards its hold; the octets were never DATA_ACKed, so the peer sends them again on another subflow (RFC 8684 L2392-2394). Only checksums hold anything, and a failed mapping held (MP_FAIL waiting) keeps it from firing.
An exemption granted on a free range slot can be overtaken by another subflow, leaving that mapping held and refused while the next frontier mapping is exempt: two exempt mappings past the cap, not one.
A subflow that ended (reset, timeout, or closed) kept its held octets: they could never complete there, and the initial subflow is retained, so free() stayed short and the window could stay at zero for good (spec G7). The hold is now drained when the subflow's TCB is closed, and the connection's held count refreshed; the octets were not DATA_ACKed, so the peer sends them again elsewhere (RFC 8684 L2392-2394).
A mapping at the frontier with a DSN near 2^63 past recv.next made would_take add its length to an offset at i64::MAX, which panics in overflow-checked builds. The end saturates, as accept_inner's does.
Rounds 2-7 all came from verified octets, TCP-ACKed already, that the assembler's range cap refused, and the machinery kept to hold and retry them. Design v2 removes that state (spec G7): - a mapping's octets enter the hold only if RecvAssembler::would_take says its completion would be taken (once admitted it stays so), else they are refused at the TCP level and the peer retransmits them; - a verified mapping always enters the assembler, past the range cap if another subflow took the free range since (as replay stores; at most ENTRY_CAP x SUBFLOW_CAP extra ranges); - retry_refused, the refused count and the 0x02 full-hold reset go: an admitted frontier mapping is always exempt from the byte cap while no failed mapping is held, so the reset could no longer fire; - an ended subflow (TIME-WAIT or closed) drops its hold, which now only has partial or failed mappings, and the held count is refreshed. The hold bound is recv_buffer + 2 x 65,535 per subflow again.
Design v2 refused a mapping at admission when the assembler's range cap would refuse its completion, which can block the very gap that would release it, where the checksums-off path goes on. Design v3 matches that path (spec G7): - no admission gate: the hold byte cap and the frontier exemption only; RecvAssembler::would_take goes; - a verified mapping goes through the same accept as without checksums; what the range cap refuses is dropped, TCP-ACKed but not DATA_ACKed, and the peer's reinjection or a redundant copy brings it; - such a drop clears the subflow's MP_FAIL-wait eligibility (other_offset), and while dropped octets lie above recv.next a fallback resets the subflow with MP_TCPRST 0x01 instead of leaving a hole that plain TCP would never fill. Verified mappings always leave the hold: recv_buffer + 2 x 65,535 per subflow, and the assembler keeps to its range cap.
Linux counts every MP_FAIL packet it processes before its fallback, and that depends on when its receive path reaches the infinite mapping. With the review fixes, an otherwise valid run (the wire checker passes) counted 9/9 where earlier runs counted 2/2. Check MPFailTx/MPFailRx >= 1, the spec section 7 invariant; the other pins stay exact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.