Repository navigation
bench 313 live again: the second tab was never a publish bug, it was a cold counter - #3525
Merged
Merged
Conversation
…a cold counter I took this bench to draft last night because its same-block tab published a zero on every row, and zero is the best score here, so five named companies carried an unearned perfect record. I read the worker's "bench skip: missing p50,p90,p99" against queries that answered when asked directly, and concluded the skip was in the publish path. It was not. increase() over a window reads 0 on a counter that has not incremented twice inside it, and the counters were hours old. The delayed tab crossed that line first and looked correct; the same-block tab had not and looked broken. One cause, two appearances, and I diagnosed the appearance. Fifteen hours later both tabs resolve through the real loader, every row carries about 106 token-questions against the 120 this bench calls a day of them, and the ordering is stable rather than a single sweep's accident: trnch 41.5 lifi 44.3 paraswap 61.0 sushi 92.5 kyberswap 95.8 Trnch first and LI.FI second is the result the method predicted and the only one worth publishing: both screen the token and decline, one with its own agent before polling any router, one with a deny list. A quote board cannot see that difference. This one is built to. Nothing changes but the status and the keys. The queries, the gating and the counters are as they were merged yesterday; what changed is that there is now enough measurement to stand behind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I took this bench to draft last night (#3522) because its same-block tab published a
0on every row, and 0 is the best score here, so five named companies carried an unearned perfect record. I read the worker'sbench skip: missing p50,p90,p99against queries that answered when asked directly, and concluded the skip was in the publish path.It was not.
increase()over a window reads0on a counter that has not incremented twice inside it, and the counters were hours old. The delayed tab crossed that line first and looked correct; the same-block tab had not and looked broken. One cause, two appearances, and I diagnosed the appearance.Fifteen hours later both tabs resolve through the real loader, with every row carrying ~106 token-questions against the
120this bench calls a day of them (healthy, not the 7 of last night):Trnch first and LI.FI second is the result the method predicted and the only one worth publishing: both screen the token and decline — one with its own agent before polling any router, one with a deny list. A quote board cannot see that difference; this one is built to.
The two tiers agreeing within 1.5 points is also the expected shape, and it says the anti-bot cooldown question the selector exists for is not what separates these routers.
Nothing changes but the status and the keys. The queries, the sample gating and the counters are as merged yesterday in #3517 / #3519; what changed is that there is now enough measurement to stand behind.
Keys
bench-unfiltered-v165,all-benchmarks-v158. Provider set unchanged.Gates:
pnpm validate261 specs,tsc --noEmitclean, 910 tests / 0 fail.🤖 Generated with Claude Code