Skip to content

bench 313 live again: the second tab was never a publish bug, it was a cold counter - #3525

Merged
Flotapponnier merged 1 commit into
devfrom
feat/sellability-live-healthy
Nov 5, 2026
Merged

Flotapponnier merged 1 commit into
devfrom
feat/sellability-live-healthy

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

I took this bench to draft last night (#3522) because its same-block tab published a 0 on every row, and 0 is the best score here, so five named companies carried an unearned perfect record. I read the worker's bench skip: missing p50,p90,p99 against queries that answered when asked directly, and concluded the skip was in the publish path.

It was not. increase() over a window reads 0 on a counter that has not incremented twice inside it, and the counters were hours old. The delayed tab crossed that line first and looked correct; the same-block tab had not and looked broken. One cause, two appearances, and I diagnosed the appearance.

Fifteen hours later both tabs resolve through the real loader, with every row carrying ~106 token-questions against the 120 this bench calls a day of them (healthy, not the 7 of last night):

[delayed]    trnch 41.5  lifi 44.3  paraswap 61.0  sushi 92.5  kyberswap 95.8   n≈106
[same_block] trnch 41.9  lifi 44.2  paraswap 61.6  sushi 93.3  kyberswap 95.2   n≈105

Trnch first and LI.FI second is the result the method predicted and the only one worth publishing: both screen the token and decline — one with its own agent before polling any router, one with a deny list. A quote board cannot see that difference; this one is built to.

The two tiers agreeing within 1.5 points is also the expected shape, and it says the anti-bot cooldown question the selector exists for is not what separates these routers.

Nothing changes but the status and the keys. The queries, the sample gating and the counters are as merged yesterday in #3517 / #3519; what changed is that there is now enough measurement to stand behind.

Keys bench-unfiltered-v165, all-benchmarks-v158. Provider set unchanged.

Gates: pnpm validate 261 specs, tsc --noEmit clean, 910 tests / 0 fail.

🤖 Generated with Claude Code

…a cold counter

I took this bench to draft last night because its same-block tab published
a zero on every row, and zero is the best score here, so five named
companies carried an unearned perfect record. I read the worker's
"bench skip: missing p50,p90,p99" against queries that answered when asked
directly, and concluded the skip was in the publish path.

It was not. increase() over a window reads 0 on a counter that has not
incremented twice inside it, and the counters were hours old. The delayed
tab crossed that line first and looked correct; the same-block tab had not
and looked broken. One cause, two appearances, and I diagnosed the
appearance.

Fifteen hours later both tabs resolve through the real loader, every row
carries about 106 token-questions against the 120 this bench calls a day of
them, and the ordering is stable rather than a single sweep's accident:

  trnch 41.5   lifi 44.3   paraswap 61.0   sushi 92.5   kyberswap 95.8

Trnch first and LI.FI second is the result the method predicted and the
only one worth publishing: both screen the token and decline, one with its
own agent before polling any router, one with a deny list. A quote board
cannot see that difference. This one is built to.

Nothing changes but the status and the keys. The queries, the gating and
the counters are as they were merged yesterday; what changed is that there
is now enough measurement to stand behind.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Flotapponnier
Flotapponnier merged commit f7ad91d into dev Nov 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant