Time to Index

This run does not support an ordering of these arms. The table below has 4 arm(s) over at most 29 event(s), and none of the 6 pairwise comparisons reaches significance after adjustment. Read it as what each arm did, not as which arm is faster.

Rendered rather than withheld, because the instrument working on a small run is itself worth showing — and because a page that quietly waits until the numbers look conclusive is a page that only ever publishes flattering ones. Are the differences real below is the panel that answers this properly, and tti power says how many more events each comparison would need.

Pre-registered. Plan bac218770f209b20 (v1, registered 2026-08-31) was locked to this run on its first probe and has not changed since.

Declared in the plan, not enabled in this run: serper/search, tavily/basic. No key was set, so no probe was ever dispatched to these arms. That is a configuration state, not a result — they did not fail, they were never asked.

The CI column is a nonparametric bootstrap on the median bracket's upper edge. It is here because a median printed without one is the most common way a short run gets over-read: at forty events, a bracket can land a rung either side on luck alone, and the ordering of this table is exactly what that luck moves. Where an arm's median is not reached inside the 72-hour window in some resamples, the share is printed rather than hidden — for a slow arm that number, not the point estimate, is the finding.

Two cost columns, because they can disagree and the disagreement is the point. Cost per event is what you pay to ask; cost per fresh answer is what you pay to get one, and a provider that returns nothing does it very cheaply. A freshness win bought at 5× the price is a different product decision than a freshness win at parity, and a leaderboard that hides the denominator is advertising.

Text served is the median number of characters the grader read per returned result. Every arm is asked for the same 1,500; only the APIs that take a characters parameter honour it, and a snippet API returns ~150 regardless. The grader reads whatever came back, because that is what an agent gets — but a tenfold difference in this column is a tenfold difference in the surface a version string can appear on, and part of any recall gap lives there. The snippet-window sensitivity variant below re-grades every arm as though it had returned short snippets.

Median is a Turnbull nonparametric MLE for interval-censored data, reported as the interval the estimate actually pins down. An arm seen absent at 15m and fresh at 1h indexed somewhere in (15m, 1h] — it did not index at 1h, and printing a point would invent precision the ladder cannot supply. Events still un-indexed at 72h are censored, not dropped: discarding them reports every provider as faster than it is, and unevenly, since the bias is largest for whoever has the most un-indexed events. >72h means the arm never accumulated half its mass inside the window, which is a different statement from "slow".

Every stored payload re-graded against a counterfactual answer: a version-shaped token of the same form as the real one that was never published. A FRESH verdict against it is a false positive by construction. This is recall's error bar, which is why it sits here and not in a footnote.

Two causes, both invisible in a leaderboard: the grader matching a version-shaped token in unrelated prose, or a provider's answer layer inventing one. Zero observed is not a zero rate — the last column is the Wilson upper bound, and it is the number that belongs beside each arm's recall. This render did not ask any registry whether the counterfactuals are really unpublished — the page is built offline. tti decoy does, and its figure is the one to quote. 27 event(s) could not be decoyed: 27 not a plain semantic version. The origin control's own re-grade: 0 of 21 page fetches matched the counterfactual (upper bound 15%) — the grader's rate on raw page bytes, with no provider in the loop.

At every rung, each event's canonical URL is also fetched directly, with robots.txt honoured, and checked for the answer token. Of the events in this run, 20 were confirmed retrievable from their own origin within 24 hours.

Without this arm, every ABSENT is ambiguous: a provider that does not return the fact might have a slow crawler, or the fact might not be fetchable from its own page yet. The 24h recall vs origin column asks the question only of events the control proved were on the web, and it is the column that is actually about the provider. Origins that refused us or that robots.txt excludes are omitted from that denominator rather than guessed at.

24-hour recall split by where the answer sat in its own document. The control arm already fetches every page, so this axis costs nothing, and it is the one that turns "provider X missed it" into something a buyer can act on. A fact in an <h1> and a fact buried in a __NEXT_DATA__ blob are different retrieval problems. An arm strong on the first and weak on the second is not slow — it does not execute JavaScript, and that is a sourcing decision, not a latency one. Denominators are events the control arm confirmed were on the web.

Kaplan–Meier needs a point event time, so feeding it the rung records an arm as indexing at the probe that first saw it. That overstates every latency by up to a bracket width, and it cannot represent a widened interval at all when a probe is dropped for budget, rung slip, or a provider error. This column is kept visible rather than deleted: if the overstatement is ever small, the ladder is fine enough that the choice of estimator does not matter, and that is worth knowing too.

Across every probe where the provider had not yet indexed the new answer and the question had a superseded one, it returned the superseded answer 22% of the time (16 of 73 opportunities).

Existing search benchmarks score a miss and a confidently-wrong stale answer identically. In production they are not the same event. An agent that gets nothing back retries, widens, or says it does not know. An agent that gets last quarter's number back cites it, and nothing downstream can tell that it is wrong.

Staleness should fall as an index catches up, so the shape matters as much as the level. A line that stays flat means the provider is holding a superseded answer with confidence rather than slowly acquiring the new one — a different problem, and a worse one. Only questions with a superseded answer count here; preprints and Federal Register documents have none, so they are excluded by Event.measures_staleness rather than by convention.

code release

gov publication

package registry

'titles-only' (2 of 4) makes arms unmeasurable rather than reordering them — the ranking looks stable under it only because there is nothing left to rank. Among the variants that keep the arms: 'shallow-walk' reorders the leaderboard (tau 0.33). The ranking is partly an artefact of the grading rules and should be published with that stated, not without it

Every grading rule here is a judgement call: match on token boundaries or not, count a v prefix, trust aliases, treat a URL as evidence. Each variant re-grades the stored payloads under one of those calls made differently — no API calls, which is why the raw responses are kept. Rank correlation alone is not enough, and this harness caught that about itself: a variant that reads no content collapses every arm equally, so the order never changes and the correlation comes back perfect for a table that has stopped meaning anything. Arms lost is the column that catches it.

6 comparisons, so the adjusted column is the one to read. Judging each pair against a raw α = 0.05 would give a 26% chance that at least one pair is called different when neither is — a leaderboard with six arms invites fifteen comparisons, and that is how a benchmark ships a difference it invented. Holm–Bonferroni rather than plain Bonferroni: same control of the family-wise error rate, uniformly more power, and no independence assumption, which matters because pairs sharing an arm are correlated. The read column uses the adjusted value.

Log-rank rather than a t-test on the indexed subset, because most observations are censored and a test that ignores censoring finds differences that are artefacts of who ran out of window first. The power column is here because the truthful answer early in a run is "not yet", and a leaderboard invites a claim long before the data can carry it: at the observed hazard ratio, Schoenfeld's formula says how many events 80% power would take. A hazard ratio of 2 needs 66 events; a ratio of 1.2 needs 945.