Why most gaps between web scraping APIs are noise
Of the 19 gaps between a provider and its track's leader in this run, 9 are big enough to rule out chance and 10 aren't. A ranking still puts every provider in order, but many of those places could swap if the same test ran on a different sample of sites.
Size alone doesn't decide it. On scrape, Linkup's 10-point gap to Context.dev cleared chance over 40 tasks. On map, Context.dev's 13-point gap to String didn't, because that track has 15 tasks and each one moves the score by 6.7 points.
How a gap is tested
Every provider runs the same tasks with the same model, Claude Haiku 5.5, and a script grades each task pass or fail. Tasks from one site tend to pass or fail together, so they aren't independent observations.
So each provider is compared with its track's leader on the same tasks, and a bootstrap redraws whole sites, not single tasks, 10,000 times. The middle 95% of those redrawn differences is the interval. If it includes zero, the gap is too small to rule out chance. An interval that ends at exactly zero still includes it.
These comparisons are exploratory. They aren't adjusted for testing many pairs at once, and the reference is whichever provider happened to score highest, which makes a false positive more likely, not less. Treat a gap that cleared chance as evidence about these tasks, not as proof for every site.
Scrape 40 tasks from 37 sites, so one task is 2.5 points.
| Provider | Behind Context.dev | 95% interval (points) | Verdict |
|---|---|---|---|
| Spider | 3 points | −8 to 0 | Too small to call |
| Scrapfly | 5 points | −12 to 0 | Too small to call |
| String | 8 points | −16 to 0 | Too small to call |
| Linkup | 10 points | −21 to −2 | Cleared chance |
| Firecrawl | 15 points | −27 to −5 | Cleared chance |
| TinyFish | 18 points | −33 to −5 | Cleared chance |
| Browserbase | 20 points | −34 to −8 | Cleared chance |
| You.com | 23 points | −38 to −8 | Cleared chance |
| Jina Reader | 80 points | −92 to −67 | Cleared chance |
Context.dev (95%) is the reference here, as the provider that finished the most scrape tasks. Spider, Scrapfly and String are behind, but each interval includes zero, so the gap is too small to call. Linkup, Firecrawl, TinyFish, Browserbase, You.com and Jina Reader each finished clearly fewer tasks: those intervals stay below zero.
Search 30 tasks from 18 sites, so one task is 3.3 points.
| Provider | Behind Linkup | 95% interval (points) | Verdict |
|---|---|---|---|
| TinyFish | 7 points | −21 to 6 | Too small to call |
| Firecrawl | 7 points | −23 to 6 | Too small to call |
| Context.dev | 10 points | −27 to 4 | Too small to call |
| String | 17 points | −31 to −4 | Cleared chance |
| You.com | 23 points | −46 to −6 | Cleared chance |
| Browserbase | 23 points | −43 to −6 | Cleared chance |
Linkup (93%) is the reference here, as the provider that finished the most search tasks. TinyFish, Firecrawl and Context.dev are behind, but each interval includes zero, so the gap is too small to call. String, You.com and Browserbase each finished clearly fewer tasks: those intervals stay below zero.
Map 15 tasks from 15 sites, so one task is 6.7 points.
| Provider | Behind String | 95% interval (points) | Verdict |
|---|---|---|---|
| Firecrawl | 7 points | −27 to 13 | Too small to call |
| Context.dev | 13 points | −40 to 13 | Too small to call |
String (93%) is the reference here, as the provider that finished the most map tasks. Firecrawl and Context.dev are behind, but each interval includes zero, so the gap is too small to call. No gap on this track cleared chance.
Crawl 12 tasks from 12 sites, so one task is 8.3 points.
| Provider | Behind Scrapfly | 95% interval (points) | Verdict |
|---|---|---|---|
| Firecrawl | 0 points | 0 to 0 | Tied |
| Context.dev | 8 points | −25 to 0 | Too small to call |
Scrapfly (100%) is the reference here, one of the providers tied for the most crawl tasks. Firecrawl finished exactly as many tasks, so the two share the lead. Context.dev is behind, but its interval includes zero, so the gap is too small to call. No gap on this track cleared chance.
Short answers
Are there benchmarks of web scraping APIs that report confidence intervals?
Yes. scrapingbench grades web data APIs on 40 scrape, 30 search, 15 map and 12 crawl tasks, each pass or fail by a script, and reports a 95% interval for every score and for every gap to a track's leader, from a bootstrap that resamples whole sites. The tasks, grader and every result row are open source and MIT licensed.
Does "too small to rule out chance" mean two providers are equally good?
No. It means this sample of tasks can't tell them apart. The provider ahead may well be better; the test just doesn't have enough sites to show it. A bigger or different task set could resolve the gap either way.
Why doesn't the benchmark name one winner per track?
Because on most tracks the leader's lead isn't big enough. On scrape, 3 of 9 gaps to Context.dev are too small to call or tied. On search, 3 of 6 gaps to Linkup are too small to call or tied. On map, 2 of 2 gaps to String are too small to call or tied. On crawl, 2 of 2 gaps to Scrapfly are too small to call or tied. The site only says a lead rules out chance when every gap to the leader cleared it.
How should I benchmark scraping APIs for my own agent?
Use tasks from the sites your agent will really visit, give every provider the same tasks, model and limits, and grade the final answer with a script rather than a model. Count the spend on failed tasks too, so cost means cost per finished task. Spread tasks over many sites, since tasks from one site count as one cluster, and compare providers on the same tasks with an interval before you call a winner.
How tasks are graded is on the methodology page, what the cost figure counts is on cost per finished task, and every pair people search for is on the compare page.