Search benchmark: which web search API gets an agent the right answer
Find a fact on the live web: a package's latest version, a filing figure, a release date. 5 providers ran the same 60 tasks with the same model and prompt. Each task allowed 3 calls, 60 seconds and $0.10.
Firecrawl finished the most search tasks, but its lead over Linkup (83%) is too small to rule out chance. TinyFish cost the least per finished task, 1.31¢. With no tool at all, the model alone passed 23%.
How a search task runs
Tasks done every answer correct, checked by script, with the cost of each finished task
The only gap big enough to rule out chance: Firecrawl finished 47 points more tasks than Browserbase. The rest can't be told apart yet.
Cost per finished task all API and model spend, failures included, divided by tasks done
Time per finished task median, lower is better
Raw output, no model does the API's own output contain every answer? Cost is the API alone
5 of the 60 tasks have two defensible answers: three founding years where Wikidata differs from the year most sources give, and two releases just after midnight UTC that providers dated in US time. Dropping them, or accepting both answers, moves Firecrawl to 92% at most and changes no provider's place.
Every provider the numbers behind the charts
| Provider | Tasks done | 95% interval | Cost per finished task | Median time |
|---|---|---|---|---|
| Firecrawl | 51 of 60 (85%) | 69%–100% | 3.04¢ | 11 s |
| Linkup | 50 of 60 (83%) | 65%–97% | 2.55¢ | 5.3 s |
| Context.dev | 45 of 60 (75%) | 52%–96% | 1.82¢ | 5.2 s |
| TinyFish | 44 of 60 (73%) | 47%–96% | 1.31¢ | 5.3 s |
| Browserbase | 23 of 60 (38%) | 21%–59% | 5.97¢ | 12 s |
Context.dev had the shortest median time per finished task, 5.2 s. Not tested: Exa, Tavily, Bright Data and Perplexity.
Head to head pairs that both ran search
Other tracks
How every task is graded, and when a gap counts, is on the methodology page.