Map benchmark: which site map API gets an agent the right answer
List every URL in a section of a site nobody has indexed yet. 2 providers ran the same 15 tasks with the same model and prompt. Each task allowed 2 calls, 2 minutes and $0.25.
Context.dev finished the most map tasks, but its lead over Firecrawl (40%) is too small to rule out chance. Context.dev cost the least per finished task, 1.84¢. With no tool at all, the model alone passed none.
How a map task runs
Tasks done every answer correct, checked by script, with the cost of each finished task
None of the gaps between providers is big enough to rule out chance on 15 tasks, since tasks from the same site are counted as one group.
Cost per finished task all API and model spend, failures included, divided by tasks done
Time per finished task median, lower is better
Raw output, no model does the API's own output contain every answer? Cost is the API alone
Every provider the numbers behind the charts
| Provider | Tasks done | 95% interval | Cost per finished task | Median time |
|---|---|---|---|---|
| Context.dev | 7 of 15 (47%) | 11%–100% | 1.84¢ | 6.1 s |
| Firecrawl | 6 of 15 (40%) | 11%–100% | 3.35¢ | 11 s |
The test site was built fresh for this run, so no provider could have indexed it. Map services returned sparse or empty lists on those new paths. That says something about cold sites, not about mature, indexed ones.
Context.dev had the shortest median time per finished task, 6.1 s. Not tested: Exa, Tavily, Bright Data and Perplexity.
Head to head pairs that both ran map
Other tracks
How every task is graded, and when a gap counts, is on the methodology page.