String vs Spider for AI agents
String and Spider both ran scrape, with the same tasks, the same model (Claude Haiku 5.5), the same prompt and the same limits. A task counts only when a script confirms every answer. Beyond that, only String ran search and only String ran map.
The benchmark tests each provider's gap to its track's leader, with 95% intervals from resampling whole sites. It doesn't test every pair directly, so this page says when a gap is real and when it isn't one anybody can claim.
Scrape 40 tasks · Test heading and structure preservation on documentation pages, plus package versions and JS fields.
Neither gap to the track's leader cleared chance, and the benchmark doesn't test these two against each other, so this 5-point gap isn't a result either can claim.
Spider cost 0.88¢ per finished task and String 1.33¢, counting all API and model spend, failures included.
More
Every provider on each track: scrape. How a gap is tested: methodology.