Sitemap crawl API

Every URL from the sitemap in one call

curl https://api.specstream.ai/firecrawl/v2/map \
  -H "Authorization: Bearer $SPECSTREAM_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://quotes.toscrape.com/", "sitemap": "include", "limit": 5000}'

The reply is Firecrawl's: {"success": true, "links": [{"url": "..."}, ...]}. Set sitemap to only for the URLs the sitemap lists, skip for the URLs found by following links, or include for both. search puts matching links first. The call costs $0.0008, one page read.

Then crawl the part you need

A crawl is a job you start and then read. includePaths and excludePaths take regular expressions, maxDiscoveryDepth stops the walk, and scrapeOptions.formats asks for markdown, html, rawHtml or links. A job reads up to 500 pages, and the status read pages through next as Firecrawl's does.

curl https://api.specstream.ai/firecrawl/v2/crawl \
  -H "Authorization: Bearer $SPECSTREAM_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://quotes.toscrape.com/", "includePaths": ["^/author/.*"], "limit": 100, "scrapeOptions": {"formats": ["markdown"]}}'

One price a map, one price a page

A map that gives links costs $0.0008 whether it returns 20 URLs or 5,000. Each crawled page that gives text costs $0.0008, which is $0.80 per 1,000, and a page that gives no text costs nothing. creditsUsed on the status read counts the pages charged. A busy call gets 429 with Retry-After and charges nothing. Credit you buy never expires.

On Tavily a map of 500 URLs is 50 credits, $0.40 at $0.008 a credit. On Firecrawl 1,000 crawled pages are 1,000 credits, a fifth of the $19 Hobby plan.

The clients you already run

firecrawl-py takes Firecrawl(api_key=KEY, api_url="https://api.specstream.ai/firecrawl"), and map, crawl, start_crawl, get_crawl_status and cancel_crawl run as before. LangChain's FirecrawlLoader and LlamaIndex's FireCrawlWebReader take the same api_url in their map and crawl modes. In Dify, set the Firecrawl data source base URL to https://api.specstream.ai/firecrawl.

tavily-python takes TavilyClient(api_key=KEY, api_base_url="https://api.specstream.ai/tavily"), and client.map(url) runs as before. Each move has its own page: Firecrawl crawl and map and Tavily map.

Scored on the official benchmarks

The SpecStream Extract API, which reads every crawled page, scores 0.908 F1 on WCXB. The Search API scores 97.9 F-score on all 1,000 SimpleQA Verified questions, 978 of them correct. Every score and its method is on the benchmarks page.

Pages, search and rows on one key

The same key reads up to 20 pages or documents a call with the Extract API, searches the web from $1 per 1,000 searches, and runs Deep Search, which returns every match for a request in plain words as structured rows at 5¢, 15¢ or $1 a row. Source URL and read time on every value.

Crawl4AI

Free to self host, or a hosted API from $10 a month for 10,000 credits, where a plain page is 0.1 credit and a page in a real browser 0.5. SpecStream answers a Crawl4AI server's /crawl, /crawl/stream and /md at https://api.specstream.ai/crawl4ai, with nothing to host.

Spider

Pay as you go from $1 per GB plus compute at $0.0001 per minute of CPU. SpecStream answers Spider's scrape at https://api.specstream.ai/spider at $0.80 per 1,000 pages, and its search as one web search.

ScrapeGraphAI

Its Sitemap service is removed in API v2, and its docs point to Crawl in links mode or to scraping the sitemap.xml file. On SpecStream the sitemap is a parameter of the map call, sitemap: only, at $0.0008 a map.

Eden AI

A sitemap extraction API behind its AI API Gateway, which lists a 5.5% platform fee. SpecStream lists the URLs and reads the pages on one key and one balance, with no fee on top.