<!-- twin of https://specstream.ai/sitemap-crawl-api -->
# Sitemap crawl API: a site map for $0.0008, pages at $0.80 per 1,000

Compared on Nov 12, 2026. The page: https://specstream.ai/sitemap-crawl-api

Firecrawl's map takes 1 credit a call and its crawl 1 credit a page, on plans from $19 a month for 5,000 credits. Tavily maps at 1 credit per 10 pages, and Cloudflare's crawl bills browser hours at $0.09 past 10 a month.

SpecStream is the live web data engine for teams and agents. One call lists a site's URLs from its sitemap and its links for $0.0008, and a crawl reads each page as markdown at $0.80 per 1,000 pages, with no plan to buy.

[Get an API key](https://specstream.ai/signup?src=alternatives-sitemap-crawl-api&utm_source=alternatives-md&utm_campaign=sitemap-crawl-api&returnTo=%2Fworkspace%2Fsearch%3Froute%3Dfirecrawl) New accounts get $10 of credit, no card. It covers 12,500 maps or pages.

## Sitemap and crawl APIs side by side, from each company's own page

| | SpecStream | Firecrawl | Tavily | Cloudflare Browser Rendering | Apify sitemap actors |
|---|---|---|---|---|---|
| List a site's URLs | `POST /firecrawl/v2/map` or `POST /tavily/map`, up to 5,000 links a map | `POST /v2/map`. URLs come mainly from the sitemap, plus search results and pages crawled before | `POST /map`, 50 links by default | A crawl job, with `source` set to `all`, `sitemaps` or `links` | An actor run that reads a sitemap or sitemap index into a flat list of URLs |
| Sitemap control | `sitemap` as `include`, `skip` or `only`, plus `search`, `includeSubdomains` and `ignoreQueryParameters` | `sitemap` as `skip`, `include` or `only`. Sitemap data is cached for up to 7 days | `select_paths` and `exclude_paths` as regular expressions | `source: sitemaps` crawls only the URLs in the sitemaps | The actor's own input |
| Price of a map | $0.0008, whatever its size | 1 credit a call | 1 API credit per 10 pages, 2 per 10 with `instructions` | Browser hours: 10 a month on Workers Paid, then $0.09 an hour | From $0.20 per 1,000 results, plus Apify platform usage |
| Crawl the pages | `POST /firecrawl/v2/crawl` with `includePaths`, `excludePaths`, `maxDiscoveryDepth` and `crawlEntireDomain`, as `markdown`, `html`, `rawHtml` or `links` | `POST /v2/crawl`, 1 credit a page | `POST /crawl`, 1 credit per 5 successful extractions | The same crawl job returns HTML, Markdown or JSON | Another actor run, at that actor's price |
| Price per 1,000 pages | $0.80. A page that gives no text costs nothing | 1 credit a page, from $19 a month for 5,000 credits on Hobby. A page with no document is 0 credits | Pay as you go at $0.008 a credit | Billed by browser time, not by page | Priced per actor, plus platform usage |
| Free to start | $10 of credit, no card, which covers 12,500 maps or pages | 1,000 free credits a month | 1,000 free API credits a month | 10 minutes of browser time a day on Workers Free | $5 a month to spend on the Free plan |

## Why map and crawl on SpecStream

### Every URL from the sitemap in one call

```
curl https://api.specstream.ai/firecrawl/v2/map \
  -H "Authorization: Bearer $SPECSTREAM_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://quotes.toscrape.com/", "sitemap": "include", "limit": 5000}'
```

The reply is Firecrawl's: `{"success": true, "links": [{"url": "..."}, ...]}`. Set `sitemap` to `only` for the URLs the sitemap lists, `skip` for the URLs found by following links, or `include` for both. `search` puts matching links first. The call costs $0.0008, one page read.

### Then crawl the part you need

A crawl is a job you start and then read. `includePaths` and `excludePaths` take regular expressions, `maxDiscoveryDepth` stops the walk, and `scrapeOptions.formats` asks for `markdown`, `html`, `rawHtml` or `links`. A job reads up to 500 pages, and the status read pages through `next` as Firecrawl's does.

```
curl https://api.specstream.ai/firecrawl/v2/crawl \
  -H "Authorization: Bearer $SPECSTREAM_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://quotes.toscrape.com/", "includePaths": ["^/author/.*"], "limit": 100, "scrapeOptions": {"formats": ["markdown"]}}'
```

### One price a map, one price a page

A map that gives links costs $0.0008 whether it returns 20 URLs or 5,000. Each crawled page that gives text costs $0.0008, which is $0.80 per 1,000, and a page that gives no text costs nothing. `creditsUsed` on the status read counts the pages charged. A busy call gets `429` with `Retry-After` and charges nothing. Credit you buy never expires.

On Tavily a map of 500 URLs is 50 credits, $0.40 at $0.008 a credit. On Firecrawl 1,000 crawled pages are 1,000 credits, a fifth of the $19 Hobby plan.

### The clients you already run

firecrawl-py takes `Firecrawl(api_key=KEY, api_url="https://api.specstream.ai/firecrawl")`, and `map`, `crawl`, `start_crawl`, `get_crawl_status` and `cancel_crawl` run as before. LangChain's `FirecrawlLoader` and LlamaIndex's `FireCrawlWebReader` take the same `api_url` in their `map` and `crawl` modes. In Dify, set the Firecrawl data source base URL to `https://api.specstream.ai/firecrawl`.

tavily-python takes `TavilyClient(api_key=KEY, api_base_url="https://api.specstream.ai/tavily")`, and `client.map(url)` runs as before. Each move has its own page: [Firecrawl crawl and map](https://specstream.ai/migrate/firecrawl-crawl?utm_source=alternatives&utm_campaign=sitemap-crawl-api) and [Tavily map](https://specstream.ai/migrate/tavily-map?utm_source=alternatives&utm_campaign=sitemap-crawl-api).

### Scored on the official benchmarks

The SpecStream Extract API, which reads every crawled page, scores 0.908 F1 on WCXB. The Search API scores 97.9 F-score on all 1,000 SimpleQA Verified questions, 978 of them correct. Every score and its method is on the [benchmarks](https://specstream.ai/benchmarks?utm_source=alternatives&utm_campaign=sitemap-crawl-api) page.

### Pages, search and rows on one key

The same key reads up to 20 pages or documents a call with the [Extract API](https://specstream.ai/extract-api?utm_source=alternatives&utm_campaign=sitemap-crawl-api), searches the web from $1 per 1,000 searches, and runs [Deep Search](https://specstream.ai/deep-search?utm_source=alternatives&utm_campaign=sitemap-crawl-api), which returns every match for a request in plain words as structured rows at 5¢, 15¢ or $1 a row. Source URL and read time on every value.

## Other sitemap and crawl APIs

### Crawl4AI

Free to self host, or a hosted API from $10 a month for 10,000 credits, where a plain page is 0.1 credit and a page in a real browser 0.5. SpecStream answers a Crawl4AI server's `/crawl`, `/crawl/stream` and `/md` at `https://api.specstream.ai/crawl4ai`, with nothing to host.

### Spider

Pay as you go from $1 per GB plus compute at $0.0001 per minute of CPU. SpecStream answers Spider's scrape at `https://api.specstream.ai/spider` at $0.80 per 1,000 pages, and its search as one web search.

### ScrapeGraphAI

Its Sitemap service is removed in API v2, and its docs point to Crawl in links mode or to scraping the `sitemap.xml` file. On SpecStream the sitemap is a parameter of the map call, `sitemap: only`, at $0.0008 a map.

### Eden AI

A sitemap extraction API behind its AI API Gateway, which lists a 5.5% platform fee. SpecStream lists the URLs and reads the pages on one key and one balance, with no fee on top.

## How to map and crawl a site

1. Create a key under Agents & API in the workspace. A new account gets $10 of credit, no card, which covers 12,500 maps or pages.

2. Map the site, then crawl the URLs you want. In Python:

```
import time
import requests

BASE = "https://api.specstream.ai/firecrawl"  # was https://api.firecrawl.dev
HEADERS = {"Authorization": f"Bearer {SPECSTREAM_KEY}"}

site_map = requests.post(f"{BASE}/v2/map", headers=HEADERS,
                         json={"url": "https://quotes.toscrape.com/", "sitemap": "include"}, timeout=60).json()
print(len(site_map["links"]), "URLs")

job = requests.post(f"{BASE}/v2/crawl", headers=HEADERS,
                    json={"url": "https://quotes.toscrape.com/", "includePaths": ["^/author/.*"], "limit": 50}, timeout=60).json()
while True:
    state = requests.get(job["url"], headers=HEADERS, timeout=60).json()
    if state["status"] in ("completed", "failed", "cancelled"):
        break
    time.sleep(2)
for page in state["data"]:
    print(page["metadata"]["url"], len(page["markdown"]))
```

3. A map in JavaScript:

```
const reply = await fetch("https://api.specstream.ai/firecrawl/v2/map", {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.SPECSTREAM_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({ url: "https://quotes.toscrape.com/", sitemap: "only" }),
});
const { links } = await reply.json();
console.log(links.map((link) => link.url));
```

4. Read `429` as a call to send again after `Retry-After`. It is not charged.

## Frequently asked questions

### What is a sitemap crawl API?

An API that lists every URL on a site from its sitemap and its links, and can then read those pages. On SpecStream a map costs $0.0008 for up to 5,000 links, and a crawled page costs $0.0008.

### How do I get every URL from a website's sitemap?

Send `POST https://api.specstream.ai/firecrawl/v2/map` with the site's `url` and `"sitemap": "only"`. The reply's `links[]` holds each URL the sitemap lists, for $0.0008 a call.

### How much does Firecrawl map and crawl cost?

A Firecrawl map is 1 credit a call and a crawled page is 1 credit, and Hobby is $19 a month for 5,000 credits. SpecStream answers the same calls at `https://api.specstream.ai/firecrawl` for $0.0008 a map and $0.80 per 1,000 pages, with no plan.

### How much does Tavily map cost?

1 API credit per 10 pages, or 2 per 10 with `instructions`, at $0.008 a credit pay as you go. SpecStream answers Tavily's `POST /map` at `https://api.specstream.ai/tavily/map` for $0.0008 a map of up to 5,000 URLs.

### Can I crawl only the pages under one path?

Yes. `includePaths` and `excludePaths` take regular expressions, for example `^/blog/.*`, and `maxDiscoveryDepth` stops the walk. Each page that gives text costs $0.0008.

### Is there a free sitemap crawl API?

A new SpecStream account gets $10 of credit with no card, which covers 12,500 maps or pages, and the credit does not expire. Firecrawl and Tavily each give 1,000 free credits a month.

## Sources

SpecStream wrote this page. Each price and parameter comes from that company's own page, read on Nov 12, 2026.

- [Firecrawl map](https://docs.firecrawl.dev/features/map), read Nov 12, 2026
- [Firecrawl map API reference](https://docs.firecrawl.dev/api-reference/endpoint/map), read Nov 12, 2026
- [Firecrawl billing](https://docs.firecrawl.dev/billing), read Nov 12, 2026
- [Firecrawl pricing](https://www.firecrawl.dev/pricing), read Nov 12, 2026
- [Tavily map API reference](https://docs.tavily.com/documentation/api-reference/endpoint/map), read Nov 12, 2026
- [Tavily crawl API reference](https://docs.tavily.com/documentation/api-reference/endpoint/crawl), read Nov 12, 2026
- [Tavily pricing](https://www.tavily.com/pricing), read Nov 12, 2026
- [Cloudflare crawl endpoint](https://developers.cloudflare.com/browser-rendering/rest-api/crawl-endpoint/), read Nov 12, 2026
- [Cloudflare Browser Rendering pricing](https://developers.cloudflare.com/browser-rendering/pricing/), read Nov 12, 2026
- [Apify Sitemap Extractor](https://apify.com/axery/sitemap-extractor), read Nov 12, 2026
- [Apify pricing](https://apify.com/pricing), read Nov 12, 2026
- [Crawl4AI pricing](https://www.crawl4ai.com/pricing), read Nov 12, 2026
- [Spider pricing](https://spider.cloud/pricing), read Nov 12, 2026
- [ScrapeGraphAI Sitemap](https://docs.scrapegraphai.com/services/sitemap), read Nov 12, 2026
- [Eden AI sitemap extraction API](https://www.edenai.co/feature/sitemap-extraction-api), read Nov 12, 2026
- [SpecStream benchmarks](https://specstream.ai/benchmarks), read Nov 12, 2026
- [SpecStream for agents](https://specstream.ai/agents.md), read Nov 12, 2026