LlamaIndex FireCrawlWebReader on SpecStream

The change is one argument

FireCrawlWebReader passes api_url to the Firecrawl client when it is set, and the client calls that address in place of Firecrawl's. Before:

from llama_index.readers.web import FireCrawlWebReader

reader = FireCrawlWebReader(
    api_key="fc-...",
    mode="crawl",
    params={"limit": 2},
)
documents = reader.load_data(url="https://example.com")

After, with the SpecStream key and one more argument:

import os
from llama_index.readers.web import FireCrawlWebReader

reader = FireCrawlWebReader(
    api_key=os.environ["SPECSTREAM_API_KEY"],
    api_url="https://api.specstream.ai/firecrawl",
    mode="crawl",
    params={"limit": 2},
)
documents = reader.load_data(url="https://example.com")

The same reader runs mode="scrape" for one page and mode="map" for a site's links. On November 10, 2026, llama-index-readers-web 0.7.0 with firecrawl-py 4.55.0 read https://example.com in scrape and crawl, and mapped https://docs.python.org/3/ into 44 documents, all on https://api.specstream.ai/firecrawl.

The same Firecrawl calls underneath

The reader's scrape is a Firecrawl v2 scrape. SpecStream answers it in Firecrawl's shape, with data.markdown and data.metadata, so the reader's Document gets the page text and its metadata. The same call in plain Python:

import os, requests

reply = requests.post(
    "https://api.specstream.ai/firecrawl/v2/scrape",
    headers={"Authorization": f"Bearer {os.environ['SPECSTREAM_API_KEY']}"},
    json={"url": "https://example.com"},
).json()

print(reply["data"]["metadata"])
print(reply["data"]["markdown"][:500])

Every parameter and field of the scrape route is in the Firecrawl guide, and the map and crawl routes are in the Firecrawl crawl and map guide.

$0.80 per 1,000 pages, and empty pages are free

A scraped page is one page read at $0.0008. A crawl charges one page read for each page with text, and a map is one page read. A page that gives no text is not charged. There is no monthly plan, credit you buy never expires, and a new account starts with $10 of credit, which covers 12,500 pages. A busy call gets a 429 with Retry-After and charges nothing.

The text your index gets is the text on the page

The Extract API, which answers every page the reader asks for, scores 0.908 F1 on WCXB. The Search API scores 97.9 F-score on all 1,000 SimpleQA Verified questions, 978 of them correct, and 64.0 item F1 and 43.4 row F1 on all 200 WideSearch tasks. Every score and its method is on the benchmarks page.

You decide what is kept, on each key

Create the key with zero_retention and a call made with it keeps no request text and no answer after the reply. spend_cap_cents is the most the key can spend, so a crawl in a loop stops at your number.

curl -X POST https://api.specstream.ai/v1/account/keys \
  -H "Authorization: Bearer $SPECSTREAM_API_KEY" -H "Content-Type: application/json" \
  -d '{"name": "production", "scopes": ["account:read", "finds:spend"], "spend_cap_cents": 500000, "zero_retention": true}'

Search, pages and rows on one key

The same key searches the live web with the Search API at $5 per 1,000 searches of up to 10 results, with no daily limit, and reads up to 20 pages or documents a call with the Extract API. Deep Search returns every match for a request in plain words as structured rows, at 5¢, 15¢ or $1 a row, with a source URL on every value. The most a run can cost is approved before it starts.

Firecrawl

The reader's default. Scrape and crawl cost 1 credit a page and a map 1 credit a call, on plans from $19 a month for 5,000 credits to $99 a month for 100,000. SpecStream runs the same reader at $0.80 per 1,000 pages, with no plan, behind one argument.

Exa

A search API that sells page contents at $1 per 1,000 pages. SpecStream reads pages at $0.80 per 1,000, 20 pages or documents to a call.

Spider

A scrape and crawl API billed by bandwidth, from $1 per GB. SpecStream bills by the page, $0.0008 each, and a page with no text costs nothing.