Open source web scraping API in 2026
Keep the client, not the server.
Two open-source scraping APIs lead the 2026 lists: Firecrawl, under AGPL-3.0, and Crawl4AI, under Apache-2.0. The code is free. The servers, the browsers, the proxies and the pages that block them are yours to run.
SpecStream is the live web data engine for teams and agents. It answers the Firecrawl and Crawl4AI APIs at their own paths, so your client stays and the server goes: $0.80 per 1,000 pages, and a page with no text costs nothing.
New accounts get $10 of credit, no card. It covers 12,500 pages.
SpecStream next to self-hosted Firecrawl and Crawl4AI
| SpecStream | Firecrawl, self-hosted | Crawl4AI, your own server | |
|---|---|---|---|
| License | A hosted API. Your code keeps the open-source client it already uses | AGPL-3.0 for the project. The SDKs and some UI components are MIT | Apache-2.0 |
| What you run | Nothing. Calls go to https://api.specstream.ai | Docker Compose with the Firecrawl API and workers, Playwright, Redis, RabbitMQ and PostgreSQL. You own security, availability, capacity, upgrades, data retention and compliance | A Docker server on your machine that runs the browsers, or the library in your own Python process |
| The API | Firecrawl v2 and v1 /scrape, /search, /map and /crawl at /firecrawl. Crawl4AI /crawl, /crawl/stream and /md at /crawl4ai. Its own POST /v1/extract reads up to 20 URLs a call | The Firecrawl API on port 3002. The default API is unauthenticated | /md, /html, /crawl, /crawl/stream, /screenshot, /pdf and /execute_js, each behind your CRAWL4AI_API_TOKEN |
| Pages that block a plain read | Tried again as a browser would ask, then in a real browser, then from the site's own feed, then from its newest public archive capture | Bundled Playwright with a basic fetch fallback. Fire-engine is a separate engine you connect | Your settings, your proxies |
| What a page costs | $0.80 per 1,000 pages from a balance, with no plan. A page with no text is not charged, and credit never expires | The code is free. The servers it runs on are yours to pay for | Free under Apache-2.0. You cover hosting |
| Structured data | Deep Search: describe the records and get rows that meet every condition, at 5¢, 15¢ or $1 a delivered row, with the source URL and read time on every value | AI features need a model provider you connect: OpenAI, an OpenAI-compatible endpoint or Ollama | The cloud's /extract needs no LLM key of your own. On your own server, the model key is yours |
| MCP | https://api.specstream.ai/mcp, with search_web and read_pages on the same key | A local MCP server that you point at your own API | MCP on your own server, for Claude Code and other agents |
| To start | $10 of credit on sign-up, no card, which covers 12,500 pages | Clone the repository, pick a release tag and run Docker Compose | docker run the server image, or install the library |
Why developers move their open-source scraper to SpecStream
Your client stays, and the server goes
Firecrawl's own self-hosting file lists what its Compose stack runs: the API and workers, Playwright, Redis, RabbitMQ, PostgreSQL and FoundationDB for an optional queue. It also says the default API is unauthenticated and that you own security, availability, capacity and upgrades.
SpecStream answers the same calls at https://api.specstream.ai/firecrawl and https://api.specstream.ai/crawl4ai, in each API's own JSON. The address changes and the parser stays. The Firecrawl migration page and the Crawl4AI migration page list each parameter and how it was measured.
Blocked pages are read, and empty pages are free
A self-hosted scraper reads what its own browser and proxies can reach. Firecrawl's self-hosted default is bundled Playwright with a basic fetch fallback, and Crawl4AI's own server runs on your settings and your proxies.
The SpecStream Extract API tries a page that refuses a plain read again as a browser would ask, then in a real browser, then from the site's own feed, then from its newest public archive capture. It reads PDF, Word and Excel files too. It scores 0.908 F1 on WCXB, and a page that gives no text is not charged.
Search and pages on one key
The same key runs the Search API, the most accurate web search API in the world: 97.9 F-score on all 1,000 SimpleQA Verified questions and 64.0 item F1 on all 200 WideSearch tasks (benchmarks). A search through the Firecrawl address costs $1 per 1,000 searches of up to 10 results, and scrapeOptions with markdown brings the page text with it.
Rows when you need records, not pages
Most scrapers end at a page of Markdown, and your code turns pages into records. Deep Search starts from the records: you describe them in plain words, see the most the run can cost before it starts, and get structured rows that meet every condition at 5¢, 15¢ or $1 a delivered row. A saved request reruns hourly to yearly, and each row comes back labeled changed, unchanged or no longer matching.
Other open-source scraping options
SpecStream
The live web data engine for teams and agents. It answers the Firecrawl and Crawl4AI APIs with nothing to host, at $0.80 per 1,000 pages and $1 per 1,000 searches through those addresses, and returns structured rows at 5¢, 15¢ or $1 a row. New accounts get $10 of credit, no card.
Firecrawl Cloud
The hosted Firecrawl bills 1 credit a page on a basic scrape or crawl, 4 more for JSON, and 2 credits per 10 search results. Hobby costs $16 a month billed yearly for 5,000 credits, Standard $83 for 100,000 and Growth $333 for 500,000. On SpecStream a page is $0.80 per 1,000 from a balance, with no plan.
Crawl4AI Cloud
Hosted Crawl4AI sells credits worth $0.001 each, with 1,000 free a month and plans from $10 a month. A plain page is 0.1 credits, a real browser page 0.5 and a hard site 2. SpecStream charges one price per page, and a page that gives no text costs nothing.
Scrapy
A fast high-level web crawling and scraping framework for Python, under BSD-3-Clause. You write a spider for each site and run it yourself. With SpecStream you send the URL, or describe the records, and get the text or the rows back.
Crawlee
A web scraping and browser automation library for Node.js from Apify, under Apache-2.0. Apify's hosted plans run from $0 to $999 a month, with compute units from $0.13 to $0.2. SpecStream has one rate card on every plan, and a top-up never expires.
How SpecStream scores
The same numbers as the benchmarks page, from the official tasks and scorers.
| SpecStream |  Parallel |  Exa |  Tavily |  Perplexity | |
|---|---|---|---|---|---|
| Web researchWideSearch, GPT-5.6 Luna | 60.9 | 45.5 | 53.0 | 47.9 | 47.0 |
| Factual accuracySimpleQA Verified, GPT-5.6 Luna | 96 | 94 | 91 | 94 | 94 |
| Factual accuracySimpleQA Verified, GPT-5.6 Sol | 99 | 97 | 91 | 92 | 95 |
How to move a self-hosted scraper to SpecStream
1. Create a key under Agents & API in the workspace. A new account gets $10 of credit, no card, which covers 12,500 pages.
2. Self-hosted Firecrawl: set the address to https://api.specstream.ai/firecrawl where your code has http://localhost:3002, and send the SpecStream key as the Bearer token. In Python:
import os
import requests
BASE = "https://api.specstream.ai/firecrawl" # was http://localhost:3002
HEADERS = {"Authorization": f"Bearer {os.environ['SPECSTREAM_API_KEY']}"}
page = requests.post(
f"{BASE}/v2/scrape",
json={"url": "https://en.wikipedia.org/wiki/Web_scraping"},
headers=HEADERS,
).json()
print(page["data"]["metadata"].get("title"), page["data"]["markdown"][:500])3. Crawl4AI server: set the address to https://api.specstream.ai/crawl4ai where your code has http://localhost:11235, and send the SpecStream key in place of the server's API token. In JavaScript:
const reply = await fetch("https://api.specstream.ai/crawl4ai/md", { // was http://localhost:11235/md
method: "POST",
headers: { Authorization: `Bearer ${process.env.SPECSTREAM_API_KEY}`, "Content-Type": "application/json" },
body: JSON.stringify({ url: "https://en.wikipedia.org/wiki/Web_scraping", f: "raw" }),
});
const { markdown } = await reply.json();
console.log(markdown.slice(0, 500));4. Read 429 as a call to send again after Retry-After. It is not charged. Each reply carries the price of the call, and every call is listed in your workspace.
Frequently asked questions
Something we didn't answer? Talk to a human
What is the best open source web scraping API?
Is Firecrawl open source?
Can I use the Firecrawl SDK without running Firecrawl?
https://api.specstream.ai/firecrawl and use a SpecStream key. Scrape, search, map and crawl answer in Firecrawl's v2 and v1 JSON, at $0.80 per 1,000 pages and $1 per 1,000 searches.Is Crawl4AI free?
/crawl and /md at $0.80 per 1,000 pages.How much does SpecStream cost per page?
Does SpecStream have an MCP server?
https://api.specstream.ai/mcp over Streamable HTTP, with OAuth or an API key. An agent gets search_web and read_pages on the same balance, inside the limits you set on its key.Sources
SpecStream wrote this page. Each figure about another project comes from that project's own page or repository.
- SpecStream pricingspecstream.ai/pricing, read Nov 12, 2026
- SpecStream for agentsspecstream.ai/agents.md, read Nov 12, 2026
- SpecStream benchmarksspecstream.ai/benchmarks, read Nov 12, 2026
- Firecrawl READMEgithub.com/firecrawl/firecrawl, read Nov 12, 2026
- Firecrawl SELF_HOST.mdgithub.com/firecrawl/firecrawl/blob/main/SELF_HOST.md, read Nov 12, 2026
- Firecrawl pricingfirecrawl.dev/pricing, read Nov 12, 2026
- Crawl4AI READMEgithub.com/unclecode/crawl4ai, read Nov 12, 2026
- Crawl4AI pricingcrawl4ai.com/pricing, read Nov 12, 2026
- Scrapygithub.com/scrapy/scrapy, read Nov 12, 2026
- Crawleegithub.com/apify/crawlee, read Nov 12, 2026
- Apify pricingapify.com/pricing, read Nov 12, 2026