<!-- twin of https://specstream.ai/open-source-web-scraping-api -->
# Open source web scraping API in 2026: keep the client, not the server

Compared on Nov 12, 2026. The page: https://specstream.ai/open-source-web-scraping-api

Two open-source scraping APIs lead the 2026 lists: Firecrawl, under AGPL-3.0, and Crawl4AI, under Apache-2.0. The code is free. The servers, the browsers, the proxies and the pages that block them are yours to run.

SpecStream is the live web data engine for teams and agents. It answers the Firecrawl and Crawl4AI APIs at their own paths, so your client stays and the server goes: $0.80 per 1,000 pages, and a page with no text costs nothing.

[Get an API key](https://specstream.ai/signup?src=alternatives-open-source-web-scraping-api&utm_source=alternatives-md&utm_campaign=open-source-web-scraping-api&returnTo=%2Fworkspace%2Fsearch%3Froute%3Dfirecrawl) New accounts get $10 of credit, no card. It covers 12,500 pages.

## SpecStream next to self-hosted Firecrawl and Crawl4AI

| | SpecStream | Firecrawl, self-hosted | Crawl4AI, your own server |
|---|---|---|---|
| License | A hosted API. Your code keeps the open-source client it already uses | AGPL-3.0 for the project. The SDKs and some UI components are MIT | Apache-2.0 |
| What you run | Nothing. Calls go to `https://api.specstream.ai` | Docker Compose with the Firecrawl API and workers, Playwright, Redis, RabbitMQ and PostgreSQL. You own security, availability, capacity, upgrades, data retention and compliance | A Docker server on your machine that runs the browsers, or the library in your own Python process |
| The API | Firecrawl v2 and v1 `/scrape`, `/search`, `/map` and `/crawl` at `/firecrawl`. Crawl4AI `/crawl`, `/crawl/stream` and `/md` at `/crawl4ai`. Its own `POST /v1/extract` reads up to 20 URLs a call | The Firecrawl API on port `3002`. The default API is unauthenticated | `/md`, `/html`, `/crawl`, `/crawl/stream`, `/screenshot`, `/pdf` and `/execute_js`, each behind your `CRAWL4AI_API_TOKEN` |
| Pages that block a plain read | Tried again as a browser would ask, then in a real browser, then from the site's own feed, then from its newest public archive capture | Bundled Playwright with a basic fetch fallback. Fire-engine is a separate engine you connect | Your settings, your proxies |
| What a page costs | $0.80 per 1,000 pages from a balance, with no plan. A page with no text is not charged, and credit never expires | The code is free. The servers it runs on are yours to pay for | Free under Apache-2.0. You cover hosting |
| Structured data | Deep Search: describe the records and get rows that meet every condition, at 5¢, 15¢ or $1 a delivered row, with the source URL and read time on every value | AI features need a model provider you connect: OpenAI, an OpenAI-compatible endpoint or Ollama | The cloud's `/extract` needs no LLM key of your own. On your own server, the model key is yours |
| MCP | `https://api.specstream.ai/mcp`, with `search_web` and `read_pages` on the same key | A local MCP server that you point at your own API | MCP on your own server, for Claude Code and other agents |
| To start | $10 of credit on sign-up, no card, which covers 12,500 pages | Clone the repository, pick a release tag and run Docker Compose | `docker run` the server image, or install the library |

## Why developers move their open-source scraper to SpecStream

### Your client stays, and the server goes

Firecrawl's own self-hosting file lists what its Compose stack runs: the API and workers, Playwright, Redis, RabbitMQ, PostgreSQL and FoundationDB for an optional queue. It also says the default API is unauthenticated and that you own security, availability, capacity and upgrades.

SpecStream answers the same calls at `https://api.specstream.ai/firecrawl` and `https://api.specstream.ai/crawl4ai`, in each API's own JSON. The address changes and the parser stays. The [Firecrawl migration page](https://specstream.ai/migrate/firecrawl?utm_source=alternatives&utm_campaign=open-source-web-scraping-api) and the [Crawl4AI migration page](https://specstream.ai/migrate/crawl4ai?utm_source=alternatives&utm_campaign=open-source-web-scraping-api) list each parameter and how it was measured.

### Blocked pages are read, and empty pages are free

A self-hosted scraper reads what its own browser and proxies can reach. Firecrawl's self-hosted default is bundled Playwright with a basic fetch fallback, and Crawl4AI's own server runs on your settings and your proxies.

The SpecStream Extract API tries a page that refuses a plain read again as a browser would ask, then in a real browser, then from the site's own feed, then from its newest public archive capture. It reads PDF, Word and Excel files too. It scores 0.908 F1 on WCXB, and a page that gives no text is not charged.

### Search and pages on one key

The same key runs the [Search API](https://specstream.ai/search-api?utm_source=alternatives&utm_campaign=open-source-web-scraping-api), the most accurate web search API in the world: 97.9 F-score on all 1,000 SimpleQA Verified questions and 64.0 item F1 on all 200 WideSearch tasks ([benchmarks](https://specstream.ai/benchmarks?utm_source=alternatives&utm_campaign=open-source-web-scraping-api)). A search through the Firecrawl address costs $1 per 1,000 searches of up to 10 results, and `scrapeOptions` with `markdown` brings the page text with it.

### Rows when you need records, not pages

Most scrapers end at a page of Markdown, and your code turns pages into records. [Deep Search](https://specstream.ai/deep-search?utm_source=alternatives&utm_campaign=open-source-web-scraping-api) starts from the records: you describe them in plain words, see the most the run can cost before it starts, and get structured rows that meet every condition at 5¢, 15¢ or $1 a delivered row. A saved request reruns hourly to yearly, and each row comes back labeled changed, unchanged or no longer matching.

## Other open-source scraping options

### SpecStream

The live web data engine for teams and agents. It answers the Firecrawl and Crawl4AI APIs with nothing to host, at $0.80 per 1,000 pages and $1 per 1,000 searches through those addresses, and returns structured rows at 5¢, 15¢ or $1 a row. New accounts get $10 of credit, no card.

### Firecrawl Cloud

The hosted Firecrawl bills 1 credit a page on a basic scrape or crawl, 4 more for JSON, and 2 credits per 10 search results. Hobby costs $16 a month billed yearly for 5,000 credits, Standard $83 for 100,000 and Growth $333 for 500,000. On SpecStream a page is $0.80 per 1,000 from a balance, with no plan.

### Crawl4AI Cloud

Hosted Crawl4AI sells credits worth $0.001 each, with 1,000 free a month and plans from $10 a month. A plain page is 0.1 credits, a real browser page 0.5 and a hard site 2. SpecStream charges one price per page, and a page that gives no text costs nothing.

### Scrapy

A fast high-level web crawling and scraping framework for Python, under BSD-3-Clause. You write a spider for each site and run it yourself. With SpecStream you send the URL, or describe the records, and get the text or the rows back.

### Crawlee

A web scraping and browser automation library for Node.js from Apify, under Apache-2.0. Apify's hosted plans run from $0 to $999 a month, with compute units from $0.13 to $0.2. SpecStream has one rate card on every plan, and a top-up never expires.

## How to move a self-hosted scraper to SpecStream

1. Create a key under Agents & API in the workspace. A new account gets $10 of credit, no card, which covers 12,500 pages.

2. Self-hosted Firecrawl: set the address to `https://api.specstream.ai/firecrawl` where your code has `http://localhost:3002`, and send the SpecStream key as the Bearer token. In Python:

```
import os
import requests

BASE = "https://api.specstream.ai/firecrawl"  # was http://localhost:3002
HEADERS = {"Authorization": f"Bearer {os.environ['SPECSTREAM_API_KEY']}"}

page = requests.post(
    f"{BASE}/v2/scrape",
    json={"url": "https://en.wikipedia.org/wiki/Web_scraping"},
    headers=HEADERS,
).json()
print(page["data"]["metadata"].get("title"), page["data"]["markdown"][:500])
```

3. Crawl4AI server: set the address to `https://api.specstream.ai/crawl4ai` where your code has `http://localhost:11235`, and send the SpecStream key in place of the server's API token. In JavaScript:

```
const reply = await fetch("https://api.specstream.ai/crawl4ai/md", { // was http://localhost:11235/md
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.SPECSTREAM_API_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({ url: "https://en.wikipedia.org/wiki/Web_scraping", f: "raw" }),
});
const { markdown } = await reply.json();
console.log(markdown.slice(0, 500));
```

4. Read `429` as a call to send again after `Retry-After`. It is not charged. Each reply carries the price of the call, and every call is listed in your workspace.

## Frequently asked questions

### What is the best open source web scraping API?

Firecrawl and Crawl4AI lead the 2026 lists of open-source scraping APIs: Firecrawl under AGPL-3.0, Crawl4AI under Apache-2.0. SpecStream answers both APIs at their own paths for $0.80 per 1,000 pages with nothing to host, so the client you chose keeps working.

### Is Firecrawl open source?

Yes. Firecrawl is open source under the AGPL-3.0 license, and its SDKs and some UI components are MIT. The hosted version adds features, and the self-hosted stack runs the API and workers, Playwright, Redis, RabbitMQ and PostgreSQL on your servers.

### Can I use the Firecrawl SDK without running Firecrawl?

Yes. Set the SDK's API address to `https://api.specstream.ai/firecrawl` and use a SpecStream key. Scrape, search, map and crawl answer in Firecrawl's v2 and v1 JSON, at $0.80 per 1,000 pages and $1 per 1,000 searches.

### Is Crawl4AI free?

Yes, the Crawl4AI library and server are free under Apache-2.0, and you cover hosting. Its hosted cloud gives 1,000 free credits a month, with plans from $10. SpecStream answers the Crawl4AI server's `/crawl` and `/md` at $0.80 per 1,000 pages.

### How much does SpecStream cost per page?

$0.80 per 1,000 pages, or $0.0008 a page, up to 20 pages a call. A page that gives no text is not charged, there is no monthly plan, and a new account gets $10 of credit, no card.

### Does SpecStream have an MCP server?

Yes, at `https://api.specstream.ai/mcp` over Streamable HTTP, with OAuth or an API key. An agent gets `search_web` and `read_pages` on the same balance, inside the limits you set on its key.

## Sources

SpecStream wrote this page. Each figure about another project comes from that project's own page or repository.

- [SpecStream pricing](https://specstream.ai/pricing), read Nov 12, 2026
- [SpecStream for agents](https://specstream.ai/agents.md), read Nov 12, 2026
- [SpecStream benchmarks](https://specstream.ai/benchmarks), read Nov 12, 2026
- [Firecrawl README](https://github.com/firecrawl/firecrawl), read Nov 12, 2026
- [Firecrawl SELF_HOST.md](https://github.com/firecrawl/firecrawl/blob/main/SELF_HOST.md), read Nov 12, 2026
- [Firecrawl pricing](https://www.firecrawl.dev/pricing), read Nov 12, 2026
- [Crawl4AI README](https://github.com/unclecode/crawl4ai), read Nov 12, 2026
- [Crawl4AI pricing](https://www.crawl4ai.com/pricing), read Nov 12, 2026
- [Scrapy](https://github.com/scrapy/scrapy), read Nov 12, 2026
- [Crawlee](https://github.com/apify/crawlee), read Nov 12, 2026
- [Apify pricing](https://apify.com/pricing), read Nov 12, 2026