Data / Repositories / Web crawler
Web crawler repositories on GitHub: 16 ranked by stars
16 public repositories tagged Web crawler are on GitHub, with 371,817 stars between them; 7 had a push in the last 30 days. Most are written in Python (6), TypeScript (3), Rust (3). The most starred: firecrawl/firecrawl (191,485), unclecode/crawl4ai (85,530), ScrapeGraphAI/Scrapegraph-ai (31,623).
SpecStream is the live web data engine for teams and agents. It reads open-source repositories on GitHub and keeps what it reads, and this page groups that reading by topic. Ask for the repositories you track by topic, language, license or activity, and new ones and changes arrive in your sheet, CRM, Slack or agent.
Every repository, most starred first
| Repository | What it is | Language | Stars | Forks | License | Last push | Read |
|---|---|---|---|---|---|---|---|
| firecrawl/firecrawl | Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. | TypeScript | 191,485 | 10,124 | AGPL-3.0 | Oct 14 | Oct 14 |
| unclecode/crawl4ai | Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. | Python | 85,530 | 8,879 | Apache-2.0 | Oct 14 | Oct 14 |
| ScrapeGraphAI/Scrapegraph-ai | Python scraper based on AI | Python | 31,623 | 3,221 | MIT | Oct 6 | Oct 8 |
| apify/crawlee | Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML… | TypeScript | 25,860 | 1,677 | Apache-2.0 | Sep 21 | Sep 21 |
| apify/crawlee-python | Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files… | Python | 9,542 | 809 | Apache-2.0 | Sep 21 | Sep 21 |
| firecrawl/firecrawl-mcp-server | Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients. | JavaScript | 7,601 | 905 | MIT | Oct 14 | Oct 14 |
| BruceDone/awesome-crawler | A collection of awesome web crawler,spider in different languages | 7,327 | 752 | MIT | Jun 16, 2024 | Oct 11 | |
| Anakin-Inc/anakin | Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up | Go | 4,403 | 181 | AGPL-3.0 | Aug 18 | Sep 21 |
| spider-rs/spider | Foundational low latency web data collecting in Rust | Rust | 2,769 | 312 | MIT | Oct 1 | Oct 11 |
| 0xMassi/webclaw | Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server. | Rust | 2,380 | 238 | AGPL-3.0 | Oct 14 | Oct 14 |
| scrapfly/scrapfly-scrapers | Scalable Python web scraping scripts for +40 popular domains | Python | 1,083 | 205 | Sep 18 | Sep 21 | |
| us/crw | Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl… | Rust | 1,059 | 84 | AGPL-3.0 | Sep 21 | Sep 21 |
| jpjacobpadilla/Stealth-Requests | Undetected web-scraping & seamless HTML parsing in Python! | Python | 564 | 57 | MIT | Sep 19 | Sep 21 |
| graphlit/graphlit-mcp-server | Model Context Protocol (MCP) Server for Graphlit Platform | TypeScript | 378 | 56 | MIT | Jan 12 | Oct 14 |
| umbrellaDocumentation/Web-Data-Scraper | Web Data Scraper - no-code internet scraping. Extract and export to CSV, Excel, JSON, Google Sheets, and Webhook. | JavaScript | 133 | 116 | Mar 17 | Sep 21 | |
| RayenMlayeh/jevcrawl | MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl. | Python | 80 | 7 | Apache-2.0 | Oct 15 | Oct 21 |
By language
- Python 6
- TypeScript 3
- Rust 3
Related topics
- Web scraping 15
- Crawler 11
- Scraper 8
- Scraping 7
- AI agents 6
- Data extraction 6
- LLM 6
- Markdown 6
- Web scraper 5
- Webscraping 5
- MCP 5
- Python 5
Read from GitHub by SpecStream between September 21, 2026 and October 21, 2026; each row shows the day its repository was read. Stars and pushes change every day, and the GitHub page is the current one. Forks and archived repositories are left out.