Data / Repositories / Web scraping
Web scraping repositories on GitHub: 53 ranked by stars
53 public repositories tagged Web scraping are on GitHub, with 650,842 stars between them; 21 had a push in the last 30 days. Most are written in Python (28), TypeScript (7), JavaScript (7). The most starred: firecrawl/firecrawl (191,485), D4Vinci/Scrapling (86,357), unclecode/crawl4ai (85,530).
SpecStream is the live web data engine for teams and agents. It reads open-source repositories on GitHub and keeps what it reads, and this page groups that reading by topic. Ask for the repositories you track by topic, language, license or activity, and new ones and changes arrive in your sheet, CRM, Slack or agent.
Every repository, most starred first
| Repository | What it is | Language | Stars | Forks | License | Last push | Read |
|---|---|---|---|---|---|---|---|
| firecrawl/firecrawl | Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. | TypeScript | 191,485 | 10,124 | AGPL-3.0 | Oct 14 | Oct 14 |
| D4Vinci/Scrapling | An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! | Python | 86,357 | 8,855 | BSD-3-Clause | Oct 8 | Oct 8 |
| unclecode/crawl4ai | Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. | Python | 85,530 | 8,879 | Apache-2.0 | Oct 14 | Oct 14 |
| scrapy/scrapy | Scrapy, a fast high-level web crawling & scraping framework for Python. | Python | 64,433 | 11,969 | BSD-3-Clause | Sep 21 | Sep 21 |
| ScrapeGraphAI/Scrapegraph-ai | Python scraper based on AI | Python | 31,623 | 3,221 | MIT | Oct 6 | Oct 8 |
| apify/crawlee | Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML… | TypeScript | 25,860 | 1,677 | Apache-2.0 | Sep 21 | Sep 21 |
| browserbase/stagehand | The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more. | TypeScript | 25,666 | 1,756 | MIT | Oct 14 | Oct 14 |
| Evil0ctal/Douyin_TikTok_Download_API | 抖音、TikTok 数据采集与无水印视频下载 API，自托管，支持 MCP 调用与 Docker 一键部署。| Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server, CLI and web console for… | Python | 20,637 | 2,881 | Apache-2.0 | Oct 2 | Oct 14 |
| yusufkaraaslan/Skill_Seekers | Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection | Python | 15,146 | 1,552 | MIT | Sep 30 | Oct 14 |
| seleniumbase/SeleniumBase | Browser automation, web scraping, and testing. CDP Mode provides a Stealthy Chromium instance that bypasses bot-detection systems. Includes an MCP server. | Python | 13,031 | 1,588 | MIT | Sep 18 | Sep 21 |
| apify/crawlee-python | Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files… | Python | 9,542 | 809 | Apache-2.0 | Sep 21 | Sep 21 |
| lorien/awesome-web-scraping | List of libraries, tools and APIs for web scraping and data processing. | JavaScript | 8,153 | 948 | Sep 19 | Sep 21 | |
| alirezamika/autoscraper | A Smart, Automatic, Fast and Lightweight Web Scraper for Python | Python | 7,990 | 827 | MIT | Jul 29 | Sep 21 |
| cporter202/API-mega-list | This GitHub repo is a powerhouse collection of APIs you can start using immediately to build everything from simple automations to full-scale applications. One of the most… | JavaScript | 7,808 | 1,504 | Oct 12 | Oct 14 | |
| firecrawl/firecrawl-mcp-server | Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients. | JavaScript | 7,601 | 905 | MIT | Oct 14 | Oct 14 |
| go-rod/rod | A Chrome DevTools Protocol driver for web automation and scraping. | Go | 7,109 | 484 | MIT | Aug 11 | Sep 21 |
| adbar/trafilatura | Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML | Python | 6,845 | 433 | Apache-2.0 | Sep 21 | Sep 21 |
| Anakin-Inc/anakin | Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up | Go | 4,403 | 181 | AGPL-3.0 | Aug 18 | Sep 21 |
| spider-rs/spider | Foundational low latency web data collecting in Rust | Rust | 2,769 | 312 | MIT | Oct 1 | Oct 11 |
| feder-cr/invisible_playwright_mcp | Playwright MCP server undetected by anti-bots and captchas: AI agent browses the web on anti-detect stealth Firefox, Python, undetected browser automation, scraping, computer use. | Python | 2,752 | 447 | MIT | Oct 13 | Oct 15 |
| brightdata/brightdata-mcp | A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access. | JavaScript | 2,675 | 332 | MIT | Oct 14 | Oct 14 |
| lorien/grab | Web Scraping Framework | Python | 2,464 | 274 | MIT | Sep 19, 2025 | Sep 21 |
| 0xMassi/webclaw | Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server. | Rust | 2,380 | 238 | AGPL-3.0 | Oct 14 | Oct 14 |
| vibheksoni/stealth-browser-mcp | Stealth browser automation for AI agents over MCP and the Chrome DevTools Protocol: navigation, network hooks, DOM extraction, and pixel-accurate UI cloning. | Python | 2,223 | 302 | MIT | Oct 12 | Oct 14 |
| orangecoding/fredy | Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen… | JavaScript | 1,573 | 243 | Apache-2.0 | Oct 13 | Oct 14 |
| tidyverse/rvest | Simple web scraping for R | R | 1,521 | 349 | Sep 17 | Sep 21 | |
| tinyfish-io/agentql | AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly… | Python | 1,464 | 171 | MIT | Sep 18 | Sep 21 |
| roach-php/core | The complete web scraping toolkit for PHP. | PHP | 1,456 | 91 | Oct 27, 2025 | Sep 21 | |
| ProxyShard/ShardBrowser | Free, open-source anti-detect browser launcher for web scraping and multi-accounting. By the ProxyShard team. Engine-level fingerprint spoofing in Chromium 152 (WebGL / WebGPU /… | TypeScript | 1,107 | 126 | MIT | Sep 13 | Sep 21 |
| scrapfly/scrapfly-scrapers | Scalable Python web scraping scripts for +40 popular domains | Python | 1,083 | 205 | Sep 18 | Sep 21 | |
| us/crw | Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl… | Rust | 1,059 | 84 | AGPL-3.0 | Sep 21 | Sep 21 |
| runesleo/x-reader | Universal content reader MCP Server for 10+ platforms | Python | 967 | 94 | MIT | Oct 14 | Oct 14 |
| je-suis-tm/web-scraping | Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and… | Python | 911 | 190 | Apache-2.0 | Feb 1, 2022 | Sep 21 |
| ScrapeGraphAI/scrapecraft | AI-powered web scraping editor with visual workflow builder. Build, test & deploy web scrapers using natural language. Powered by ScrapeGraphAI & LangGraph. | Python | 699 | 112 | MIT | Dec 26, 2025 | Sep 21 |
| jpjacobpadilla/Stealth-Requests | Undetected web-scraping & seamless HTML parsing in Python! | Python | 564 | 57 | MIT | Sep 19 | Sep 21 |
| davidteather/everything-web-scraping | Learn everything web scraping with David Teather Codes on YouTube | HTML | 478 | 87 | GPL-3.0 | Jul 31, 2023 | Sep 21 |
| yusuzech/r-web-scraping-cheat-sheet | Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium. | R | 394 | 99 | MIT | Dec 20, 2022 | Sep 21 |
| lkuffo/web-scraping | Más de 50 ejemplos de web scraping utilizando: Requests | Scrapy | Selenium | LXML | BeautifulSoup | Python | 393 | 213 | GPL-3.0 | Oct 20, 2025 | Sep 21 |
| graphlit/graphlit-mcp-server | Model Context Protocol (MCP) Server for Graphlit Platform | TypeScript | 378 | 56 | MIT | Jan 12 | Oct 14 |
| walissonsilva/web-scraping-python | Repositório com o conteúdo (slides, exemplos, códigos) da série de vídeos no YouTube sobre Web Scraping com Python. | Python | 324 | 55 | Jul 13, 2021 | Sep 21 | |
| oxylabs/Python-Web-Scraping-Tutorial | In this Python Web Scraping Tutorial, we will outline everything needed to get started with web scraping. We will begin with simple examples and move on to relatively more complex. | Python | 305 | 35 | Sep 24, 2025 | Sep 21 | |
| Germey/AwesomeWebScraping | List of libraries, tools and APIs for web scraping and data processing. | Makefile | 265 | 47 | Jul 30 | Sep 21 | |
| justmarkham/trump-lies | Tutorial: Web scraping in Python with Beautiful Soup | Jupyter Notebook | 246 | 214 | Nov 18, 2018 | Sep 21 | |
| nuhmanpk/WebScrapper | Powerful Telegram bot for web scraping and crawling. Fast, easy, and loved by thousands! | Python | 211 | 116 | MIT | Mar 19 | Sep 21 |
| tinyfish-io/agentql-mcp | Model Context Protocol server that integrates AgentQL's data extraction capabilities. | TypeScript | 181 | 44 | MIT | Oct 8 | Oct 14 |
| chienbm98/foxprofile | Free, open-source anti-detect browser profile manager: per-profile fingerprint, cookies and proxy, Camoufox and Chrome engines, desktop app, REST API and MCP server for AI agents.… | Python | 151 | 70 | MIT | Oct 14 | Oct 21 |
| umbrellaDocumentation/Web-Data-Scraper | Web Data Scraper - no-code internet scraping. Extract and export to CSV, Excel, JSON, Google Sheets, and Webhook. | JavaScript | 133 | 116 | Mar 17 | Sep 21 | |
| codingforentrepreneurs/Web-Scraping | Learn how to leverage Python's amazing tools to scrape data from other websites. The end goal of this course is to scrape blogs to analyze trending keywords and phrases. We'll be… | Python | 117 | 65 | MIT | Dec 14, 2018 | Sep 21 |
| sadiuysal/crawl4ai-mcp-server | A lightweight Model Context Protocol (MCP) server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents. Similar to Firecrawl's API but self-hosted… | Python | 115 | 22 | MIT | Feb 6 | Oct 14 |
| Sharan-Kumar-R/Custom-MCP-Server | MCP server for scraping LinkedIn, Facebook, Instagram profiles and Google search. | Python | 98 | 76 | Jun 27, 2025 | Oct 14 | |
| idan-rubin/browserclaw | The AI-native browser automation library. Snapshot + ref targeting — born from OpenClaw, built for agents, by agents. | TypeScript | 80 | 7 | MIT | Oct 10 | Oct 14 |
| RayenMlayeh/jevcrawl | MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl. | Python | 80 | 7 | Apache-2.0 | Oct 15 | Oct 21 |
| monocrawl/monocrawl-mcp | Monocrawl MCP server and agent skill: live public data from 70+ platforms for AI agents | JavaScript | 7 | 0 | MIT | Sep 29 | Oct 14 |
By language
- Python 28
- TypeScript 7
- JavaScript 7
- Rust 3
Related topics
- Python 23
- Crawler 20
- Scraping 19
- Webscraping 17
- AI agents 15
- Web crawler 15
- MCP server 15
- MCP 14
- Playwright 13
- Scraper 12
- AI 11
- Automation 11
Read from GitHub by SpecStream between September 21, 2026 and October 21, 2026; each row shows the day its repository was read. Stars and pushes change every day, and the GitHub page is the current one. Forks and archived repositories are left out.