Data / Repositories / Crawling
Crawling repositories on GitHub: 15 ranked by stars
15 public repositories tagged Crawling are on GitHub, with 215,111 stars between them; 2 had a push in the last 30 days. Most are written in Python (6), TypeScript (2), JavaScript (2). The most starred: D4Vinci/Scrapling (86,357), scrapy/scrapy (64,433), apify/crawlee (25,860).
SpecStream is the live web data engine for teams and agents. It reads open-source repositories on GitHub and keeps what it reads, and this page groups that reading by topic. Ask for the repositories you track by topic, language, license or activity, and new ones and changes arrive in your sheet, CRM, Slack or agent.
Every repository, most starred first
| Repository | What it is | Language | Stars | Forks | License | Last push | Read |
|---|---|---|---|---|---|---|---|
| D4Vinci/Scrapling | An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! | Python | 86,357 | 8,855 | BSD-3-Clause | Oct 8 | Oct 8 |
| scrapy/scrapy | Scrapy, a fast high-level web crawling & scraping framework for Python. | Python | 64,433 | 11,969 | BSD-3-Clause | Sep 21 | Sep 21 |
| apify/crawlee | Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML… | TypeScript | 25,860 | 1,677 | Apache-2.0 | Sep 21 | Sep 21 |
| apify/crawlee-python | Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files… | Python | 9,542 | 809 | Apache-2.0 | Sep 21 | Sep 21 |
| lorien/awesome-web-scraping | List of libraries, tools and APIs for web scraping and data processing. | JavaScript | 8,153 | 948 | Sep 19 | Sep 21 | |
| go-rod/rod | A Chrome DevTools Protocol driver for web automation and scraping. | Go | 7,109 | 484 | MIT | Aug 11 | Sep 21 |
| exa-labs/exa-mcp-server | Exa MCP for web search and web crawling! | TypeScript | 5,127 | 403 | MIT | Oct 7 | Oct 14 |
| lorien/grab | Web Scraping Framework | Python | 2,464 | 274 | MIT | Sep 19, 2025 | Sep 21 |
| roach-php/core | The complete web scraping toolkit for PHP. | PHP | 1,456 | 91 | Oct 27, 2025 | Sep 21 | |
| elixir-crawly/crawly | Crawly, a high-level web crawling & scraping framework for Elixir. | Elixir | 1,117 | 125 | Apache-2.0 | Jul 16, 2025 | Sep 21 |
| scrapfly/scrapfly-scrapers | Scalable Python web scraping scripts for +40 popular domains | Python | 1,083 | 205 | Sep 18 | Sep 21 | |
| MorvanZhou/easy-scraping-tutorial | Simple but useful Python web scraping tutorial code. | Jupyter Notebook | 822 | 536 | MIT | Apr 7, 2024 | Sep 21 |
| slotix/dataflowkit | Extract structured data from web sites. Web sites scraping. | Go | 713 | 82 | BSD-3-Clause | Mar 7, 2023 | Sep 21 |
| zhuyingda/webster | a reliable high-level web crawling & scraping framework for Node.js. | JavaScript | 559 | 52 | GPL-3.0 | Mar 12, 2025 | Sep 21 |
| alephdata/memorious | Lightweight web scraping toolkit for documents and structured data. | Python | 316 | 64 | MIT | May 20 | Sep 21 |
By language
- Python 6
Related topics
- Scraping 11
- Crawler 10
- Web scraping 9
- Scraper 6
- Automation 5
- Python 5
- Spider 5
- Headless 4
- Playwright 3
- Webscraping 3
- Headless chrome 3
- Web crawler 3
Read from GitHub by SpecStream between September 21, 2026 and October 14, 2026; each row shows the day its repository was read. Stars and pushes change every day, and the GitHub page is the current one. Forks and archived repositories are left out.