Data / Repositories / Crawler / Python
Python Crawler repositories on GitHub: 14 ranked by stars
14 public Python repositories tagged Crawler are on GitHub, with 318,837 stars between them; 5 had a push in the last 30 days. The most starred: D4Vinci/Scrapling (86,357), unclecode/crawl4ai (85,530), scrapy/scrapy (64,433).
SpecStream is the live web data engine for teams and agents. It reads open-source repositories on GitHub and keeps what it reads, and this page groups that reading by topic. Ask for the repositories you track by topic, language, license or activity, and new ones and changes arrive in your sheet, CRM, Slack or agent.
Every repository, most starred first
| Repository | What it is | Stars | Forks | License | Last push | Read |
|---|---|---|---|---|---|---|
| D4Vinci/Scrapling | An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! | 86,357 | 8,855 | BSD-3-Clause | Oct 8 | Oct 8 |
| unclecode/crawl4ai | Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. | 85,530 | 8,879 | Apache-2.0 | Oct 14 | Oct 14 |
| scrapy/scrapy | Scrapy, a fast high-level web crawling & scraping framework for Python. | 64,433 | 11,969 | BSD-3-Clause | Sep 21 | Sep 21 |
| ScrapeGraphAI/Scrapegraph-ai | Python scraper based on AI | 31,623 | 3,221 | MIT | Oct 6 | Oct 8 |
| Evil0ctal/Douyin_TikTok_Download_API | 抖音、TikTok 数据采集与无水印视频下载 API，自托管，支持 MCP 调用与 Docker 一键部署。| Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server, CLI and web console for… | 20,637 | 2,881 | Apache-2.0 | Oct 2 | Oct 14 |
| apify/crawlee-python | Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files… | 9,542 | 809 | Apache-2.0 | Sep 21 | Sep 21 |
| alirezamika/autoscraper | A Smart, Automatic, Fast and Lightweight Web Scraper for Python | 7,990 | 827 | MIT | Jul 29 | Sep 21 |
| adbar/trafilatura | Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML | 6,845 | 433 | Apache-2.0 | Sep 21 | Sep 21 |
| lorien/grab | Web Scraping Framework | 2,464 | 274 | MIT | Sep 19, 2025 | Sep 21 |
| howie6879/ruia | Async Python 3.6+ web scraping micro-framework based on asyncio | 1,737 | 185 | Apache-2.0 | Sep 17 | Sep 21 |
| scrapfly/scrapfly-scrapers | Scalable Python web scraping scripts for +40 popular domains | 1,083 | 205 | Sep 18 | Sep 21 | |
| oxylabs/Python-Web-Scraping-Tutorial | In this Python Web Scraping Tutorial, we will outline everything needed to get started with web scraping. We will begin with simple examples and move on to relatively more complex. | 305 | 35 | Sep 24, 2025 | Sep 21 | |
| nuhmanpk/WebScrapper | Powerful Telegram bot for web scraping and crawling. Fast, easy, and loved by thousands! | 211 | 116 | MIT | Mar 19 | Sep 21 |
| RayenMlayeh/jevcrawl | MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl. | 80 | 7 | Apache-2.0 | Oct 15 | Oct 21 |
Other languages
Related topics
- Scraping 20
- Web scraping 20
- Scraper 13
- Web crawler 11
- Spider 11
- Webscraping 10
- Crawling 10
- Python 10
- LLM 7
- Automation 7
- AI 6
- Data extraction 6
Read from GitHub by SpecStream between September 21, 2026 and October 21, 2026; each row shows the day its repository was read. Stars and pushes change every day, and the GitHub page is the current one. Forks and archived repositories are left out.