Skip to the page
Get started – free

Data / Repositories / Crawler

Crawler repositories on GitHub: 31 ranked by stars

31 public repositories tagged Crawler are on GitHub, with 584,076 stars between them; 9 had a push in the last 30 days. Most are written in Python (14), JavaScript (6), TypeScript (3). The most starred: firecrawl/firecrawl (191,485), D4Vinci/Scrapling (86,357), unclecode/crawl4ai (85,530).

SpecStream is the live web data engine for teams and agents. It reads open-source repositories on GitHub and keeps what it reads, and this page groups that reading by topic. Ask for the repositories you track by topic, language, license or activity, and new ones and changes arrive in your sheet, CRM, Slack or agent.

Get Crawler repositories

Every repository, most starred first

RepositoryWhat it isLanguageStarsForksLicenseLast pushRead
firecrawl/firecrawlSupercharge your AI agents with data from the web and beyond. Building the library for superintelligence.TypeScript191,48510,124AGPL-3.0Oct 14Oct 14
D4Vinci/ScraplingAn adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!Python86,3578,855BSD-3-ClauseOct 8Oct 8
unclecode/crawl4aiOpen-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.Python85,5308,879Apache-2.0Oct 14Oct 14
scrapy/scrapyScrapy, a fast high-level web crawling & scraping framework for Python.Python64,43311,969BSD-3-ClauseSep 21Sep 21
ScrapeGraphAI/Scrapegraph-aiPython scraper based on AIPython31,6233,221MITOct 6Oct 8
apify/crawleeCrawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML…TypeScript25,8601,677Apache-2.0Sep 21Sep 21
Evil0ctal/Douyin_TikTok_Download_API抖音、TikTok 数据采集与无水印视频下载 API，自托管，支持 MCP 调用与 Docker 一键部署。| Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server, CLI and web console for…Python20,6372,881Apache-2.0Oct 2Oct 14
code4craft/webmagicA scalable web crawler framework for Java.Java11,6764,100Apache-2.0Dec 20, 2025Sep 21
apify/crawlee-pythonCrawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files…Python9,542809Apache-2.0Sep 21Sep 21
lorien/awesome-web-scrapingList of libraries, tools and APIs for web scraping and data processing.JavaScript8,153948Sep 19Sep 21
alirezamika/autoscraperA Smart, Automatic, Fast and Lightweight Web Scraper for PythonPython7,990827MITJul 29Sep 21
BruceDone/awesome-crawlerA collection of awesome web crawler,spider in different languages7,327752MITJun 16, 2024Oct 11
adbar/trafilaturaPython & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XMLPython6,845433Apache-2.0Sep 21Sep 21
niespodd/browser-fingerprintingAnalysis of Bot Protection systems with available countermeasures . How to defeat anti-bot system and get around browser fingerprinting scripts when scraping the web?JavaScript5,140275Jul 27Sep 21
dotnetcore/DotnetSpiderDotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping frameworkC#4,1401,052MITApr 3Sep 21
geziyor/geziyorGeziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.Go2,778155MPL-2.0Jul 2Sep 21
spider-rs/spiderFoundational low latency web data collecting in RustRust2,769312MITOct 1Oct 11
lorien/grabWeb Scraping FrameworkPython2,464274MITSep 19, 2025Sep 21
howie6879/ruiaAsync Python 3.6+ web scraping micro-framework based on asyncioPython1,737185Apache-2.0Sep 17Sep 21
orangecoding/fredyFredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen…JavaScript1,573243Apache-2.0Oct 13Oct 14
elixir-crawly/crawlyCrawly, a high-level web crawling & scraping framework for Elixir.Elixir1,117125Apache-2.0Jul 16, 2025Sep 21
scrapfly/scrapfly-scrapersScalable Python web scraping scripts for +40 popular domainsPython1,083205Sep 18Sep 21
us/crwFast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl…Rust1,05984AGPL-3.0Sep 21Sep 21
MorvanZhou/easy-scraping-tutorialSimple but useful Python web scraping tutorial code.Jupyter Notebook822536MITApr 7, 2024Sep 21
speed/newcrawlerFree Web Scraping Tool with JavaJavaScript585112Nov 25, 2023Sep 21
zhuyingda/webstera reliable high-level web crawling & scraping framework for Node.js.JavaScript55952GPL-3.0Mar 12, 2025Sep 21
oxylabs/Python-Web-Scraping-TutorialIn this Python Web Scraping Tutorial, we will outline everything needed to get started with web scraping. We will begin with simple examples and move on to relatively more complex.Python30535Sep 24, 2025Sep 21
nuhmanpk/WebScrapperPowerful Telegram bot for web scraping and crawling. Fast, easy, and loved by thousands!Python211116MITMar 19Sep 21
umbrellaDocumentation/Web-Data-ScraperWeb Data Scraper - no-code internet scraping. Extract and export to CSV, Excel, JSON, Google Sheets, and Webhook.JavaScript133116Mar 17Sep 21
RayenMlayeh/jevcrawlMCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.Python807Apache-2.0Oct 15Oct 21
supadata-ai/mcpOfficial Supadata MCP Server - Adds powerful video & web scraping to Cursor, Claude and any other LLM clients.TypeScript6310MITOct 14Oct 14

By language

Related topics

Read from GitHub by SpecStream between September 21, 2026 and October 21, 2026; each row shows the day its repository was read. Stars and pushes change every day, and the GitHub page is the current one. Forks and archived repositories are left out.