Data / Repositories / LLM evaluation
LLM evaluation repositories on GitHub: 11 ranked by stars
11 public repositories tagged LLM evaluation are on GitHub, with 90,618 stars between them; 11 had a push in the last 30 days. Most are written in Python (8), TypeScript (1), JavaScript (1). The most starred: langfuse/langfuse (35,820), comet-ml/opik (22,597), Arize-ai/phoenix (11,834).
SpecStream is the live web data engine for teams and agents. It reads open-source repositories on GitHub and keeps what it reads, and this page groups that reading by topic. Ask for the repositories you track by topic, language, license or activity, and new ones and changes arrive in your sheet, CRM, Slack or agent.
Every repository, most starred first
| Repository | What it is | Language | Stars | Forks | License | Last push | Read |
|---|---|---|---|---|---|---|---|
| langfuse/langfuse | Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform. | TypeScript | 35,820 | 3,987 | Oct 14 | Oct 14 | |
| comet-ml/opik | Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards. | Python | 22,597 | 1,858 | Apache-2.0 | Oct 14 | Oct 14 |
| Arize-ai/phoenix | AI Observability & Evaluation | Python | 11,834 | 1,203 | Oct 14 | Oct 14 | |
| Tencent/AI-Infra-Guard | A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation. | Python | 6,868 | 661 | Apache-2.0 | Oct 14 | Oct 14 |
| Q00/ouroboros | Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI… | Python | 6,219 | 627 | MIT | Oct 14 | Oct 14 |
| EvolvingLMMs-Lab/lmms-eval | One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks | Python | 4,448 | 674 | Oct 14 | Oct 14 | |
| jwuthri-cloud/Tracely-ai | Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable. | Python | 1,166 | 146 | MIT | Oct 19 | Oct 21 |
| v-modal/awesome-jev-tools | A curated list of tools built for Jev — TypeSafe AI's System One model for typed decisions. | 742 | 51 | Oct 9 | Oct 15 | ||
| OmniJev/awesome-jev-gallery | Awesome Jev: open-source models, projects, benchmarks, and independent evaluations for Jev and System One models. | JavaScript | 507 | 36 | Oct 15 | Oct 15 | |
| scaleapi/agentenv-framework | Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments tools, dynamism of the… | Python | 312 | 28 | Apache-2.0 | Oct 15 | Oct 15 |
| openlayer-ai/jevals | Agent evals and guardrails as Jev decisions: one request per trace, a fraction of a cent, fast enough for the agent loop. Runs locally with Kev or Laya. | Python | 105 | 11 | MIT | Oct 15 | Oct 15 |
By language
- Python 8
Related topics
- LLM 6
- Llmops 4
- LangChain 3
- OpenAI 3
- Prompt engineering 3
- Agents 3
- Agentic AI 3
- AI agents 3
- MCP 3
- Jev 3
- Large language models 2
- Observability 2
Read from GitHub by SpecStream between October 14, 2026 and October 21, 2026; each row shows the day its repository was read. Stars and pushes change every day, and the GitHub page is the current one. Forks and archived repositories are left out.