Data / Repositories / LLM evaluation / Python
Python LLM evaluation repositories on GitHub: 8 ranked by stars
8 public Python repositories tagged LLM evaluation are on GitHub, with 53,549 stars between them; 8 had a push in the last 30 days. The most starred: comet-ml/opik (22,597), Arize-ai/phoenix (11,834), Tencent/AI-Infra-Guard (6,868).
SpecStream is the live web data engine for teams and agents. It reads open-source repositories on GitHub and keeps what it reads, and this page groups that reading by topic. Ask for the repositories you track by topic, language, license or activity, and new ones and changes arrive in your sheet, CRM, Slack or agent.
Every repository, most starred first
| Repository | What it is | Stars | Forks | License | Last push | Read |
|---|---|---|---|---|---|---|
| comet-ml/opik | Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards. | 22,597 | 1,858 | Apache-2.0 | Oct 14 | Oct 14 |
| Arize-ai/phoenix | AI Observability & Evaluation | 11,834 | 1,203 | Oct 14 | Oct 14 | |
| Tencent/AI-Infra-Guard | A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation. | 6,868 | 661 | Apache-2.0 | Oct 14 | Oct 14 |
| Q00/ouroboros | Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI… | 6,219 | 627 | MIT | Oct 14 | Oct 14 |
| EvolvingLMMs-Lab/lmms-eval | One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks | 4,448 | 674 | Oct 14 | Oct 14 | |
| jwuthri-cloud/Tracely-ai | Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable. | 1,166 | 146 | MIT | Oct 19 | Oct 21 |
| scaleapi/agentenv-framework | Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments tools, dynamism of the… | 312 | 28 | Apache-2.0 | Oct 15 | Oct 15 |
| openlayer-ai/jevals | Agent evals and guardrails as Jev decisions: one request per trace, a fraction of a cent, fast enough for the agent loop. Runs locally with Kev or Laya. | 105 | 11 | MIT | Oct 15 | Oct 15 |
Other languages
Related topics
- LLM 6
- Llmops 4
- LangChain 3
- OpenAI 3
- Prompt engineering 3
- Agents 3
- Agentic AI 3
- AI agents 3
- MCP 3
- Jev 3
- Large language models 2
- Observability 2
Every LLM evaluation repository
Read from GitHub by SpecStream between October 14, 2026 and October 21, 2026; each row shows the day its repository was read. Stars and pushes change every day, and the GitHub page is the current one. Forks and archived repositories are left out.