Evals and observability

τ-Bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

2,419 stars+41 this week

Open-source benchmark for browser AI agents on daily tasks.

1,240 stars+51 this week

Evaluate and improve models and agents using environments

1,325 stars+13 this week

An ops AI Agent that understands your infrastructure, finds the root cause, and fixes it — right from Slack, Telegram, Lark or DingTalk.

1,268 stars+21 this week

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

2,256 stars+17 this week

ANOLISA (Agentic Nexus Operating Layer & Interface System Architecture) | Agentic OS with runtime, security, observability, and Tokenless response compression for lower token usage and cost.

662 stars+7 this week

ARTEMIS turns natural-language instructions into reliable Android automation. It automates end-to-end workflows, captures logs, and integrates seamlessly with AI coding assistants such as Antigravity, Codex, and Claude Code. It also achieves 99%+ success rate on AndroidWorld Benchmark.

17,055 stars+790 this week

Hire Google's Antigravity CLI (agy) as a fast Gemini staffer for Claude Code and OpenAI Codex.

834 stars+23 this week

Self-hosted control plane & usage observability console for CLIProxyAPI (CPA) — AI providers, client keys, streaming telemetry, and model cost snapshots.

348 stars+48 this week

OpenTelemetry Instrumentation for AI Observability

1,309 stars+9 this week

lightweight system-level observability for AI Agents

779 stars+10 this week

The platform for continuously improving AI agents.

754 stars+8 this week

DevOps cockpit in one window: terminal with local AI, Kubernetes, SSH, databases, IDE with git, alerts, notes and KeePass. Free, open source, Linux/Windows/macOS. Рабочее место DevOps-инженера в одном окне.

219 stars+5 this week

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

23,276 stars+147 this week

Open-source AI text detector and ChatGPT detector: 23 engines, one calibrated score, every number measured on a public benchmark. Self-hosted, runs on CPU.

103 stars+18 this week

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

28,727 stars+101 this week

JevBench by Benchmark Heaven: benchmark and leaderboard for AI decision models, measuring typed decision accuracy, calibration, latency and cost.

338 stars+13 this week

The fastest path to AI-powered full stack observability, even for lean teams.

81,316 stars+90 this week

End-to-end AI short-video production pipeline. FastAPI orchestration + Spring Boot gateway with multi-model failover, circuit breaker, metering, and full-stack observability. AI quality gating: prompt anchoring, CLIP consistency, AV sync auto-rescue.

807 stars+26 this week

A comprehensive evaluation framework for AI agents and LLM applications.

252 stars+2 this week

The open-source observability database. One columnar engine for metrics, logs, and traces, on object storage.

6,797 stars+17 this week

Local observability for AI coding agents. Agenttrail Map shows your project as a live map of what each agent is working on, in the browser or natively in Herdr.

995 stars+17 this week

The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.

240 stars+9 this week

Build, Manage and Deploy AI/ML Systems

10,341 stars+7 this week

Evaluation and Tracking for LLM Experiments and AI Agents

3,631 stars+9 this week

IdeaTaste - A benchmark for research taste: let LLM agents propose research ideas, then check how many of them the research area actually went on to do.

31 stars+29 this week

Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable.

946 stars-30 this week

Benchmark and fix LLM hallucinations when extracting figures from SEC 10-K filings

24 stars0 this week

DecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.

154 stars+2 this week

DR Pixel: Page-Native Deep Research with Vision-Language Models — benchmark, evaluation harness and training code

42 stars+41 this week

ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.

5,638 stars+4 this week

🐢 Open-Source Evaluation & Testing library for LLM Agents

5,946 stars+7 this week

An AI-powered next-generation open source real-time observability system.

7,461 stars+9 this week

Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.

5,019 stars+16 this week

A Python library for anomaly detection across tabular, time series, graph, text, image, and audio data. 60+ detectors, benchmark-backed ADEngine orchestration, and an agentic workflow for AI agents.

10,081 stars+12 this week

Run Inspect AI evals in the cloud

105 stars+3 this week

Open-source observability for your GenAI or LLM application, based on OpenTelemetry

7,527 stars+15 this week

Agent evals and guardrails as Jev decisions: one request per trace, a fraction of a cent, fast enough for the agent loop. Runs locally with Kev or Laya.

117 stars+6 this week

Run your real agent with and without your skills, MCPs, and rules. See which ones actually help, and what they cost in tokens. Supports Claude Code, Codex, Pi, and Hermes.

229 stars+4 this week

OrcaReplay — Time travel for AI agents. Record, replay, fork, and debug any agent run with any model. Built by the OrcaRouter.ai team.

440 stars+147 this week

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

6,935 stars+8 this week

Hands-on, framework-free Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set — model APIs, structured output, tool calling, RAG, evals-as-the-spine, agents (loop from scratch, tool design, guardrails, MCP, Skills), fine-tuning vs LoRA, prompt-injection/security, LLMOps, and customer craft. Runs on the free Groq API.

741 stars+17 this week

eBPF-powered network observability for Kubernetes. Indexes L4/L7 traffic with full K8s context, decrypts TLS without keys. Queryable by AI agents via MCP and humans via dashboard.

12,133 stars+8 this week

RL environments + evals for AI agents. Define once, train anything.

324 stars+2 this week

AI+金融（量化）：1.多因子股票量化框架开源教程 2.学界和业界的经典资料收录 3.AI + 金融的相关工作，包括LLM, Agent, benchmark(evaluation), etc.

1,873 stars+13 this week

OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.

7,577 stars+20 this week

Evidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.

8,050 stars+22 this week

Open-source status page, incident response and uptime monitoring

9,254 stars+17 this week

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

2,625 stars+3 this week

OpenTelemetry plugins for every coding agent, in the Agent Plugins format — a marketplace that runs itself: open an issue, the repository validates, reviews, lists and follows your plugin.

118 stars+10 this week

agentevals is a framework-agnostic evaluations solution based on OpenTelemetry traces

169 stars+1 this week

[Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.

269 stars0 this week

Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

1,299 stars-11 this week

Instant Kubernetes-Native Application Observability

6,557 stars+7 this week

Instill trust in your agent evals.

317 stars+1 this week

The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.

1,073 stars0 this week

The robust European language model benchmark.

204 stars+1 this week

Minecraft-style voxel benchmark for comparing AI models (Arena + Sandbox)

343 stars0 this week

Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, trace, persist, and execute on your own infrastructure.

2,596 stars+6 this week

eBPF Observability - Distributed Tracing and Profiling

4,319 stars+7 this week

Open Source AI Tracing Framework built on Opentelemetry for AI Applications and Frameworks

266 stars0 this week

See your agent think. Zero-config observability & governance for 30 AI agent runtimes: Claude Code, OpenAI Codex, Hermes, OpenClaw & 26 more. Live token costs, sessions, tool calls, crons.

436 stars0 this week

Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developers.

1,286 stars+1 this week

Open-source operational advisor for ClickHouse — real-time monitoring plus AI-driven index/partition/materialized-view recommendations.

303 stars0 this week

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

6,273 stars+21 this week

A Benchmark for Evaluating Agent Self-Evolution on Real Business Tasks

121 stars+2 this week

🦞 Official plugin for OpenClaw that exports agent traces to Opik. See and monitor agent behaviour, cost, tokens, errors and more.

724 stars0 this week

Local-first observability your agent reads back. TMA1 records every LLM call, then routes what it sees into the agent's next turn via hooks and MCP.

124 stars0 this week

Claude Code skill: hand bulk judgment calls to Jev. 86% fewer Claude tokens on a 12-task benchmark, up to 20x faster. One-line npx install.

142 stars0 this week

The IDE that understands AI agents. Run parallel agents on isolated branches with full observability. Diffs, PRs, CI, usage dashboards — one workspace, zero context loss.

169 stars+1 this week

LLM evaluation framework for Elixir: evaluate and test LLM outputs, detect hallucinations, measure response quality

119 stars0 this week

ClearML Agent - MLOps/LLMOps made easy. MLOps/LLMOps scheduler & orchestration solution

316 stars+1 this week

High-performance LLM evaluation framework with parallel API calls — up to 17× faster than sequential tools. Supports box, math, and logit-based evaluation.

118 stars+2 this week

AgenticVBench: Can AI Agents Complete Real-World Post-Production Tasks?

112 stars0 this week

Full-market stock discovery for AI agents: A-shares plus US/HK/JP/KR/TW/IN, auditable YAML strategies, optional LLM ranking, sector-diversified picks, and true T+N evaluation vs. benchmark. AI选股

428 stars+49 this week

ML models + benchmark for tabular data classification and regression

403 stars0 this week

mloda.ai - Open Data Access for AI and ML. Plugin-based. Traceable. Framework-agnostic.

100 stars0 this week

AI agent-native observability and debugging for Django.

180 stars0 this week

Find your agents errors be fore your real users do

235 stars+1 this week

An evaluation framework for machine learning models simulating high-throughput materials discovery.

259 stars+1 this week

Sister project to OpenLLMetry, but in Typescript. Open-source observability for your LLM application, based on OpenTelemetry

412 stars+1 this week

High-scale LLM gateway, written in Rust. OpenTelemetry-based observability included

233 stars0 this week

Benchmark of federated learning. Dedicated to the community. 🤗

695 stars+3 this week

Open-source observability tool that uses AI agents to self-heal your software

1,479 stars+6 this week

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

1,229 stars0 this week

🕹️ Open-source, developer-first LLMOps platform designed to streamline prompt design, version management, instant delivery, collaboration, troubleshooting, observability and more.

3,279 stars+3 this week

🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the world's largest catalog of tools and data for end-to-end AI benchmarking

215 stars0 this week

postgresai – Postgres observability built for humans and AI agents

140 stars0 this week

SecCodeBench is a benchmark suite focusing on evaluating the security of code generated by large language models (LLMs).

138 stars+2 this week

🦞 ResearchClawBench: Evaluating AI Agents for Automated Research from Re-Discovery to New-Discovery

277 stars+2 this week

Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.

194 stars+1 this week

React components for visualizing traces from AI agents

407 stars+3 this week

Evaluation framework for oncology foundation models (FMs)

175 stars0 this week

ClawProBench is a live-first benchmark harness for evaluating LLM agents in the OpenClaw runtime with deterministic grading and repeated-trial reliability.

829 stars+1 this week

MLPerf® Storage Benchmark Suite

208 stars+1 this week

LabTether

126 stars-12 this week

GPU Profiler for AI agents

276 stars0 this week

Benchmark for automated failure attributions in agentic systems (🏆 ICML 2025 Spotlight)

396 stars+1 this week

Benchmarking models as runtime Controllers for Loop Engineering

154 stars+1 this week

A banchmark list for evaluation of large language models.

174 stars+1 this week

Self-hosted, enterprise-ready AI workflow orchestration with event sourcing, CQRS, fault tolerance, and observability. Single-binary control plane, Python + Go SDKs, embedded Postgres + NATS for zero-config dev.

161 stars0 this week

This repo contains evaluation code for the paper "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI"

600 stars+1 this week

An I/O benchmark for deep Learning applications

113 stars0 this week

Deliver safe & effective language models

561 stars0 this week

🧪Yet Another ICU Benchmark: a holistic framework for the standardization of clinical prediction model experiments. Provide custom datasets, cohorts, prediction tasks, endpoints, preprocessing, and models. Paper: https://arxiv.org/abs/2306.05109

112 stars+1 this week

Benchmark of Apple MLX operations on all Apple Silicon chips (GPU, CPU) + MPS and CUDA.

241 stars+1 this week

[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.

635 stars0 this week

A benchmark for evaluating LLM × harness performance.

127 stars+1 this week

Automated system for LLM evaluation via agents. Doc as below:

165 stars+1 this week

ZKE（Z Kubernetes Engine）：AI 原生的 Kubernetes 云操作环境，桌面式多集群控制台加受控 AIOps Agent，基于 Server + Agent 与 QUIC/mTLS，适用于私有云、混合云及边缘环境

107 stars0 this week

[ACM MM 2026] Code for paper "Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models".

116 stars0 this week

MaxKB4j is an open-source LLMOps platform for LLM workflow applications and RAG developed based on the Java language. The project mainly draws on MaxKB, Dify and FastGPT, and combines the advantages of the two into one project. It is redesigned and developed using the high-performance, high-stability and secure reliable JAVA language.

139 stars0 this week

Claude Code Agent Monitoring & Observability on VSCode

118 stars+1 this week

Benchmark AI Agents on Enterprise Workflows

111 stars+5 this week

Same model, different wrapper: a from-scratch benchmark comparing coding-agent harnesses (codex, pi, opencode, cursor, devin) and open models on correctness, speed, and token cost

137 stars0 this week

LLM Evaluation Framework for Rails apps to be used with production data.

148 stars+1 this week

Language AI Engineering Lab, a place where you can deeply understand and build modern Language AI systems, from fundamentals to production.

129 stars0 this week

ChinaTravel: A Real-World Benchmark for Language Agents in Chinese Travel Planning

138 stars+1 this week

Benchmark CLI for comparing AI coding agents on TypeScript workloads.

243 stars0 this week

Self-hosted AI SRE for Kubernetes — zero-instrumentation eBPF observability plus a copilot that fixes issues through guardrailed, self-verifying actions. BYO-LLM, air-gapped capable.

113 stars-1 this week

Vision Document Retrieval (ViDoRe): Benchmark. Evaluation code for the ColPali paper.

278 stars0 this week

🌟 [NeurIPS '25 Spotlight] Fair and transparent benchmark of machine learning interatomic potentials (MLIPs), beyond basic error metrics https://openreview.net/forum?id=SAT0KPA5UO

109 stars0 this week

Local-first, auditable Python code agent. Ships its own 30-task hidden-test benchmark plus SWE-bench Verified scored by the official harness -- every number reproducible, negative results included.

151 stars0 this week

WebLINX is a benchmark for building web navigation agents with conversational capabilities

165 stars+1 this week

⚡ Unit tests for AI. Test prompts, compare models, save money.

108 stars0 this week

OpenSource Production ready Customer service with built in Evals and monitoring

1,547 stars0 this week

Observability platform for Digital Employee, providing real-time tracing, session insights, and cost analysis for multi-agent workflows

136 stars0 this week

SRA-Bench and SR-Agents: a benchmark and toolkit for skill-retrieval-augmented LLM agents.

110 stars0 this week

Modular, open source LLMOps stack that separates concerns: LiteLLM unifies LLM APIs, manages routing and cost controls, and ensures high-availability, while Langfuse focuses on detailed observability, prompt versioning, and performance evaluations.

155 stars+2 this week

Not stated

102 stars0 this week