lobehub / awesome-rsi

What it does

• AutoLab. Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks? - Expert-curated benchmark of 36 realistic long-horizon, closed-loop optimization tasks for evaluating frontier agents on automated research and engineerin
• MLS-Bench. A Holistic and Rigorous Assessment of AI Systems on Building Better AI - Benchmark of 140 tasks across 12 ML research domains measuring whether AI systems can invent generalizable and scalable ML methods. (arXiv 2026)
• SAEScientist-Bench. Can AI Agents Conduct Autonomous SAE Interpretability Research? - Gives an agent a target concept and a Gemma Scope dictionary of over 131K features to probe contrastively, and measures how close the isolated feature com
• MLE-bench. Measures end-to-end machine-learning engineering performance across 75 Kaggle competitions and is used to track model self-improvement capability. (ICLR 2025)
• PaperBench. Evaluates agents on replicating state-of-the-art AI research from paper descriptions. (ICML 2025)