cxcscmu / IdeaTaste
IdeaTaste - A benchmark for research taste: let LLM agents propose research ideas, then check how many of them the research area actually went on to do.
Why this job: says: benchmark; topics: evaluation, benchmark
What it does
IdeaTaste is a benchmark for research taste. LLM agents read a literature corpus frozen at 2025 and write research proposals; those proposals are then scored against the papers the same communities published in 2026. Proposals and papers are compared both on the literature they cite and on the ideas they state, giving four metrics: citation matching, idea matching, citation diversity and idea diversity.
Get started
pip install -e ".[dev]"
python -m pytest
# runs on the small sample in tests/fixtures/data - Maintenance
- Active
- Last commit 7 days ago
- Stars
- 32
- +30 this week
- Licence
- MIT
- Allows business use
- Downloads for
- None listed
- No release files
- Latest version
- None yet
- No releases
- Built with
- Python
- 100% of the code
Is it looked after?
- Contributors
- 1
- Last 90 days
- Issues
- 0% closed
- 1 still open
- Pull requests
- 0 merged
- 0 waiting
- Releases
- None
- Never published
- No contributing guide
- No security policy
- No code of conduct
Similar repos
| Repo | Stars | Last commit | Licence | Downloads for |
|---|---|---|---|---|
| IBM / unitxt | 215 | 3 days ago | Apache-2.0 | No downloads |
| EuroEval / EuroEval | 204 | 4 days ago | MIT | No downloads |
| suyoumo / ClawProBench | 830 | 2 months ago | Apache-2.0 | No downloads |
More: languages, topics and project record
Languages
- Python 100%
Topics
- ai-scientists
- benchmark
- evaluation
- evaluation-framework
- evaluation-metrics
- idea-generation
- llm
- llm-agents
- machine-learning
- nlp
- research-agents
- scientific-discovery
Project record
- JobEvals and observability
- StatusNew to our list
- Commits8
- Forks2
- Watching1
- CreatedNov 1, 2026
- First seen by usNov 7, 2026
- CheckedNov 10, 2026