Datasetpublic

AlgorithmicResearchGroup/aria-repo-benchmark

ARIA Repo Benchmark The ARIA Repo Benchmark is part of the ARIA benchmark suite, a collection of closed-book benchmarks probing the ML knowledge that frontier models have internalized during training. This dataset contains 58 curated research paper implementations with metadata for evaluating whether ML experiments described in papers can be reproduced. Dataset Summary Size: 58 entries Coverage: Computer Vision, NLP, Time Series, Graph, Bioinformatics Purpose:… See the full description on the dataset page: https://huggingface.co/datasets/AlgorithmicResearchGroup/aria-repo-benchmark.

sourceHugging Faceupdated 6mo agoView on Hugging Face

0likes13downloads

Dataset Card

ARIA Repo Benchmark

The ARIA Repo Benchmark is part of the ARIA benchmark suite, a collection of closed-book benchmarks probing the ML knowledge that frontier models have internalized during training. This dataset contains 58 curated research paper implementations with metadata for evaluating whether ML experiments described in papers can be reproduced.

Dataset Summary

—Size: 58 entries
—Coverage: Computer Vision, NLP, Time Series, Graph, Bioinformatics
—Purpose: Evaluate AI agents on their ability to locate, understand, and reproduce ML research experiments

Each entry links a research paper to its code repository, dataset, metrics, and compute requirements, along with verification of whether the experiment is reproducible on constrained hardware.

Dataset Structure

Key Fields

Field	Type	Description
`paper_title`	string	Title of the research paper
`paper_url`	string	ArXiv URL
`paper_date`	timestamp	Publication date
`paper_text`	string	Full paper text
`dataset`	string	Dataset used in the paper
`dataset_link`	string	Link to the dataset
`model_name`	string	Model name
`code_links`	list[string]	GitHub repository links
`metrics`	string	Performance metrics reported
`table_metrics`	list[string]	Detailed metrics from tables
`prompts`	string	Evaluation prompts
`modality`	string	Data modality (CV, NLP, Time Series, Graph, etc.)

Compute & Reproducibility Fields

Field	Type	Description
`compute_hours`	float64	Estimated training compute hours
`num_gpus`	int64	Number of GPUs required
`reasoning`	string	Reasoning about compute estimates
`trainable_single_gpu_8h`	string	Trainable on a single GPU in 8 hours
`verified`	string	Verification status
`time_and_compute_verification`	string	Compute verification notes
`link_to_colab_notebook`	string	Google Colab notebook link
`run_possible`	string	Whether the code runs successfully
`notes`	string	Additional notes

Usage

python

from datasets import load_dataset

ds = load_dataset("AlgorithmicResearchGroup/aria-repo-benchmark", split="train")

for entry in ds:
    print(f"{entry['paper_title']} - {entry['modality']} - Reproducible: {entry['run_possible']}")

Related Resources

Citation

bibtex

@misc{aria_repo_benchmark,
    title={ARIA Repo Benchmark},
    author={Algorithmic Research Group},
    year={2024},
    publisher={Hugging Face},
    url={https://huggingface.co/datasets/AlgorithmicResearchGroup/aria-repo-benchmark}
}