replication
replicationbenchReplicationBench
ReplicationBench
arXiv: ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?
GitHub: https://github.com/Christine8888/replicationbench-release
Dataset Description
The ReplicationBench dataset contains 111 astrophysics research replication tasks, spanning complete replications of 20 research papers. The dataset includes:
Original and masked manuscript text
Metadata (title, abstract, publication info, etc.)
Pointers to datasets and dataset access… See the full description on the dataset page: https://huggingface.co/datasets/ChristineYe8/ReplicationBench.jmh-grpo-replication-data
JMH-Bench GRPO replication data
Data for replicating the BSc thesis Reinforcement Learning with Verifiable Execution Rewards
for Synthesizing Regression-Detecting Microbenchmarks. The code is in the replication repository
RL4JMH-thesis-submission. The trained models are
run 9 (evaluated in the
thesis) and run 8.
Path
Contents
classpaths/
Mutant-bearing jars and dependency jars of the 246 training libraries, as used by GRPO run 9 (1,764 jars, 1.2 GB)… See the full description on the dataset page: https://huggingface.co/datasets/bookxd/jmh-grpo-replication-data.replicationbench
ReplicationBench Dataset
Dataset Description
A benchmark to evaluate AI agents in astrophysics research through replicating existing research papers.
Dataset Structure
Data Splits
ReplicationBench (source: epxert): Core expert-written benchmark
ReplicationBench-Plus (source: showyourwork): Extension dataset generated through hybrid LLM-expert system
Data Configurations
Each split contains two types of data:
metadata: Paper metadata and… See the full description on the dataset page: https://huggingface.co/datasets/replicationbench-submission/replicationbench.oai-hf-incident-replication
OpenAI–Hugging Face Incident Replication
Raw agent transcripts behind the write-up
"OpenAI–Hugging Face: A Reproduction and Lessons for Alignment".
They capture attempts to elicit the four pivotal misaligned behaviours from the
May–July 2026 incident that Hugging Face described here:
Inappropriate writes to shared infrastructure (a mock model registry).
Requesting help from other agents to get around a blocker.
Sharing solutions/exploits with peers, including an… See the full description on the dataset page: https://huggingface.co/datasets/stewy33/oai-hf-incident-replication.dpst-replication
DPST Replication
Replication package for the EMNLP 2025 paper:
Leveraging Semantic Triples for Private Document Generation with Local Differential Privacy GuaranteesStephen Meisenbacher, Maulik Chevli, Florian Mattheshttps://aclanthology.org/2025.emnlp-main.455/
All credit goes to the original authors. This repository provides a replication environment for running the method, based on the official released code: https://github.com/sjmeis/DPST.… See the full description on the dataset page: https://huggingface.co/datasets/weijunl/dpst-replication.