Datasetpublic

foksly/orbit-mt-eval

ORBIT-MT-Eval: Expert Annotations and Prompts for Machine Translation Meta-Evaluation ORBIT-MT-Eval is an English-to-Russian reference dataset for machine translation meta-evaluation: comparing evaluator outputs against professional expert annotations. Each of the 600 source–translation segments has three independent expert annotations following the RATE protocol. In Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators, two annotations serve as gold… See the full description on the dataset page: https://huggingface.co/datasets/foksly/orbit-mt-eval.

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes22downloads

Dataset Card

ORBIT-MT-Eval: Expert Annotations and Prompts for Machine Translation Meta-Evaluation

ORBIT-MT-Eval is an English-to-Russian reference dataset for machine translation meta-evaluation: comparing evaluator outputs against professional expert annotations. Each of the 600 source–translation segments has three independent expert annotations following the RATE protocol. In Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators, two annotations serve as gold references and the third is scored against each reference and averaged to produce the Human_E agreement baseline. The release also includes the prompts used by ORBIT and the locally maintained prompting baselines. GEMBA and unified-mqm-boosted-v5 are linked to their official upstream implementations.

Dataset at a glance

Property	Value
Language direction	English → Russian
Source segments	300
Source–translation segments	600 (two translations for each source)
Professional expert annotators	35
Expert annotations	3 independent annotations per segment from the expert pool, for 1,800 records total
Meta-evaluation protocol	2 reference annotations + 1 Human_E agreement baseline
Error spans	9,771
Annotation scheme	RATE severity 1–5 with separate Accuracy and Fluency scores
Domains	Literary, news, social media, and speech

expert_1, expert_2, and expert_3 identify the three annotation slots within each segment. They are not person-level IDs and do not track individual annotators across the dataset.

Files

—`data/rate_annotations/test.jsonl`: the annotation records.
—`prompts/README.md`: readable prompt pages with copy-ready JSON messages objects.
—`prompts/prompts.jsonl`: the machine-readable prompt collection.

Load

python

from datasets import load_dataset

dataset = load_dataset("foksly/orbit-mt-eval", "rate_annotations", split="test")
print(dataset[0])

Paper results

Evaluator	R	P	F1	R4+	P4+	MQM
Human_E	46.8	44.3	45.5	65.3	65.3	75.3
Human_A-SbS	39.3	47.1	42.8	55.7	63.2	68.0
Human_A-PW	31.8	46.4	37.7	49.8	57.1	65.0
Human_WMT	12.3	61.9	20.6	26.6	62.8	42.7
GEMBA-MQM	24.6	43.0	31.3	45.0	51.0	58.7
GEMBA-ESA	24.9	41.3	31.0	44.6	45.7	57.3
MQM-AE	24.0	51.8	32.8	45.4	54.5	60.0
ESA-AE	27.0	54.8	36.2	49.5	58.6	53.7
UMB-v5	27.8	44.5	34.2	48.7	56.4	57.3
xCOMET-XXL	18.3	36.4	24.3	33.5	36.8	58.3
ORBIT-SC	51.5	46.2	48.7	74.7	63.5	67.0
ORBIT-MC	60.3	45.8	52.1	82.0	70.6	69.3

Values are on a 0–100 scale for the English→Russian evaluation set. R/P/F1 are micro-averaged span metrics excluding RATE severity 1, R4+/P4+ use severity ≥4, and MQM is pairwise ranking accuracy.

Prompting baselines and ORBIT-SC use GPT-5.4. ORBIT-MC fuses three evaluator backbones with an LLM fuser.

Citation

bibtex

@inproceedings{popov-etal-2026-beyond-prompts,
  title     = {Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators},
  author    = {Popov, Dmitry and Bokhyan, Roman and Enikeeva, Ekaterina and
               Mekhraliev, Artem and Vysotsky, Stepan and Karpachev, Nikolay},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026}
}

License

The original RATE annotations and prompt material are released under the Apache License 2.0. Source and translation material retain their applicable upstream terms.