jumelet/multiblimp

MultiBLiMP

MultiBLiMP is a massively Multilingual Benchmark for Linguistic Minimal Pairs. The dataset is composed of synthetic pairs generated using Universal Dependencies and UniMorph.

The paper can be found here.

We split the data set by language: each language consists of a single .tsv file. The rows contain many attributes for a particular pair, most important are the sen and wrong_sen fields, which we use for evaluating the language models.

Using MultiBLiMP

To download the final datasets for a given language, you can usethe example code. Make sure to use the ISO 639-3 code, e.g. eng for English (see table below).

from huggingface_hub import hf_hub_download

hf_hub_download(repo_id="jumelet/multiblimp", filename=f"eng/data.tsv", repo_type="dataset", local_dir='hf_cache/')

To run MultiBLiMP on a model, follow the instructions in this repo. Example evaluation code:

python scripts/lm_eval/eval_model.py 
        --model meta-llama/Meta-Llama-3-8B 
        --data_dir hf_cache/eng/ 
        --src_dir multiblimp 
        --results_dir multiblimp_results/ 
        --cache_dir hf_cache

Languages

This table contains the languages covered in MultiBLiMP and the number of items for each language.

Citation Information

@misc{jumelet2025multiblimp10massivelymultilingual,
      title={MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs}, 
      author={Jaap Jumelet and Leonie Weissweiler and Arianna Bisazza},
      year={2025},
      eprint={2504.02768},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2504.02768}, 
}