ai4bharat/MANGO
MANGO: A Corpus of Human Ratings for Speech MANGO (MUSHRA Assessment corpus using Native listeners and Guidelines to understand human Opinions at scale) is the first large-scale dataset designed for evaluating Text-to-Speech (TTS) systems in Indian languages. Key Features: 255,150 human ratings of TTS-generated outputs and ground-truth human speech. Covers two major Indian languages: Hindi & Tamil, and English. Based on the MUSHRA (Multiple Stimuli with Hidden… See the full description on the dataset page: https://huggingface.co/datasets/ai4bharat/MANGO.
MANGO: A Corpus of Human Ratings for Speech
MANGO (MUSHRA Assessment corpus using Native listeners and Guidelines to understand human Opinions at scale) is the first large-scale dataset designed for evaluating Text-to-Speech (TTS) systems in Indian languages.
Key Features:
- 255,150 human ratings of TTS-generated outputs and ground-truth human speech.
- Covers two major Indian languages: Hindi & Tamil, and English.
- Based on the MUSHRA (Multiple Stimuli with Hidden Reference and Anchor) test methodology.
- Ratings are provided on a continuous scale from 0 to 100, with discrete quality categories:
- 100-80: Excellent
- 80-60: Good
- 60-40: Fair
- 40-20: Poor
- 20-0: Bad
- Includes evaluations involving:
- MUSHRA: with explicitly mentioned high-quality references.
- MUSHRA-NMR: without explicitly mentioned high-quality references.
- MUSHRA-DG: with detailed guidelines across fine-grained dimensions
- MUSHRA-DG-NMR: with detailed guidelines across fine-grained dimensions and without explicitly mentioned high-quality references.
Available Splits
The dataset includes the following splits based on the test type and language.
Getting Started
import os
from datasets import load_dataset, Audio
from huggingface_hub import snapshot_download
def get_audio_paths(example):
for column in example.keys():
if "Audio" in column and isinstance(example[column], str):
example[column] = os.path.join(download_dir, example[column])
return example
# Download
repo_id = "ai4bharat/MANGO"
download_dir = snapshot_download(repo_id=repo_id, repo_type="dataset")
dataset = load_dataset(download_dir, split='Hindi__MUSHRA')
dataset = dataset.map(get_audio_paths)
# Cast audio columns
for column in dataset.column_names:
if 'Audio' in column:
dataset = dataset.cast_column(column, Audio())
# Explore
print(dataset)
'''
Dataset({
features: ['Rater_ID', 'FS2_Score', 'VITS_Score', 'ST2_Score', 'ANC_Score',
'REF_Score', 'FS2_Audio', 'VITS_Audio', 'ST2_Audio', 'ANC_Audio',
'REF_Audio'],
num_rows: 11300
})
'''
# # Print first instance
print(dataset[0])
'''
{'Rater_ID': 389, 'FS2_Score': 16, 'VITS_Score': 76, 'ST2_Score': 28,
'ANC_Score': 40, 'REF_Score': 100, 'FS2_Audio': {'path': ...
'''
# # Available Splits
dataset = load_dataset(download_dir, split=None)
print("Splits:", dataset.keys())
'''
Splits: dict_keys(['Tamil__MUSHRA_DG_NMR', 'Hindi__MUSHRA_DG',
'Hindi__MUSHRA_NMR', 'Hindi__MUSHRA_DG_NMR', 'Tamil__MUSHRA_NMR',
'Tamil__MUSHRA_DG', 'Tamil__MUSHRA', 'Hindi__MUSHRA',
'English__MUSHRA', 'English_MUSHRA_DG_NMR'])
'''
Why Use MANGO?
- Addresses limitations of traditional MOS and CMOS tests.
- Enables robust benchmarking for:
- Comparative analysis across multiple TTS systems.
- Evaluations in diverse linguistic contexts.
- Large-scale studies with multiple raters.
We believe this dataset is a valuable resource for researchers and practitioners working on speech synthesis evaluation, and related fields.
Quick Overview of TTS Systems
- Dataset: All Indian TTS systems were trained on the IndicTTS dataset. For English, we use models trained on LJSpeech.
- Models: FastSpeech2, VITS, StyleTTS2, XTTS
Citation
@article{ai4bharat2025rethinking,
title={Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluation},
author={Praveen Srinivasa Varadhan and Amogh Gulati and Ashwin Sankar and Srija Anand and Anirudh Gupta and Anirudh Mukherjee and Shiva Kumar Marepally and Ankur Bhatia and Saloni Jaju and Suvrat Bhooshan and Mitesh M. Khapra},
journal={Transactions on Machine Learning Research},
year={2025},
url={https://openreview.net/forum?id=oYmRiWCQ1W},
}
License
This dataset is released under the Creative Commons Attribution 4.0 International License (CC BY 4.0).