sudoping01/bam-asr-benchmark

Bambara ASR Benchmark

The first standardized evaluation set for Automatic Speech Recognition in Bambara (Bamanankan). One hour of studio-quality constitutional text, transcribed and validated by linguists from Mali's Direction Nationale de l'Éducation Non Formelle et des Langues Nationales (DNENF-LN).

This benchmark accompanies the paper "Where Are We at with Automatic Speech Recognition for the Bambara Language?" and the public leaderboard at MALIBA-AI/bambara-asr-leaderboard.

Purpose: Evaluation only. This dataset is a test set — it is not intended for training.

Overview

Why This Benchmark

Bambara ASR has seen rapid growth since 2022, but until now there was no common test set for comparing models. Researchers reported WER/CER on internal splits, making cross-model comparison unreliable.

This benchmark fills that gap. It is deliberately formal and domain-specific: the vocabulary is legal, the orthography is strict, and there is no code-switching. This makes it a stress test for domain robustness and out-of-vocabulary handling.

Because the acoustic conditions are near-optimal (studio recording, clean audio, professional speaker), the error rates reported here should be interpreted as approximate lower bounds — real-world performance on naturalistic Bambara speech will almost certainly be worse.

Vocabulary Isolation

The vocabulary in this dataset has very low overlap with existing Bambara ASR training corpora. To verify that, we have drawn two one-hour subsets from the two publicly available ASR dataset (at the time of compiling this report) by randomly selecting utterances and we observed the following word overlap figures:

So this benchmark features about 75% unique words when compared with equal sized samples of these two datasets, confirming how narrow the domain is. Words like yuruguyuruguli (disorder/embezzlement) and yamaruyasariya (ordinance/regulation) are central to the constitutional register but virtually absent from conversational Bambara.

Leaderboard

Full results and adjustable metric weights at the public leaderboard.

Dataset Construction

Source. The text and audio originate from the DNENF-LN's official Bambara translation of the Malian Constitution (as of July 2023, 191 articles).

Segmentation. Manual segmentation and forced alignment were performed using Audacity. The resulting segments range from 600ms to 46 seconds.

Transcription validation. An annotator reviewed every segment to align the transcription with the actual spoken audio, correcting cases where the speaker paraphrased or interpreted the written text rather than reading it verbatim. All words were verified against the DNENF-LN dictionary and Bamadaba.

Code-switching protocol. French-derived words already adopted into the official Bambara lexicon (e.g., politiki, afiriki, minisiri) were transcribed in standard Bambara spelling. No French-spelled words remain in the dataset.

Silent/music segments. 8 non-speech segments from the original recording were retained to test model robustness against hallucination on silence.

Known Limitations

• —Single speaker, single domain. One male voice reading constitutional text. No speaker or topic diversity.
• —Near-ideal acoustics. Studio recording, high SNR. Does not reflect real-world noise, phone-quality audio, or multi-speaker scenarios.
• —No code-switching. Urban Bambara frequently mixes French. This benchmark does not test that ability.
• —Normalization artifacts. Standard WER penalizes valid Bambara orthographic alternations (e.g., b'a vs. bɛ a, compound segmentation like yɛrɛmahɔrɔnya vs. yɛrɛma hɔrɔnya). A linguistically-aware normalization would yield different error rates.
• —1 hour is small. Consistent patterns across 37 models suggest the findings generalize, but a larger benchmark would strengthen conclusions.

Future versions will incorporate speaker diversity, domain variation, naturalistic speech, varied acoustic conditions, and code-switching.

Usage

from datasets import load_dataset

ds = load_dataset("MALIBA-AI/bambara-asr-benchmark", "eval", split="test")

Links

• —Leaderboard: huggingface.co/spaces/MALIBA-AI/bambara-asr-leaderboard
• —Code: github.com/MALIBA-AI/bambara-asr-leaderboard
• —Paper: Where Are We at with Automatic Speech Recognition for the Bambara Language?

Citation

@misc{BambaraASRBenchmark2025,
  title        = {Where Are We at with Automatic Speech Recognition for the Bambara Language?},
  author       = {Seydou Diallo and Yacouba Diarra and Mamadou K. Keita and Panga Azazia Kamat{\'e} and Adam Bouno Kampo and Aboubacar Ouattara},
  year         = {2025},
  howpublished = {Hugging Face Datasets},
  url          = {https://huggingface.co/datasets/MALIBA-AI/bambara-asr-benchmark}
}