Wonder239/DEAF

DEAF

DEAF is a collection of audio data, aligned text metadata, and data-generation scripts accompanying the paper DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models.

This repository is organized as a Hugging Face dataset repository and contains the locally hosted resources used in the paper: BSC audio, SIC audio, paired text metadata, and the scripts used to generate the speech-related subsets.

Repository structure

Data overview

• —BSC.csv contains 84 rows with two columns: code and sentence.
• —Data/Audio/BSC/ contains 1,260 .wav files derived from the 84 BSC metadata rows across multiple acoustic conditions.
• —SIC.csv contains 82 rows with two columns: code and sentence.
• —Data/Audio/SIC/ contains 248 .wav files.

Intended use

DEAF is intended for research use, especially:

• —benchmarking acoustic faithfulness in audio language models,
• —studying robustness of speech generation and speech understanding systems,
• —analyzing how textual prompts map to generated or synthesized speech under different acoustic conditions.

Data fields

Both metadata files use the same schema:

• —code: sample identifier used to align metadata with audio filenames or prompt templates,
• —sentence: the text content associated with the audio sample.

Examples:

code,sentence
DKITCHEN_E01,"I'm cooking dinner in the kitchen, preparing food slowly while enjoying the quiet routine."
GDR_EX_F_01,"As a mother of three, I have learned to balance work, family, and personal responsibilities while always trying to set a good example for my children."

Audio format

• —Audio files are stored as .wav.
• —The speech-generation scripts indicate a workflow targeting 16 kHz WAV output for generated assets.
• —Users should verify any subset-specific preprocessing assumptions directly from the scripts before large-scale reuse.

Included code

The repository includes the scripts used to build parts of the dataset:

• —Code For Speech Generation/BSC/edgeTTS.py: speech synthesis for the BSC pipeline.
• —Code For Speech Generation/BSC/mp3_to_wav.py: conversion of generated and source audio into WAV format.
• —Code For Speech Generation/BSC/addNoise.py: noise augmentation and mixing for BSC samples.
• —Code For Speech Generation/SIC/SIC_audio_generation.py: end-to-end generation script for the SIC subset.

Not included

ESC data referenced in the paper are not hosted in this repository. Please obtain them from the source described in arXiv:2510.25054.

Citation

If you use this repository or the associated paper, please cite:

@misc{xiong2026deaf,
  title         = {DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models},
  author        = {Jiaqi Xiong and Yunjia Qi and Qi Cao and Yu Zheng and Yutong Zhang and Ziteng Wang and Ruofan Liao and Weisheng Xu and Sichen Liu},
  year          = {2026},
  eprint        = {2603.18048},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2603.18048}
}

License

This dataset is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Parts of the dataset are generated using text-to-speech (TTS) systems. In addition, environmental noise from the DEMAND dataset is incorporated into the audio samples. Users of this dataset must comply with the original licenses of all third-party resources.

The authors do not claim ownership over third-party components. All such components are redistributed in accordance with their respective licenses and are used for research purposes only.

Third-Party Data

This dataset incorporates environmental noise from the DEMAND dataset. If you use this dataset, please also cite:

@article{thiemann2013demand,
  title   = {The Diverse Environments Multi-Channel Acoustic Noise Database (DEMAND): A database of multichannel environmental noise recordings},
  author  = {Thiemann, Joachim and Ito, Nobutaka and Vincent, Emmanuel},
  journal = {The Journal of the Acoustical Society of America},
  year    = {2013},
  volume  = {133},
  number  = {5},
  pages   = {3591}
}