datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Multitask-National-Speech-Corpus-v1-extendaudio_achilles_heelVideos-Dataset-For-LLMs-RAG-That-Require-Audio-Vidoes-And-Text
Dataset Overview
A collection of 27 domains (“topics”) and 3100 question-answer pair.
Each topic comes with average 117 QA pairs.Every QA entry comes with:
references: one or more source files the answer is extracted from
time with each reference comes the starting and ending time the answer is extracted from the reference
video_files: the video files where the answer can be found
(future) video title & description from metadata.csv
File structure
You-Are-Here!/… See the full description on the dataset page: https://huggingface.co/datasets/elmoghany/Videos-Dataset-For-LLMs-RAG-That-Require-Audio-Vidoes-And-Text.audio-llm-trainmusicai-background-music-audio-llm-benchmark
Does Background Music Matter to Speech in Pre-trained Language Models
The completed September 2026 study covers 8 model families, 55 instrumental recordings, and 10 evaluation settings. It studies how adding background music to the same spoken question changes model responses.
Latest release and artifact guide
Technical report PDF
Complete LaTeX project
LaTeX GitHub repository
Matrices, figures, and supporting data
Regenerated speech and mixtures: 550 archives / 250,800… See the full description on the dataset page: https://huggingface.co/datasets/Elfsong/musicai-background-music-audio-llm-benchmark.aishell_1_zh_test@inproceedings{bu2017aishell,
title={Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline},
author={Bu, Hui and Du, Jiayu and Na, Xingyu and Wu, Bengu and Zheng, Hao},
booktitle={2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech I/O systems and assessment (O-COCOSDA)},
pages={1--5},
year={2017},
organization={IEEE}
}
@article{wang2024audiobench,
title={AudioBench: A Universal… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/aishell_1_zh_test.llm_sam_audio_dataspoken_squad_testThis dataset is licensed under the terms of the CC-BY-SA-4.0 license.
https://github.com/Chia-Hsuan-Lee/Spoken-SQuAD/blob/master/LICENSE.md
Author: @michaellee886
@article{li2018spoken,
title={Spoken SQuAD: A study of mitigating the impact of speech recognition errors on listening comprehension},
author={Li, Chia-Hsuan and Wu, Szu-Lin and Liu, Chi-Liang and Lee, Hung-yi},
journal={arXiv preprint arXiv:1804.00320},
year={2018}
}
@article{wang2024audiobench,
title={AudioBench: A… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/spoken_squad_test.earnings22_test@article{del2022earnings,
title={Earnings-22: A practical benchmark for accents in the wild},
author={Del Rio, Miguel and Ha, Peter and McNamara, Quinten and Miller, Corey and Chandra, Shipra},
journal={arXiv preprint arXiv:2203.15591},
year={2022}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and Zou, Xunlong and Lin, Geyu and Sun, Shuo and Liu, Zhuohan and Zhang, Wenyu and Liu, Zhengyuan and Aw, AiTi… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/earnings22_test.gigaspeech2-test@article{yang2024gigaspeech,
title={GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement},
author={Yang, Yifan and Song, Zheshu and Zhuo, Jianheng and Cui, Mingyu and Li, Jinpeng and Yang, Bo and Du, Yexing and Ma, Ziyang and Liu, Xunying and Wang, Ziyuan and others},
journal={arXiv preprint arXiv:2406.11546},
year={2024}
}
@article{wang2024audiobench,
title={AudioBench: A Universal… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/gigaspeech2-test.gigaspeech_test@article{chen2021gigaspeech,
title={Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio},
author={Chen, Guoguo and Chai, Shuzhou and Wang, Guanbo and Du, Jiayu and Zhang, Wei-Qiang and Weng, Chao and Su, Dan and Povey, Daniel and Trmal, Jan and Zhang, Junbo and others},
journal={arXiv preprint arXiv:2106.06909},
year={2021}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/gigaspeech_test.details_Thytu__phi-2-audio-super
Dataset Card for Evaluation run of Thytu/phi-2-audio-super
Dataset automatically created during the evaluation run of model Thytu/phi-2-audio-super on the Open LLM Leaderboard.
The dataset is composed of 63 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 8 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard-old/details_Thytu__phi-2-audio-super.peoples_speech_test@article{galvez2021people,
title={The people's speech: A large-scale diverse english speech recognition dataset for commercial usage},
author={Galvez, Daniel and Diamos, Greg and Ciro, Juan and Cer{\'o}n, Juan Felipe and Achorn, Keith and Gopi, Anjali and Kanter, David and Lam, Maximilian and Mazumder, Mark and Reddi, Vijay Janapa},
journal={arXiv preprint arXiv:2111.09344},
year={2021}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/peoples_speech_test.audioLLM_judgemeld_emotion_test@article{poria2018meld,
title={Meld: A multimodal multi-party dataset for emotion recognition in conversations},
author={Poria, Soujanya and Hazarika, Devamanyu and Majumder, Navonil and Naik, Gautam and Cambria, Erik and Mihalcea, Rada},
journal={arXiv preprint arXiv:1810.02508},
year={2018}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and Zou, Xunlong and Lin, Geyu and Sun, Shuo and Liu, Zhuohan and… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/meld_emotion_test.librispeech_test_clean@inproceedings{panayotov2015librispeech,
title={Librispeech: an asr corpus based on public domain audio books},
author={Panayotov, Vassil and Chen, Guoguo and Povey, Daniel and Khudanpur, Sanjeev},
booktitle={2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)},
pages={5206--5210},
year={2015},
organization={IEEE}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/librispeech_test_clean.clotho_aqa_test@inproceedings{lipping2022clotho,
title={Clotho-aqa: A crowdsourced dataset for audio question answering},
author={Lipping, Samuel and Sudarsanam, Parthasaarathy and Drossos, Konstantinos and Virtanen, Tuomas},
booktitle={2022 30th European Signal Processing Conference (EUSIPCO)},
pages={1140--1144},
year={2022},
organization={IEEE}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and Zou, Xunlong and… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/clotho_aqa_test.audiocaps_test@inproceedings{kim2019audiocaps,
title={Audiocaps: Generating captions for audios in the wild},
author={Kim, Chris Dongjoo and Kim, Byeongchang and Lee, Hyunmin and Kim, Gunhee},
booktitle={Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
pages={119--132},
year={2019}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/audiocaps_test.covost2_en_id_test@inproceedings{wang2021covost,
title={CoVoST 2 and massively multilingual speech translation.},
author={Wang, Changhan and Wu, Anne and Gu, Jiatao and Pino, Juan},
booktitle={Interspeech},
volume={2021},
pages={2247--2251},
year={2021}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and Zou, Xunlong and Lin, Geyu and Sun, Shuo and Liu, Zhuohan and Zhang, Wenyu and Liu, Zhengyuan and Aw, AiTi and Chen… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/covost2_en_id_test.tedlium3_test@inproceedings{hernandez2018ted,
title={TED-LIUM 3: Twice as much data and corpus repartition for experiments on speaker adaptation},
author={Hernandez, Fran{\c{c}}ois and Nguyen, Vincent and Ghannay, Sahar and Tomashenko, Natalia and Esteve, Yannick},
booktitle={Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18--22, 2018, Proceedings 20},
pages={198--208},
year={2018},
organization={Springer}
}
@article{wang2024audiobench… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/tedlium3_test.common_voice_15_en_test@article{ardila2019common,
title={Common voice: A massively-multilingual speech corpus},
author={Ardila, Rosana and Branson, Megan and Davis, Kelly and Henretty, Michael and Kohler, Michael and Meyer, Josh and Morais, Reuben and Saunders, Lindsay and Tyers, Francis M and Weber, Gregor},
journal={arXiv preprint arXiv:1912.06670},
year={2019}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and Zou, Xunlong… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/common_voice_15_en_test.covost2_en_zh_test@inproceedings{wang2021covost,
title={CoVoST 2 and massively multilingual speech translation.},
author={Wang, Changhan and Wu, Anne and Gu, Jiatao and Pino, Juan},
booktitle={Interspeech},
volume={2021},
pages={2247--2251},
year={2021}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and Zou, Xunlong and Lin, Geyu and Sun, Shuo and Liu, Zhuohan and Zhang, Wenyu and Liu, Zhengyuan and Aw, AiTi and Chen… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/covost2_en_zh_test.MMAU-mini-do-not-useWARNING: The original dataset is revised and pleased refer to new data source. Please refer to: MMAU-v05.15.25: https://github.com/Sakshi113/MMAU
@misc{sakshi2024mmaumassivemultitaskaudio,
title={MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark},
author={S Sakshi and Utkarsh Tyagi and Sonal Kumar and Ashish Seth and Ramaneswaran Selvakumar and Oriol Nieto and Ramani Duraiswami and Sreyan Ghosh and Dinesh Manocha},
year={2024},
eprint={2410.19168}… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/MMAU-mini-do-not-use.librispeech_test_other@inproceedings{panayotov2015librispeech,
title={Librispeech: an asr corpus based on public domain audio books},
author={Panayotov, Vassil and Chen, Guoguo and Povey, Daniel and Khudanpur, Sanjeev},
booktitle={2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)},
pages={5206--5210},
year={2015},
organization={IEEE}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/librispeech_test_other.openhermes_instruction_test@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and Zou, Xunlong and Lin, Geyu and Sun, Shuo and Liu, Zhuohan and Zhang, Wenyu and Liu, Zhengyuan and Aw, AiTi and Chen, Nancy F},
journal={NAACL},
year={2025}
}
covost2_en_ta_test@inproceedings{wang2021covost,
title={CoVoST 2 and massively multilingual speech translation.},
author={Wang, Changhan and Wu, Anne and Gu, Jiatao and Pino, Juan},
booktitle={Interspeech},
volume={2021},
pages={2247--2251},
year={2021}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and Zou, Xunlong and Lin, Geyu and Sun, Shuo and Liu, Zhuohan and Zhang, Wenyu and Liu, Zhengyuan and Aw, AiTi and Chen… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/covost2_en_ta_test.iemocap_emotion_recognition@article{busso2008iemocap,
title={IEMOCAP: Interactive emotional dyadic motion capture database},
author={Busso, Carlos and Bulut, Murtaza and Lee, Chi-Chun and Kazemzadeh, Abe and Mower, Emily and Kim, Samuel and Chang, Jeannette N and Lee, Sungbok and Narayanan, Shrikanth S},
journal={Language resources and evaluation},
volume={42},
pages={335--359},
year={2008},
publisher={Springer}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/iemocap_emotion_recognition.CoM_Audio_Image_LLM_Generation
This dataset is a Mixture of DIBT/10k_prompts_ranked, lj_speech and Falah/image_generation_prompts_SDXL
Repartition
Why this dataset ?
Training a multimodal router holds crucial significance in the realm of artificial intelligence. By harmonizing different specialized models within a constellation, the router plays a central role in intelligently orchestrating tasks. This approach not only enables precise classification but also paves the way for diverse… See the full description on the dataset page: https://huggingface.co/datasets/Nielzac/CoM_Audio_Image_LLM_Generation.voxceleb_accent_test@article{nagrani2020voxceleb,
title={Voxceleb: Large-scale speaker verification in the wild},
author={Nagrani, Arsha and Chung, Joon Son and Xie, Weidi and Zisserman, Andrew},
journal={Computer Speech \& Language},
volume={60},
pages={101027},
year={2020},
publisher={Elsevier}
}
@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and Zou, Xunlong and Lin, Geyu and Sun, Shuo and Liu, Zhuohan and Zhang… See the full description on the dataset page: https://huggingface.co/datasets/AudioLLMs/voxceleb_accent_test.cn_college_listen_mcq_test@article{wang2024audiobench,
title={AudioBench: A Universal Benchmark for Audio Large Language Models},
author={Wang, Bin and Zou, Xunlong and Lin, Geyu and Sun, Shuo and Liu, Zhuohan and Zhang, Wenyu and Liu, Zhengyuan and Aw, AiTi and Chen, Nancy F},
journal={NAACL},
year={2025}
}