Datasetpublic

OmniEvalKit/omnievalkit-dataset

OmniEvalKit Evaluation Datasets Evaluation datasets for OmniEvalKit, a comprehensive evaluation framework for omni-modal (audio + video + image + text) models. Overview Total subsets: 65 Total samples: 315,264 Total size: 620.3 GB (Parquet with embedded audio/image/video) Subsets with embedded video: 15 Subsets requiring external video download: 2 Usage from datasets import load_dataset ds = load_dataset("OmniEvalKit/omnievalkit-dataset"… See the full description on the dataset page: https://huggingface.co/datasets/OmniEvalKit/omnievalkit-dataset.

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face

0likes246downloads

Dataset Card

OmniEvalKit Evaluation Datasets

Evaluation datasets for OmniEvalKit, a comprehensive evaluation framework for omni-modal (audio + video + image + text) models.

Overview

—Total subsets: 65
—Total samples: 315,264
—Total size: 620.3 GB (Parquet with embedded audio/image/video)
—Subsets with embedded video: 15
—Subsets requiring external video download: 2

Usage

python

from datasets import load_dataset

ds = load_dataset("OmniEvalKit/omnievalkit-dataset", "aishell1_test")
for sample in ds["test"]:
    print(sample)

Available Subsets

Audio (44 subsets, 267,616 samples, 186.5GB)

Subset	Display Name	Samples	Size	Subcategory	Video
`aishell1_test`	AISHELL-1 Test	7,176	1.1GB	asr	No
`aishell2_test`	AISHELL-2 Test	5,000	432MB	asr	No
`audio_trivia_qa`	Audio Trivia QA	1,024	322MB	qa	No
`audio_web_questions`	Audio Web Questions	2,032	222MB	qa	No
`audiocaps_test`	AudioCaps Test	3,985	139.2GB	caption	No
`clothocaption_test`	ClothoCaption Test	1,045	1.8GB	caption	No
`commonvoice_en`	CommonVoice English v15	16,386	664MB	asr	No
`commonvoice_fr`	CommonVoice French v15	16,132	653MB	asr	No
`commonvoice_yue`	CommonVoice Cantonese v15	5,593	217MB	asr	No
`commonvoice_zh`	CommonVoice Chinese v15	10,625	470MB	asr	No
`covost2_en_zh`	CoVoST2 EN-ZH	15,530	672MB	ast	No
`covost2_zh_en`	CoVoST2 ZH-EN	4,897	225MB	ast	No
`fleurs_en`	FLEURS 英文	647	383MB	asr	No
`fleurs_zh`	FLEURS 中文	945	664MB	asr	No
`gigaspeech_test`	GigaSpeech Test	19,870	4MB	asr	No
`kespeech_test`	KeSpeech Test	19,723	3.2GB	asr	No
`librispeech_dev_clean`	LibriSpeech Dev Clean	2,703	326MB	asr	No
`librispeech_dev_other`	LibriSpeech Dev Other	2,864	304MB	asr	No
`librispeech_test_clean`	LibriSpeech Test Clean	2,620	334MB	asr	No
`librispeech_test_other`	LibriSpeech Test Other	2,939	318MB	asr	No
`meld`	MELD	2,610	1.4GB	cls	Yes
`mmar_bench`	MMAR Bench	1,000	3.3GB	qa	No
`mmau_test_mini`	MMAU Test Mini	1,000	1.4GB	qa	No
`mmsu_bench`	MMSU Bench	4,996	1.3GB	qa	No
`peoples_speech_test`	People's Speech Test	34,898	3.8GB	asr	No
`tedlium3_test`	TED-LIUM v3 Test	3,737	790MB	asr	No
`vocalsound`	VocalSound	3,591	1.0GB	cls	No
`voice_cmmlu`	Voice CMMLU	11,582	13.3GB	qa	No
`voicebench_advbench`	VoiceBench AdvBench	520	0MB	qa	No
`voicebench_alpacaeval`	VoiceBench AlpacaEval	199	0MB	qa	No
`voicebench_alpacaeval_full`	VoiceBench AlpacaEval Full	636	0MB	qa	No
`voicebench_bbh`	VoiceBench BBH	1,000	0MB	qa	No
`voicebench_commoneval`	VoiceBench CommonEval	200	0MB	qa	No
`voicebench_ifeval`	VoiceBench IFEval	345	0MB	qa	No
`voicebench_mmsu`	VoiceBench MMSU	3,074	0MB	qa	No
`voicebench_openbookqa`	VoiceBench OpenBookQA	455	0MB	qa	No
`voicebench_sdqa`	VoiceBench SD-QA	6,083	0MB	qa	No
`voicebench_wildvoice`	VoiceBench WildVoice	1,000	0MB	qa	No
`voxpopuli_en`	VoxPopuli English	1,842	899MB	asr	No
`wavcaps_audioset_sl`	WavCaps AudioSet_SL	11,676	3.7GB	caption	No
`wavcaps_freesound`	WavCaps FreeSound	1,060	2.7GB	caption	No
`wavcaps_soundbible`	WavCaps SoundBible	1,232	530MB	caption	No
`wenetspeech_test_meeting`	WenetSpeech Test Meeting	8,370	377MB	asr	No
`wenetspeech_test_net`	WenetSpeech Test Net	24,774	682MB	asr	No

Omni (20 subsets, 46,648 samples, 432.1GB)

Subset	Display Name	Samples	Size	Subcategory	Video
`av_odyssey`	AV-Odyssey	4,555	31.2GB	qa	No
`avmeme_full`	AVMeme-Exam Full	1,032	777MB	meme_understanding	Yes
`avmeme_main`	AVMeme-Exam Main	846	643MB	meme_understanding	Yes
`avut_benchmark_gemini`	AVUT-Benchmark Gemini	9,874	49.4GB	qa	Yes
`avut_benchmark_human`	AVUT-Benchmark Human	1,734	12.3GB	qa	Yes
`daily_omni`	Daily-Omni	1,197	966MB	qa	Yes
`futureomni`	FutureOmni	1,034	50.7GB	qa	Yes
`jointavbench`	JointAVBench	2,853	37.4GB	qa	Yes
`livesports3k_cc`	LiveSports-3K CC	1,702	63.0GB	caption	Yes
`omnibench`	OmniBench	1,142	1.2GB	qa	No
`ovavel`	OV-AVEL Test	5,818	11.2GB	event_localization	Yes
`ovobench`	OVO-Bench	1,468	64.1GB	qa	Yes
`streamingbench_real`	StreamingBench-Real (Offline)	2,499	55.6GB	qa	Yes
`streamingbench_sqa`	StreamingBench-SQA (Offline)	250	6.5GB	qa	Yes
`unobench`	UNO-Bench	3,730	7.8GB	qa	No
`unobench_mc`	UNO-Bench-MC	1,000	3.6GB	qa	No
`video_holmes`	Video-Holmes	1,837	5.8GB	qa	Yes
`videomme`	Video-MME	5	26MB	qa	Yes
`videomme_short`	Video-MME (Short)	900	3.8GB	qa	Yes
`worldsense`	WorldSense	3,172	26.0GB	qa	Yes

Unknown (1 subsets, 1,000 samples, 1.7GB)

Subset	Display Name	Samples	Size	Subcategory	Video
`streamingbench_omni`	streamingbench_omni	1,000	1.7GB	unknown	No

Data Format

Each subset is stored as Parquet file(s):

—Metadata fields: All original JSONL fields (question, answer, paths, etc.)
—`audio`: HuggingFace Audio type — struct<bytes: binary, path: string>, playable in Dataset Viewer
—`image`: HuggingFace Image type — struct<bytes: binary, path: string>, viewable in Dataset Viewer
—`video`: Video binary — struct<bytes: binary, path: string>, embedded for applicable subsets
—`audio_bytes_dict`: Multiple audio files as base64-encoded JSON string
—`image_bytes_dict`: Multiple image files as base64-encoded JSON string

License

Apache 2.0. Individual datasets may have their own licenses.