KimZoey/reaction_data

🧠 Overview

Reaction Data v1.0 is a curated multimodal dataset built from YouTube human reaction videos, designed for studying singing voice synthesis (SVS) evaluation, audio–text alignment, and natural-language reaction modeling. Each included creator (e.g., Alex Hefner, Beth Roars, The Charismatic Voice) provides a distinct commentary style and persona, offering diverse reactions across genres and vocal styles.

Each reviewer’s folder contains processed data, transcripts, and raw video material, enabling flexible multimodal experiments.

📁 Dataset Structure

Each reviewer (e.g., Alex_Hefner, BethRoars, The_Charismatic_Voice, etc.) has three corresponding archives:

Additionally, the file user_info.json stores reviewer metadata and file list.

⚙️ Data Processing Pipeline

The dataset is created through the following pipeline:

1. 1.Video & Subtitle Retrieval
2. 2.Download with yt-dlp.
3. 3.Retrieve subtitles via the YouTube Transcript API.

1. 1.Audio Extraction & Diarization
2. 2.Extract audio from each video.
3. 3.Apply pyannote.audio for speaker diarization and merge utterances per speaker.

1. 1.Audio Classification
2. 2.Each utterance is classified as singing or spoken commentary using MIT/ast-finetuned-audioset-10-10-0.4593.

1. 1.Segment Alignment
2. 2.Align contiguous singing segments with subsequent critic commentary.
3. 3.Match commentary timestamps with subtitles to extract review text.

1. 1.Metadata Integration
2. 2.Attach reviewer persona metadata (from channel introductions).
3. 3.Add song metadata (from Wikipedia).
4. 4.Form complete multimodal training samples.

1. 1.Quality Filtering
2. 2.Remove samples with:
3. 3.Empty subtitles or missing text
4. 4.Audio shorter than 10 seconds
5. 5.Text shorter than 8 words → Ensures linguistically and acoustically rich data.

💡 Research Motivation

This dataset supports research on:

• —Generating and evaluating natural-language feedback for singing performances
• —Modeling human-like reaction patterns across multiple reviewer personas
• —Assessing singing voice synthesis systems under realistic multimodal conditions

By including authentic reaction speech, expressive commentary, and varied recording setups, it fosters robust model generalization.

🧩 Notes & Limitations

Alternative pipelines (ASR-first, diarization-only) were tested but found less reliable due to:

• —Overlapping singing and commentary causing poor segmentation
• —Garbled ASR transcriptions under mixed speech
• —Subtitle–speaker mismatch for short utterances

The adopted hybrid pipeline provides the best balance between accuracy, diversity, and robustness.

📜 Metadata

• —Reviewer list and metadata: stored in user_info.json
• —Fields include:
• —channel_name
• —persona_description
• —video_count
• —language
• —estimated_total_duration

🏷️ Tags

audio-text, multimodal, reaction, singing, music-evaluation, svs, huggingface-datasets, LLM4Music, commentary