zed-m97/nano4m-Audio

nano4M-Audio — Team (week-1)

Week-1 data preparation for nano4M-Audio, an extension of EPFL's nano4M (the educational nano version of 4M / 4M-21) that adds audio as a fifth modality alongside RGB, depth, surface normals and captions.

This dataset covers all 12 VGGSound classes assigned to the three-person team:

It contains the pre-tokenized modalities consumed by nano4M's SimpleMultimodalDataset (tok_audio@256, tok_rgb@256, scene_desc). The raw WAV+JPG pairs are kept on each teammate's machine and are not redistributed here; the tokens are the canonical artifact for training.

Source attribution. Audio + image content is derived from VGGSound (Chen et al., ICASSP 2020), whose primary key is (ytid, start). Every file in this dataset is a 1.71 s, 24 kHz mono crop centered on the CLAP-best audio peak of the corresponding VGGSound clip and the CLIP-best frame of the same window. We re-distribute the raw clips here for reproducibility under VGGSound's CC-BY-4.0 terms.

Counts

12 VGGSound classes, ~4.9k clips after link rot + CLIP/CLAP filtering. Per-class breakdown (see stats_team.md for full distributions and qualitative samples):

All clips are uniformly 41,040 samples = 1.7100 s @ 24 kHz mono PCM16 before tokenization (frames sliced by the CLAP-best 1.71 s window from a 10 s VGGSound clip).

Files

.
├── manifest_team.csv                     canonical join table (6,529 VGGSound rows; 4,905 OK)
├── stats_team.md                         per-class counts + score histograms
├── manifest_person2.csv                  legacy Person-2 manifest (kept for reference)
├── stats_person2.md                      legacy Person-2 stats
└── tokenized/
    ├── train/tok_audio@256/{stem}.npy    EnCodec tokens, shape (1, 256), int16, [0, 1023]
    ├── train/tok_rgb@256/{stem}.npy      Cosmos tokens,  shape (1, 256), int32, [0, 63999]
    ├── train/scene_desc/{stem}.json      JSON list ["<class>"] (K=1 augmentation)
    └── test/...                          same layout

{stem} is {ytid}_{start:06d} — globally unique across the 12-class team union since (ytid, start) is VGGSound's primary key.

Tokenizer details

On-disk contract for nano4M

Three details that matter for compatibility with nano4M's SimpleMultimodalDataset:

1. 1.`@256` is part of the directory name. The dataloader reads from {root}/{split}/{modality_string}/{stem}{ext} and the modality string in the config is literally "tok_audio@256", not "tok_audio".
2. 2.*`scene_desc/.json is a JSON list, not a dict.** The dataloader does captions[augmentationidx]`. Run with `samplefromkaugmentations=1`.
3. 3.Captions are tokenized at load time, not pre-tokenized. The dataloader instantiates GPT-2 with [SOS] $A [EOS] template (vocab 50,304) on the fly.

Provenance & limitations

• —Link rot is permanent. 385 of 2,155 source rows (17.9%) failed yt-dlp because the YouTube videos are unavailable / private / region-blocked. These are listed in raw/failed.txt with reason FAIL_DL. Re-running the downloader at a later date will lose more clips, not gain any back — that is why the raw WAV+JPG pairs are re-distributed here.
• —CLIP/CLAP filtering. 26 clips dropped for FAIL_CLIP (best frame's CLIP score < 0.20 vs class label) and 14 for FAIL_CLAP (best 1.71 s window's CLAP logit < 0.20 vs class label). 1 FAIL_PEAKS (silent / sub-1.71 s clip). All in failed.txt.
• —Class imbalance. elephant trumpeting has ~3× fewer clips than dog barking due to lower YouTube availability.
• —Single keyframe per clip. Only the CLIP-best frame within the chosen 1.71 s window is provided, not a full video.

Reproducing

Pipeline scripts and full README are in the project repo (private). Briefly: download via yt-dlp from VGGSound's official CSV → CLAP-pick best 1.71 s window → CLIP-pick best frame in that window → EnCodec tokenize audio → Cosmos tokenize image → emit manifest. The Cosmos step requires a CUDA GPU (this dataset's tok_rgb@256 was generated on an NVIDIA L40S).

Citation

If you use VGGSound (the underlying source), please cite:

@inproceedings{chen2020vggsound,
  title={VGGSound: A Large-scale Audio-Visual Dataset},
  author={Chen, Honglie and Xie, Weidi and Vedaldi, Andrea and Zisserman, Andrew},
  booktitle={ICASSP},
  year={2020}
}

For 4M / nano4M:

@inproceedings{mizrahi20234m,
  title={4M: Massively Multimodal Masked Modeling},
  author={Mizrahi, David and Bachmann, Roman and Kar, O{\u{g}}uzhan Fatih and Yeo, Teresa and Gao, Mingfei and Dehghan, Afshin and Zamir, Amir},
  booktitle={NeurIPS},
  year={2023}
}

License

Released under CC-BY-4.0, inheriting from VGGSound. You must cite VGGSound and respect YouTube's Terms of Service when using this data.