TalTechNLP/voxlingua107_wds

VoxLingua107

VoxLingua107 is a speech dataset for training spoken language identification models. The dataset consists of short speech segments automatically extracted from YouTube videos and labeled according the language of the video title and description, with some post-processing steps to filter out false positives.

VoxLingua107 contains data for 107 languages. The total amount of speech in the training set is 6628 hours. The average amount of data per language is 62 hours. However, the real amount per language varies a lot. There is also a seperate development set containing 1609 speech segments from 33 languages, validated by at least two volunteers to really contain the given language.

For more information, see the paper Jörgen Valk, Tanel Alumäe. _VoxLingua107: a Dataset for Spoken Language Recognition_. Proc. SLT 2021.

Why

VoxLingua107 can be used for training spoken language recognition models that work well with real-world, varying speech data. You can try a demo system trained on this dataset here.

How

We extracted audio data from YouTube videos that are retrieved using language-specific search phrases (random phrases from Wikipedia of the particular language). If the language of the video title and description matched with the language of the search phrase, the audio in the video was deemed likely to be in that particular language. This allowed to collect large amounts of somewhat noisy data relatively cheaply. Speech/non-speech detection and speaker diarization was used to segment the videos into short sentence-like utterances. A data-driven post-filtering step was applied to remove clips that were very different from other clips in this language's dataset, and thus likely not in the given language. Due to the automatic data collection process, there are still clips in the dataset that are not in the given language or contain non-speech (around 2% overall), especially for some languages (like Welsh).

License and copyright

The VoxLingua107 dataset is distributed under the Creative Commons Attribution 4.0 International License. The copyright remains with the original owners of the video.

We also point out that the distribution of languages, accents, dialects, genders, races and societal factors in this dataset is not representative of the global population. Using this dataset for training and deploying models may thus introduce unintended biases.

Notice and take down policy

Notice: Should you consider that our data contains material that is owned by you and should therefore not be reproduced here, please:

• —Clearly identify yourself, with detailed contact data such as an address, telephone number or email address at which you can be contacted.
• —Clearly identify the copyrighted work claimed to be infringed.
• —Clearly identify the material that is claimed to be infringing and information reasonably sufficient to allow us to locate the material.
• —Send the request to Tanel Alumäe

Take down: We will comply to legitimate requests by removing the affected sources from the corpus.

Languages and sizes

Usage

Although webdataset can be used in a streaming fashion, it is recommended to first make a local copy fo the dataset using git clone.

git lfs install git clone git@hf.co:datasets/TalTechNLP/voxlingua107_wds

Then you can use the Python webdataset library to create an iterable dataset out of it:

import webdataset as wds import random trainfiles = glob.glob("voxlingua107wds/train/*/.tar") # you can also limit the training data to selected languages devfiles = glob.glob("voxlingua107wds/dev/*.tar")

random.shuffle(train_files)

def mapper(sample): # "audio" field represents 16 kHz raw audio return {"audio": sample[0], "lang": sample[1]["lang"]}

# Since each shard contains 500 samples for a single language, it is good to use a reasonably large buffer size to get nicely shuffled samples buffersize = 100000 dataset = wds.WebDataset(trainurls, shardshuffle=2000).shuffle(buffersize, initial=buffersize).decode(wds.torchaudio).totuple("wav","json").map(mapper) devdataset = wds.WebDataset(devurls).decode(wds.torchaudio).totuple("wav","json").map(mapper)

trainiter = iter(dataset) print(next(trainiter)) {'audio': (tensor([[-9.7656e-04, -8.5449e-04, -3.0518e-05, ..., 2.7466e-03, 3.7842e-03, 5.1880e-03]]), 16000), 'lang': 'kn'}

Citing

@inproceedings{valk2021slt,
  title={{VoxLingua107}: a Dataset for Spoken Language Recognition},
  author={J{\"o}rgen Valk and Tanel Alum{\"a}e},
  booktitle={Proc. IEEE SLT Workshop},
  year={2021},
}