Roblox/voice-safety-classifier-v3

Roblox voice safety classifier v3

Model description

We present a voice safety violation detection and classification model. The model is a transformer distilled from a larger teacher model. All the model training has been conducted with Roblox internal voice chat datasets, using both machine and human-labeled data, with 300k hours of training data in total.

The classifier expects 16 kHz mono (or first-channel) WAV input. The intended segment length is up to 15 seconds. Accuracy may degrade on longer segments. The maximum supported audio duration is 30s. The functions to load audio (load_audio, load_audio_batch, or the CLI) truncate the audio to max 30s.

Toxicity heads

The model outputs one score per head (sigmoid). Labels follow the policy enum names in config.json: ABUSE_TYPE_PRIVACY_ASKING_FOR_PII, ABUSE_TYPE_DISCRIMINATORY, ABUSE_TYPE_HARASSMENT, ABUSE_TYPE_SEXUAL_CONTENT, ABUSE_TYPE_ILLEGAL_AND_REGULATED_CONTENT, ABUSE_TYPE_DATING_AND_ROMANTIC_CONTENT, ABUSE_TYPE_PROFANITY, ABUSE_TYPE_DISRUPTIVE_AUDIO.

See Roblox Community Standards for how these categories relate to moderation policy.

Supported languages

The model supports 30 languages: Arabic, Bulgarian, Chinese, Croatian, Czech, Danish, Dutch, English, Finnish, French, German, Greek, Hungarian, Indonesian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese, Romanian, Russian, Serbian, Slovak, Spanish, Swedish, Tagalog, Thai, Turkish, Ukrainian.

The model provides auxiliary heads for language detection. For the mapping of language heads to language codes, see languages in config.json. Note that Croatian and Serbian are detected by the same language head hr.

Evaluation

Metrics below are recall and precision on internal held-out sets measuring whether the given phrase contained abuse or not. Disruptive audio was excluded from the evaluation. Operating points were chosen for a binary 1% false positive rate for each language.

Comparing the languages supported by the v2 model (English, French, German, Italian, Japanese, Korean, Portuguese, and Spanish) weighted by Roblox voice chat language distribution, the recall improved 14% relative and precision 5% relative with respect to Roblox/voice-safety-classifier-v2.

Usage

Install dependencies:

pip install -r requirements.txt

Optional — Hugging Face AutoModel.from_pretrained and the packaged VoiceToxicityClassifier config:

pip install -r requirements-optional.txt

Run inference on one or more WAV files. The model directory must contain config.json and model.safetensors (defaults to the current directory):

python inference.py --model-dir /path/to/model_dir /path/to/audio.wav
python inference.py /path/to/audio.wav --device cuda --output results.json

For batch inputs and JSON output:

python inference.py --model-dir /path/to/model_dir a.wav b.wav c.wav --output results.json

Python API (see inference.py for full signatures):

from inference import load_model, load_audio, run_inference, extract_label_scores

model, config = load_model("/path/to/model_dir")
audio = load_audio("clip.wav")
raw = run_inference(model, audio)
out = extract_label_scores(raw["probs"], raw.get("language_probs"), config, index=0)
# out["label_scores"], out.get("language_probs")

Hugging Face AutoModel

With transformers installed, you can load the same checkpoint via AutoModel.from_pretrained.

import json
from pathlib import Path

# you need the inference.py file to run
import inference  # registers VoiceToxicityClassifier with AutoModel
from transformers import AutoModel

from inference import extract_label_scores, load_audio, run_inference

model = AutoModel.from_pretrained("Roblox/voice-safety-classifier-v3")
audio = load_audio("clip.wav")
raw = run_inference(model, audio)
# Make human-readable output
out = extract_label_scores(raw["probs"], raw.get("language_probs"), model.config.to_dict(), index=0)
print(out)

Audio files must be 16 kHz WAV (mono or stereo; first channel is used).

License

Apache License 2.0, see LICENSE.md.