kehanlu/interspeech-tutorial

1---2license: apache-2.03language: en4tags: [speech-llm, desta, librispeech, tutorial]5---6 7# Interspeech tutorial — DeSTA-style SpeechLLM checkpoints8 9Whisper-large-v3 encoder (frozen) → concat+MLP adapter → Qwen3-4B-Instruct-2507 + LoRA r32.10Only the adapter and the LoRA weights are trained, so each checkpoint is ~147 MB; the base11models are downloaded from their own repos at load time.12 13Code, configs and the full recipe: <https://github.com/kehanlu/interspeech-tutorial>14 15| folder | training data | test-clean ASR | test-clean gender |16|---|---|---|---|17| `asr_gender` | 281k ASR + a fresh 30% of the gender rows each epoch | **1.81** WER | **98.85** |18| `selfgen` | 281k self-generated conversational replies, no task labels | 3.93 WER | 98.24 |19 20`asr_gender` is ordinary task SFT, and it matches whisper-large-v3 on ASR (1.89) while also21answering the gender question. It is the baseline.22 23`selfgen` is the interesting one: its targets were written by Qwen3-4B given only the24transcript and the speaker's gender, so **it has never seen a transcription or a gender25label as a training target**. The self-generation prompt was26 27    <audio>{transcription} (Gender: {gender})</audio>28 29    The audio is a passage read aloud from a book. Respond directly as a natural30    conversation partner. Do not mention the audio, the transcription, or the speaker31    attributes.32 33It can still do both tasks, but only if the prompt leaves room for a short answer. Asked the34way `asr_gender` was trained ("Transcribe the speech into text") it replies with an essay35about the passage and scores 52.35 WER; asked for a format it reaches 3.93:36 37| prompt | ASR |38|---|---|39| `Transcribe the speech into text` | 52.35 → 16.54 after clean-up |40| `Transcribe the speech word for word. Output only the transcription, with no explanation, in this format:\nAnswer: "<transcription>"` | 8.94 → **3.93** |41 42Gender goes 81.87 → **98.24** the same way, with "The audio is a passage read aloud from a43book. Is the speaker male or female? Answer with one word." The clean-up is the rule-based44`postprocess()` in the tutorial repo's `example/evaluate/evaluate_asr.py`; it is a no-op on45`asr_gender`, which already answers with a bare transcript.46 47## Usage48 49The model code is in the GitHub repo:50 51```bash52git clone https://github.com/kehanlu/interspeech-tutorial53```54 55```python56from huggingface_hub import hf_hub_download57import sys, torch58 59sys.path.insert(0, "interspeech-tutorial")60from inference import SpeechLLMForInference61 62ckpt = hf_hub_download("kehanlu/interspeech-tutorial", "selfgen/model.ckpt")63pipe = SpeechLLMForInference.from_checkpoint(ckpt, dtype=torch.float16)   # float16 for a Colab T464print(pipe.generate([{"role": "user",65                      "content": "<audio><|AUDIO|></audio>\n\nTranscribe the speech into text",66                      "audios": [{"audio": "sample.flac"}]}]))67```68 69A few LibriSpeech dev-clean clips are in `samples/` with their transcripts in70`samples/samples.json`.71