Modelpublic

jahnaviym/crate-clap-lora

sourceHugging Facemitupdated 2mo agoView on Hugging Face

0likes20downloads

Model Card

Crate — CLAP LoRA for producer vocabulary

LoRA adapter over `laion/clap-htsat-unfused` fine-tuned so producer terms base CLAP barely knows — boom-bap, tape-saturated, rimshot, reese bass — pull the right audio. Part of Crate, sound-native search for music producers (hum / drop a track / describe it → one embedding space).

Results — producer-vocab retrieval (held-out)

metric	base CLAP	fine-tuned
recall@1	0.210	0.405
recall@10	0.746	0.951
recall@5	0.580	0.868

Usage

python

from transformers import ClapModel, ClapProcessor
from peft import PeftModel

base = "laion/clap-htsat-unfused"
model = PeftModel.from_pretrained(ClapModel.from_pretrained(base), "jahnaviym/crate-clap-lora").merge_and_unload()
proc = ClapProcessor.from_pretrained(base)
# proc(text=[...]) / proc(audios=[...], sampling_rate=48000) → get_text/audio_features

Training

—Symmetric InfoNCE on (audio, text) pairs from Freesound (CC0/CC-BY) + free packs.
—LoRA on the audio/text projection heads (r=16, alpha=32).
—Label-preserving augmentation (pitch/stretch/noise/EQ) for positive pairs.

See the repo for the full pipeline and eval.