Team Ai
Modelpublic

giangndm/omniASR-LLM-300M-encoder

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face
0likes12kdownloads
Model Card

omniASR-LLM-300M Audio Encoder (bfloat16 Transformers SafeTensors)

This repository contains the standalone bfloat16 `safetensors` Audio Encoder extracted from `ziywang50/omniASR-LLM-300M` / `facebook/omniASR-LLM-300M`.

It is $100\%$ natively compatible with the Hugging Face transformers library (AutoModel / Wav2Vec2Model / AutoFeatureExtractor).

Specifications:

  • —Architecture: Wav2Vec2Model (Conformer-based SSL 24 layers, 16 heads, 1024 hidden size)
  • —Precision: bfloat16
  • —Model Size: ~630 MB (model.safetensors)
  • —Sampling Rate: 16 kHz
  • —Output: 50 Hz frame representations (20ms/frame) in $\mathbb{R}^{1024}$

Usage with Hugging Face transformers:

python
import torch
from transformers import AutoFeatureExtractor, AutoModel

# 1. Load Audio Feature Extractor and Encoder
model_id = "giangndm/omniASR-LLM-300M-encoder"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, torch_dtype=torch.bfloat16).to("cuda")

# 2. Extract Speech Embeddings
audio_input = torch.randn(1, 16000) # 1 sec dummy audio at 16kHz
inputs = feature_extractor(audio_input.squeeze().numpy(), sampling_rate=16000, return_tensors="pt").to("cuda")
inputs["input_values"] = inputs["input_values"].to(torch.bfloat16)

with torch.no_grad():
    hidden_states = model(**inputs).last_hidden_state  # [1, T_frames, 1024]

print("Audio hidden states shape:", hidden_states.shape) # [1, 49, 1024]