Weidows/laya-multilingual-GGUF

laya-multilingual GGUF

GGUF conversion of convaiinnovations/laya-multilingual (Apache-2.0), encoder only (mmBERT-base, 22 layers, hidden 768). Bit-identical to an independent community conversion (134/134 tensors, max diff 0).

The decision head (36 tensors, 30 MB) ships separately as laya-multilingual-head.safetensors (same split as an mmproj file); laya_head.py has load_head(path) returning config + weights under original PyTorch names.

Token ids must come from the shipped HF tokenizer.json (Gemma-style tokenizer; llama.cpp text tokenization not reliable for this vocab).

Files

imatrix stats from 400-line 10-language Wikipedia sample (imatrix-laya.dat used at quant time; dat file not shipped). Weight-space rel err is a smoke signal only — see eval below.

Eval (end-to-end, llama.cpp server + original decision head)

12 samples (6 XNLI choice across en/zh/ar/hi/es/ja + 3 dept choice + 3 urgency noul), same HF token ids into PyTorch FP32 and each GGUF via llama.cpp server (--pooling none, --embd-normalize -1), identical head weights. F16 matches FP32 exactly (12/12, drift 0.0015).

Mismatches only in Q6K (1 hi) and Q3KM (1 ar + 1 hi, drift 0.34+). Recommendation: Q80 for anything important; Q4KM-imat / IQ4XS-imat for size; avoid Q3K_M on non-English.

[image]

Use

llama-server -m laya-multilingual-Q8_0.gguf --embeddings --pooling none -c 2048 -ub 2048 -b 2048 --port 8080
pip install laya requests torch safetensors

import requests, torch, laya
from laya_head import load_head

agent = laya.load("convaiinnovations/laya-multilingual", device="cpu")
_, head = load_head("laya-multilingual-head.safetensors")
sd = agent.model.state_dict()
for k, v in head.items():
    if k in sd:
        sd[k].copy_(v)

D = agent.model.encoder.config.hidden_size
def llamacpp_encoder(input_ids, attention_mask=None, **_):
    out = []
    for i, row in enumerate(input_ids):
        n = int(attention_mask[i].sum())
        r = requests.post("http://localhost:8080/embedding", json={"content": [row[:n].tolist()]}).json()
        h = torch.zeros(input_ids.shape[1], D); h[:n] = torch.tensor(r[0]["embedding"]); out.append(h)
    return torch.stack(out)