Badalt/intent-classifier-mpnet

intent-classifier-mpnet

Routes a short user message from a logistics / supply-chain chat assistant to one of 12 intents, and abstains with `unknown` when it is not confident (calibrated reject rule stored in config.json). Fine-tuned from `sentence-transformers/paraphrase-multilingual-mpnet-base-v2` as a standard AutoModelForSequenceClassification (no custom code).

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "Badalt/intent-classifier-mpnet"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

texts = ["whats the eta on LD-55501 pls", "¿Cuántos remolques hay en el patio?"]
with torch.no_grad():
    logits = model(**tok(texts, padding=True, truncation=True, max_length=64, return_tensors="pt")).logits
probs = torch.softmax(logits / model.config.calibration_temperature, dim=-1)
conf, idx = probs.max(-1)
for c, i in zip(conf, idx):
    label = "unknown" if c < model.config.unknown_threshold else model.config.id2label[int(i)]
    print(label, round(float(c), 3))

Reject rule: unknown if max softmax(logits / 0.4622) < 0.5735. The threshold keeps 95.0% of known-intent validation messages; the temperature was fitted on the same validation set.

Labels

ai_agent_performance, appointment_manager, chitchat, customer_support, document_processing, knowledge_base, orders, other, shipment_information.analytics, shipment_information.disruptions, shipment_information.realtime_query, yard_management (+ unknown via the reject rule).

Training data

500 labelled synthetic chat messages (12 intents, 30–55 per class; ~13% non-English: ES, FR, DE, ZH, NL, AR, IT; noisy, colloquial). The dataset is confidential and not distributed with this model. Split: stratified 70/15/15 (347 / 76 / 77) with near-duplicate paraphrases kept in the same split; seed 42.

Training procedure

lr 5e-05, batch 16, ≤15 epochs with early stopping on validation macro-F1 (best epoch 4), linear decay, warmup 0.1, weight decay 0.01, class-weighted cross-entropy, max_length 64, seed 42. Backbone and learning rate chosen by 5-fold cross-validation among 3 multilingual encoders × 2 learning rates.

Evaluation

Held-out test set (n=77, never used for any decision): macro-F1 0.885 (95% bootstrap CI 0.791–0.950), accuracy 0.896; English 0.879 vs non-English 1.000 accuracy (n=11).

5-fold CV on train+val (n=423, out-of-fold): macro-F1 0.920 (fold mean 0.920 ± 0.028); non-English accuracy 0.942.

Open-set (unknown intents). A twin model trained without yard_management and document_processing was tested on known messages plus all messages of those unseen intents: max-softmax reject rule AUROC 0.791, rejection recall 26.2% at 93.8% retention; best scorer (kNN k=1 [mean]) AUROC 0.821.

Intended use & limitations

• —First-step router for a logistics assistant; low-confidence messages should go to a clarification / fallback path.
• —Trained on 500 synthetic rows: expect lower accuracy on real traffic; monitor confidence, unknown-rate and language mix for drift and re-calibrate the threshold on fresh labelled data.
• —The three shipment_information.* intents are semantically close and account for a large share of errors.
• —Non-English evaluation rests on few examples; languages not seen in training (e.g. Arabic, Italian) are untested beyond anecdotes.
• —other / chitchat are trained catch-alls; genuinely new intents are handled only by the confidence threshold.

Training curves and metrics: https://wandb.ai/badalthakur2212-iisc/intent-classifier/runs/ffy90jmz