Modelpublic

AbhishekG711/Qwen3-0.6B-Insight-Extractor

sourceHugging Faceapache-2.0updated 1d agoView on Hugging Face

0likes967downloads

Model Card

Model Summary

Qwen3-0.6B-Insight-Extractor is a LoRA fine-tune of [`Qwen/Qwen3-0.6B`](https://huggingface.co/Qwen/Qwen3-0.6B). It converts a raw customer-support ticket into a single structured JSON object across 9 fields, for automated ticket classification and downstream entity-extraction workflows.


Base model	`Qwen/Qwen3-0.6B`
Fine-tuning method	LoRA Adapters via Unsloth + TRL `SFTTrainer`
Task	Structured JSON extraction from support-ticket text
Training data	`AbhishekG711/processed_support_tickets` (`train` split)
Target repo	`AbhishekG711/Qwen3-0.6B-Insight-Extractor`

Base Model Architecture (Qwen3-0.6B)


Type	Dense, causal decoder-only Transformer
Parameters	0.6B total (~0.44B non-embedding)
Layers	28
Attention	Grouped-Query Attention — 16 query heads / 8 KV heads
Native context length	32,768 tokens
Reasoning modes	Hybrid — supports both "thinking" (`<think>...</think>`) and "non-thinking" modes via `enable_thinking`
Base model license	Apache 2.0

Intended Use

Given a raw support ticket, produce one JSON object for automated triage, routing, sentiment/urgency dashboards, or downstream alerting — not a conversational assistant.

Fine-Tuning Training Details

Hyperparameter	Value
Quantization	None (`load_in_4bit=False`)
LoRA rank (`r`)	32
LoRA alpha	64 (`= 2 × r`)
LoRA dropout	0.0
LoRA target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
LoRA bias	`none`
Max sequence length	4096
Per-device train batch size	4
Target global batch size	32
Gradient accumulation steps	8 (`= 32 // 4`)
Epochs	1
Learning rate	2e-4
LR scheduler	linear
Warmup steps	10
Weight decay	0.01
Optimizer	`adamw_8bit`
Precision	bf16 if supported, else fp16 (auto-detected via `is_bf16_supported()`)
Completion-only loss	`True` — loss computed only on `completion` tokens, not the prompt
Sequence packing	`False`
Eval strategy	every 25 steps
Save strategy	every 50 steps, `save_total_limit=3`
Load best model at end	`True` (`metric_for_best_model="eval_loss"`, `greater_is_better=False`)
Gradient checkpointing	`"unsloth"` mode
Seed	3407
Experiment tracker	`report_to="none"` — console/log file only

Evaluation

Metric	Value
Validation loss (`eval_loss`, best checkpoint)	0.233829

How to Prompt This Model

The model was fine-tuned against exactly this system/user structure — matching it as closely as possible at inference time will give the best results.

System prompt:

# ROLE
You are a deterministic extraction engine. You convert raw support emails into exactly one structured JSON object. You never converse, explain, or output anything except that JSON object.

User message template:

Analyze the following customer support email and extract structured insights.

<raw ticket text>

Expected output: One JSON object with exactly these 9 keys, in this order: is_actionable, summary, sentiment, category, intent, aspect, urgency, reported_cause, entities.

Inference example

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AbhishekG711/Qwen3-0.6B-Insight-Extractor"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

SYSTEM_PROMPT = """
# ROLE
You are a deterministic extraction engine. You convert raw support emails into exactly one structured JSON object. You never converse, explain, or output anything except that JSON object.
"""
USER_INSTRUCTION = "Analyze the following customer support email and extract structured insights.\n\n"

ticket_text = "My order #4471 arrived damaged and support hasn't replied in 3 days."

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": USER_INSTRUCTION + ticket_text},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
).to(model.device)

output = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(output[0][inputs.shape[-1]:], skip_special_tokens=True))