Modelpublic

gittensor-model-hub/Qwythos-9B-nsys-SFT

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes125downloads

Model Card

Qwythos-9B nsys SFT (self-taught)

LoRA + GGUF fine-tune of Qwythos-9B for Nsight Systems (`nsys`) / CUDA-L1 tool-calling.

Self-taught (not API distillation)

The SFT trajectories were generated by this same local model stack, not by an external teacher LLM API (no GPT/Claude/OpenRouter demos):

1.Qwythos-9B served locally with sparkinfer on an RTX 5090
2.Autonomous agent loop (prof_dataset_gen) + real nsys / CUDA-L1 tools
3.Quality-filtered dataset → cuda-nsys-training
4.QLoRA SFT back into the same Qwythos base

Base pretrained weights still come from Empero/Qwen; the tool-use teaching signal is self-generated.

Path	Description
`adapter/`	PEFT LoRA (`r=16`, α=32)
`gguf/Qwythos-9B-nsys-SFT-Q4_K_M.gguf`	Merged no-MTP Q4KM (~5.3G) for sparkinfer
`gguf/Qwythos-9B-nsys-SFT-noMTP-BF16.gguf`	Merged no-MTP BF16 (~17G), non-quantized
`eval/`	Holdout A/B results: base Q4, SFT Q4, SFT BF16

Eval (first-turn XML tool calls, n=20)

Same holdout prompts from sft_eval.jsonl, temp=0.2, max_tokens=768:

Model	Runtime	parse_rate	valid tool names
Base Mythos Q4KM	sparkinfer	0%	0%
SFT Q4KM	sparkinfer	30%	30%
SFT BF16 (non-quantized)	llama.cpp (`enable_thinking=false`)	85%	85%

See eval/summary.json, eval/sft_bf16_llamacpp_n20_v2.json.

Serve

Q4 (sparkinfer):

bash

sparkinfer_server -m Qwythos-9B-nsys-SFT-Q4_K_M.gguf --tokenizer tokenizer.json --ctx 65536 --model-name qwythos-9b --port 8080

BF16 (llama.cpp):

bash

llama-server -m Qwythos-9B-nsys-SFT-noMTP-BF16.gguf -c 4096 -ngl 99 --jinja -fa on --port 8081
# API: chat_template_kwargs={"enable_thinking": false}

Load LoRA (transformers)

python

from transformers import Qwen3_5ForConditionalGeneration, AutoTokenizer
from peft import PeftModel

base_id = "empero-ai/Qwythos-9B-Claude-Mythos-5-1M"
tok = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True)
model = Qwen3_5ForConditionalGeneration.from_pretrained(base_id, torch_dtype="bfloat16", trust_remote_code=True)
model = PeftModel.from_pretrained(model, "gittensor-model-hub/Qwythos-9B-nsys-SFT", subfolder="adapter")

License