Jwuthrich/selfjev-4b-vision-4bit

SelfJev-4B Vision, 4-bit (NF4)

Structured decisions from text. A 4B backbone, quantized to run on an 8 GB GPU.

SelfJev-4B Vision with its LoRA merged into Qwen3.5-4B and the weights quantized to 4-bit (NF4) with bitsandbytes (NF4 4-bit with bf16 compute). It answers yes/no, one-choice, scored and multi-select questions over the options you supply and returns probabilities, through the SelfJev engine and API. The checkpoint is 3.3 GB (the full merged model is about 9.3 GB).

Quickstart

Needs an NVIDIA GPU (CUDA) and selfjev 0.4.0 or later.

pip install "selfjev[serve,gpu,quant]"
selfjev serve --quantized-model Jwuthrich/selfjev-4b-vision-4bit

--quantized-model takes this repository (or a local folder) in place of --adapter. The first start also downloads the Qwen3.5-4B tokenizer; the vision tower is only fetched from the base model if you send an image. The HTTP API is the same as the full model's: API guide.

Quality

Scored on all 3,657 questions of SelfJev Decision Bench through the native tree engine on an A10G whose memory was capped at 7.2 GiB (the usable memory of an 8 GB card).

The suites are authored with AI-checked labels and have informed research decisions, so read them as a development benchmark. The full-model rows are the vision release's own reports; the compact bf16 figure was re-run on the same A10G. Reports: reports/quant.

Memory and speed

Peak GPU memory for one question and the median time of one request (A10G, --max-batch-tokens 4096, selfjev 0.4.2; memory is what the tensors allocate):

This is the build for an 8 GB card: texts up to 16K tokens fit with room to spare, and speed matches the full model on this GPU.

Limits

• —Scored on text only. Images use the base model's vision tower in bf16 (not quantized, loaded on the first image); the quantized path has not been scored on images.
• —Texts above 16K tokens were not validated (the model was trained on texts up to 16K).
• —Quantization costs accuracy: about 1 point on the pooled benchmark and 1.4 on text decisions, mostly in multi-select questions where the exact set of answers must match. The numbers above are the measurement; there is no calibration step to recover it.
• —Same license and intended use as the full model; built from Qwen3.5-4B (Apache 2.0).

How this was made

selfjev quantize --adapter weights/selfjev_4b_vision --bits 4bit: the adapter is merged into the pinned Qwen3.5-4B (851bf6e) in bf16, then loaded with bitsandbytes NF4 4-bit with bf16 compute and saved. Source: Jwuthri/SelfJev.