nphearum/Gemma-4-e2b-CodeX-Distill-v1-GGUF

Gemma-4-e2b-CodeX-Distill-v1-GGUF

A distilled code-focused variant of Gemma-4 e2b, optimized for efficient local inference using GGUF format. This model is designed for coding assistance, reasoning, and structured generation tasks, with optional “thinking” mode enabled via chat templates.

Example usage:

• —For text only LLMs: llama-cli -hf nphearum/Gemma-4-e2b-CodeX-Distill-v1-GGUF --jinja
• —For multimodal models: llama-mtmd-cli -hf nphearum/Gemma-4-e2b-CodeX-Distill-v1-GGUF --jinja

📦 Available Model Files

• —gemma-4-e2b-it.Q8_0.gguf — Quantized model (Q8_0 for high quality)
• —gemma-4-e2b-it.BF16-mmproj.gguf — Multimodal projection (required for full functionality)

🚀 Features

• —Strong code generation & reasoning (CodeX-style distillation)
• —Long context support (tested up to 131k tokens)
• —Optimized for llama.cpp
• —Supports structured chat templates (Jinja-based)
• —Optional “thinking mode” for better reasoning traces

🖥️ Running with llama.cpp

Make sure you’re using a recent build of llama.cpp with:

• —Flash Attention enabled
• —Jinja/chat template support compiled

Start Server

llama-server \
  -m gemma-4-e2b-it.Q8_0.gguf \
  --port 53281 \
  -c 131072 \
  --parallel 1 \
  --flash-attn on \
  --no-context-shift \
  -ngl -1 \
  --jinja \
  --chat-template-kwargs "{\"enable_thinking\": true}" \
  --mmproj gemma-4-e2b-it.BF16-mmproj.gguf

Key Flags Explained

• —-c 131072 → Enables long context (131k tokens)
• —--flash-attn on → Faster attention (requires compatible GPU)
• —-ngl -1 → Offload all layers to GPU
• —--jinja → Enables chat template rendering
• —--chat-template-kwargs → Activates thinking mode
• —--mmproj → Required for multimodal projection

Test Request

curl http://localhost:53281/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Write a Python function to reverse a linked list"}
    ]
  }'

🧠 Notes on Thinking Mode

When enable_thinking=true, the model may:

• —Produce intermediate reasoning steps
• —Improve structured problem solving
• —Slightly increase latency

Disable it if you need faster responses.

🦙 Running with Ollama

Important: ⚠️ Ollama Note for Vision Models, currently does not support separate mmproj files for vision models.

Create a Modelfile:

FROM ./gemma-4-e2b-it.Q8_0.gguf

PARAMETER num_ctx 131072
PARAMETER num_gpu -1
PARAMETER stop "<end_of_turn>"

TEMPLATE """{{ if .System }}<start_of_turn>system
{{ .System }}<end_of_turn>
{{ end }}{{ if .Prompt }}<start_of_turn>user
{{ .Prompt }}<end_of_turn>
<start_of_turn>model
{{ end }}"""

# Optional: enable reasoning-style outputs
SYSTEM "You are a highly capable coding assistant with strong reasoning ability."

Build & Run

ollama create gemma-4-codex -f Modelfile
ollama run gemma-4-codex

⚙️ Recommended Settings

⚠️ Limitations

• —Requires significant VRAM for full 131k context
• —Thinking mode increases latency
• —Multimodal projection file must match model variant

📜 License

Follow the original Gemma license and any additional terms from this distillation.

🙌 Credits

• —Base model: Google Gemma family
• —Distillation: Code-focused adaptation
• —Runtime: llama.cpp ecosystem