Modelpublic

mkd-hika/keural-vision-encoder-poc

sourceHugging Facemitupdated 3mo agoView on Hugging Face

0likes22downloads

Model Card

👁️ Keural Vision Encoder <sub>(PoC · V0.1)</sub>

A 24.7M-parameter vision encoder trained _from scratch_ — no pretrained backbone, no CLIP weights. Structured tokens that plug into any LLM.

<br>

<br>

The encoder (CNN Stem → ATB → Spatial Transformer) feeds the LevelAware Projector → Mistral-7B in the full [Keural VLM](https://huggingface.co/mkd-hika/keural-vlm-poc).

</div>

✨ Key Innovations

🎯 Adaptive Token Budget (ATB) Tokenization — token count is a runtime parameter. Tokens are allocated to information-dense regions: a blank wall gets fewer, a dense document gets more.

python

out = encoder(image, token_budget=64)    # cheap
out = encoder(image, token_budget=256)   # default
out = encoder(image, token_budget=1024)  # full fidelity

🪜 Hierarchical Concept Tokenization (HCT) — every token carries a semantic level tag: global (whole-image), region (object-scale), or detail (fine-grained).

python

out = encoder(image)
print(out.level_ids)  # tensor of {0=global, 1=region, 2=detail}

⚙️ Model Specifications

Property	Value
Parameters	24.7M (trained from scratch)
Architecture	CNN Stem + ATB Tokenizer + 12-layer Spatial Transformer
Embedding dim	384
Token budget	256 (default) · runtime-adjustable
Token split	5% global / 25% region / 70% detail
Precision	bfloat16
Training data	CC3M + CC12M (~6.9M image-text pairs)
Hardware	1× RTX 5090 (32 GB VRAM)

🎨 Saliency & Token Placement

The ATB tokenizer concentrates tokens on salient regions. Left → right: original · saliency heatmap · token placement (global / region / detail).

🧪 Training

Phase 1 — Vision Encoder Pretraining ✅ COMPLETE. Trained from scratch for ~75,000 steps on CC3M + CC12M (~6.9M image–text pairs), SigLIP-style contrastive objective, 1× RTX 5090. The frozen encoder is then integrated into the full VLM via LevelAwareProjector (384 → 2048 → 4096) → Mistral-7B-Instruct-v0.3 (4-bit NF4 QLoRA), fine-tuned with SFT (LLaVA-Instruct-150K, 30K steps).

📊 Benchmark Results — SFT-30K

Downstream VLM benchmarks using this encoder (frozen) + projector + Mistral-7B at the SFT-30K checkpoint (supervised fine-tuning, 30,000 steps; before DPO). Evaluated on 1,000 samples each where applicable. The Keural encoder is 12.4× smaller than LLaVA's CLIP encoder (307M).

Benchmark	Keural SFT-30K	LLaVA 1.5 <sub>(307M enc)</sub>	LLaVA 1.6 <sub>(307M enc)</sub>
VQAv2 Accuracy	12.9%	78.5%	81.8%
POPE F1	66.9%	85.9%	86.5%
MME Total Score	704.3	1510.7	1519.3
TextVQA Accuracy	0.8%	58.2%	64.9%
ScienceQA Accuracy <sub>(img)</sub>	39.7%	66.8%	70.6%

These are the SFT-30K numbers (measured from outputs/eval/sft_30k/). Applying DPO alignment (RLHF-V, 3K steps) improves every metric — e.g. VQAv2 → 43.6%, ScienceQA → 53.7%, MME → 838.8. Full SFT+DPO results, the comparison chart, and the complete VLM live on [mkd-hika/keural-vlm-poc](https://huggingface.co/mkd-hika/keural-vlm-poc).

⚡ Adaptive Token Budget — latency vs budget

Encode latency vs token budget — trade visual detail for speed at runtime, no retraining.

📐 Output Contract

Every forward pass returns a KeuralEncoderOutput:

Field	Shape	Description
`tokens`	`(B, N, 384)`	Token vectors
`attention_mask`	`(B, N)`	1 = real token, 0 = padding
`level_ids`	`(B, N)`	0=global, 1=region, 2=detail
`spatial_metadata`	`(B, N, 4)`	Per-token (cx, cy, w, h) in [0,1]
`saliency_scores`	`(B, N)`	Allocator confidence per token
`pooled`	`(B, 384)`	Single-vector image summary
`token_budget_used`	`(B,)`	Actual tokens used per image

🚀 Usage

python

import torch
from architecture.encoder import KeuralVisionEncoder
from keural_config import KeuralConfig

cfg = KeuralConfig.from_yaml("configs/keural_tiny_poc.yaml")
model = KeuralVisionEncoder.from_pretrained("mkd-hika/keural-vision-encoder-poc")
model.eval()

from PIL import Image
from torchvision import transforms

transform = transforms.Compose([
    transforms.Resize((256, 256)),
    transforms.ToTensor(),
    transforms.Normalize([0.5], [0.5]),
])

image = Image.open("image.jpg").convert("RGB")
x = transform(image).unsqueeze(0)  # (1, 3, 256, 256)

with torch.no_grad():
    out = model(x)

print(out.tokens.shape)        # (1, 256, 384)
print(out.level_ids[0, :8])    # [0, 0, 1, 1, 1, 2, 2, 2]
print(out.pooled.shape)        # (1, 384)

🗺️ Roadmap

Phase	Params	Hardware	Status
PoC (this model)	24.7M	1× RTX 5090	✅ Phase 1 complete
Mid-level	~230M	8× H100 80 GB	🔜 Planned
Commercial	~1.1B	64× H100 80 GB	🔮 Future

The mid-level model will use knowledge distillation from SigLIP-400M (natural images) and InternViT-300M (documents/OCR). Teachers are discarded after training — not part of the final model.

📝 Citation

bibtex

@misc{keural_vision_encoder_2026,
  title  = {Keural Vision Encoder: Content-Adaptive Vision Encoding via Saliency-Guided Token Budgets},
  author = {Barki, Hika and MKD Co., Ltd.},
  year   = {2026},
  note   = {V0.1 — Phase 1 complete},
}

📄 License

Code: MIT. Model weights trained on CC3M + CC12M — data licenses apply.

MKD Co., Ltd. — 2026

</div>