webAI-Official/webAI-ColVec1-4b

webAI-Official/webAI-ColVec1-4b

!!! A new version of this model is available here [webAI-ColVec1.1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b) !!!

⚡ Summary

webAI-Official/webAI-ColVec1-4b is a state-of-the-art ColBERT-style multimodal embedding model based on [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B). It maps text queries, visual documents (images, PDFs) into aligned multi-vector embeddings.

The model has been fine-tuned on a merged multimodal dataset of ~2M question-image pairs, including DocVQA, PubTables-1M, TAT-QA, ViDoRe-ColPali-Training, VDR Multilingual, VisRAG-Ret-Train-In-domain-data, VisRAG-Ret-Train-Synthetic-data and proprietary domain-specific synthetic data

The datasets were filtered, balanced, and merged to produce a comprehensive training set optimized for multilingual, multimodal retrieval and document-image understanding. The model achieves competitive performance across ViDoRe V1 & V3 (English and multilingual).

🛠️ Model Specifications

Key Properties

• —Unified Encoder (Single-Tower): A single shared language model processes both images and text. Images are converted into visual tokens via a vision encoder and injected into the token stream, no separate dual encoders.

• —Projection Head: A single linear layer projects final hidden states → compact embedding space (hidden_size → 640 dim).
• —No activation
• —Fully trained
• —Replaces LM head for retrieval

• —Multi-Vector Representation: Each token becomes an embedding → enables fine-grained token-level matching instead of single-vector pooling.

📊 Evaluation Results

We report results on the ViDoRe benchmark suite. The tables below summarize the image-modality accuracy of webAI-ColVec1-4b on the ViDoRe V1 and V3 benchmarks, alongside other webAI ColVec1 models. Note that (M)MTEB leaderboards use Borda ranking. Each task acts like a voter that ranks models based on how well they perform. Models earn more points when they rank higher on a task. The model with the most total points across all tasks gets the top overall rank.

ViDoRe V3 (NDCG@10)

ViDoRe V1 (NDCG@5)

💻 Usage

The processor exposes three primary methods for encoding inputs and computing retrieval scores.

process_images(images, max_length=None)

Encodes a batch of document images into model-ready tensors. Pass the result directly to the model with **batch.

batch = processor.process_images(images=pil_images)
batch = {k: v.to(device) for k, v in batch.items()}
embeddings = model(**batch)  # shape: (B, seq_len, embed_dim)

process_queries(texts, max_length=None)

Encodes a batch of text queries into model-ready tensors.

batch = processor.process_queries(texts=["What is the revenue for Q3?"])
batch = {k: v.to(device) for k, v in batch.items()}
embeddings = model(**batch)  # shape: (B, seq_len, embed_dim)

score_multi_vector(qs, ps, batch_size=128, device=None)

Computes ColBERT-style MaxSim late-interaction scores between a list of query embeddings and a list of passage (document) embeddings. For each query token, the maximum dot product across all passage tokens is found; these maxima are summed to produce a single scalar score per (query, passage) pair.

Returns a torch.Tensor of shape (n_queries, n_passages) on CPU in float32. Higher scores indicate greater relevance.

scores = processor.score_multi_vector(query_embeddings, doc_embeddings)
# scores[i, j] is the relevance of document j to query i
best_doc_per_query = scores.argmax(dim=1)

Prerequisites

We strongly suggest flash-attn to be installed. If not, please change to attention_impl="sdpa"

Currently we only support torch==2.8.0, for higher pytorch version, please build flash attention manually, otherwise performance throughput could be low. Also, Note that torch==2.8.0 supports Python Versions: >= 3.9 and <= 3.13.

pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128
pip install transformers pillow requests
pip install flash-attn --no-build-isolation

Inference Code

import torch
from transformers import AutoModel, AutoProcessor
from PIL import Image, UnidentifiedImageError
import requests
from io import BytesIO

# Configuration
MODEL_ID = "webAI-Official/webAI-ColVec1-4b"
DTYPE = torch.bfloat16
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

# Load Model & Processor
processor = AutoProcessor.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
)

model = AutoModel.from_pretrained(
    MODEL_ID,
    dtype=DTYPE,
    attn_implementation="flash_attention_2",
    trust_remote_code=True,
    device_map=DEVICE,
).eval()

# Sample Data
queries = [
    "Retrieve the city of Singapore",
    "Retrieve the city of Beijing"
]
docs = [
    "https://upload.wikimedia.org/wikipedia/commons/2/27/Singapore_skyline_2022.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/6/61/Beijing_skyline_at_night.JPG"
]

def load_image(url: str) -> Image.Image:
    # Some CDNs (e.g., Wikimedia) expect a browser-like UA to avoid 403s.
    for headers in ({}, {"User-Agent": "Mozilla/5.0 (compatible; ColQwen3-demo/1.0)"}):
        resp = requests.get(url, headers=headers, timeout=10)
        if resp.status_code == 403:
            continue
        resp.raise_for_status()
        try:
            return Image.open(BytesIO(resp.content)).convert("RGB")
        except UnidentifiedImageError as e:
            raise RuntimeError(f"Failed to decode image from {url}") from e
    raise RuntimeError(f"Could not fetch image (HTTP 403) from {url}; try downloading locally and loading from file path.")

# Helper Functions
def encode_queries(texts, batch_size=8):
    outputs = []
    for start in range(0, len(texts), batch_size):
        batch = processor.process_queries(texts=texts[start : start + batch_size])
        batch = {k: v.to(DEVICE) for k, v in batch.items()}
        with torch.inference_mode():
            embeddings = model(**batch)
            vecs = embeddings.to(torch.bfloat16).cpu()
        outputs.extend(vecs)
    return outputs

def encode_docs(urls, batch_size=4):
    pil_images = [load_image(url) for url in urls]
    outputs = []
    for start in range(0, len(pil_images), batch_size):
        batch_imgs = pil_images[start : start + batch_size]
        features = processor.process_images(images=batch_imgs)
        features = {k: v.to(DEVICE) if isinstance(v, torch.Tensor) else v for k, v in features.items()}
        with torch.inference_mode():
            embeddings = model(**features)
            vecs = embeddings.to(torch.bfloat16).cpu()
        outputs.extend(vecs)
    return outputs

# Execution
query_embeddings = encode_queries(queries)
doc_embeddings = encode_docs(docs)

# MaxSim Scoring
scores = processor.score_multi_vector(query_embeddings, doc_embeddings)
print(scores)

⚖️ Strengths & Limitations

Strengths

• —Performance: State of the art retrieval performance on ViDoRe V1 & V3 dataset with excellent performance on multimodal document retrieval.
• —Complex Layouts: Excellent handling of chart-rich PDFs, domain-specific documents.
• —End-to-end Retrieval: Capable of OCR-free retrieval on unseen multimodal documents without using an intermediate vision LLM to generate summary for retrieval.
• —Multilingualism: Strong performance on non-English document inputs.

Limitations

• —Storage Cost: Still larger than single‑vector baselines despite the smaller token dimension.

License & Data

LICENSE

📚 Citation

If you use this model, please cite:

@misc{webAI-ColVec1,
  title={webAI-ColVec1: Late-Interaction Multi-Vector Embedding Model for Visual Document Retrieval},
  author={webAI},
  year={2026},
  url={https://huggingface.co/webAI-Official/webAI-ColVec1-4b}
}