robro612/scifact_answerai_colbert_small

scifactansweraicolbert_small

Multi-vector (late-interaction) embeddings of BEIR scifact (beir/scifact/test), encoded with [lightonai/answerai-colbert-small-v1](https://huggingface.co/lightonai/answerai-colbert-small-v1) at revision e507cd12947a2b4b52201d150967df3c19a90590.

Source data: ir_datasets beir/scifact/test (irdatasets 0.6.3), which downloads [scifact.zip](https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/scifact.zip) (md5 `5f7d1de60b170fc8027bb7898e2efca1`). BEIR also publishes this corpus on the Hub as [`BeIR/scifact`](https://huggingface.co/datasets/BeIR/scifact), whose card gives this dataset's license; the data here was loaded through irdatasets, not from that repo. Document, query and qrel ids are the source's own ids, unchanged.

Every document is one variable-length set of 96-d vectors; every query is one variable-length set of 96-d vectors. Documents and queries are stored at different precisions (fp16 and fp32 respectively), see Encoding.

Files

All positional indices (the gt_top*.tsv files, and the row order of every .npy file) refer to the order of doc_ids.npy and queries_ids.npy. Reordering either file invalidates the ground truth.

Statistics

Encoding

Ground truth: gt_top1000.tsv and gt_top100.tsv

Exact brute-force MaxSim top-1000 per query over the full corpus, from the vectors in this repo. gt_top100.tsv holds the first 100 ranks per query of the same lists (the original layout of these exports).

No header; tab-separated qidx docidx rank score:

• —qidx: 0-based row into queries_ids.npy / queries.npy
• —docidx: 0-based position into doc_ids.npy / doclens.npy
• —rank: 1-based, descending score
• —score: sum over the query's query_lens[qidx] vectors of max over the document's vectors of the dot product, computed in fp32 with the fp16 document vectors upcast to fp32. Expansion vectors are included in the sum. Printed to 6 decimals.

Retrieval quality

Sanity check of the vectors, not a leaderboard number: gt_top1000.tsv (exact MaxSim over the full corpus) scored against qrels.test.tsv with ir_measures.

Loading

import numpy as np

documents = np.load("documents.npy", mmap_mode="r")      # [n_tokens, 96] float16
doclens = np.load("doclens.npy")                         # [n_docs] int32
offsets = np.concatenate([[0], np.cumsum(doclens)])
doc_ids = np.load("doc_ids.npy")                         # [n_docs] str

def document(i):
    return documents[offsets[i]:offsets[i + 1]]           # [doclens[i], 96]

queries = np.load("queries.npy")                         # [n_queries, 32, 96] float32
query_lens = np.load("query_lens.npy")                   # [n_queries] int32
query_ids = np.load("queries_ids.npy")                   # [n_queries] str

def query(j):
    return queries[j, :query_lens[j]]                     # [query_lens[j], 96]

def maxsim(q, d):
    return (q @ d.astype(np.float32).T).max(axis=1).sum()

Validation

Checks run by the exporter on the files exactly as written here:

• —✅ file set — missing=[] extra=[]
• —✅ documents.npy dtype/shape — <f2 (1221388, 96)
• —✅ doclens.npy dtype/shape — <i4 (5183,)
• —✅ doc_ids.npy is a string array — <U9 (5183,)
• —✅ queries.npy dtype/shape — <f4 (300, 32, 96)
• —✅ query_lens.npy dtype/shape — <i4 (300,)
• —✅ queries_ids.npy is a string array — <U4 (300,)
• —✅ sum(doclens) == n_tokens — 1221388 vs 1221388
• —✅ no empty documents — min doclen 50
• —✅ len(doc_ids) == len(doclens) == corpus size — 5183, 5183, 5183
• —✅ doc_ids unique
• —✅ query arrays aligned — 300, 300, 300
• —✅ doc and query dim agree — 96 / 96
• —✅ token_ids.npy dtype/shape — <u4 (1221388,)
• —✅ document vectors unit-norm (100k sample) — norm range [0.9994, 1.0006]
• —✅ query vectors unit-norm — norm range [1.000000, 1.000000]
• —✅ all vectors finite
• —✅ gt_top100.tsv has k rows per query — 30000 rows, k=100
• —✅ gt_top100.tsv rows grouped by qidx with ranks 1..k and descending scores
• —✅ gt_top100.tsv indices in range
• —✅ gt_top1000.tsv has k rows per query — 300000 rows, k=1000
• —✅ gt_top1000.tsv rows grouped by qidx with ranks 1..k and descending scores
• —✅ gt_top1000.tsv indices in range
• —✅ gttop100.tsv is the first 100 ranks of gttop1000.tsv

Provenance