lil-lab/CoLMLM-Fact-Span-Annotator
CoLMLM-Fact-Span-Annotator
The fact-span annotator used to build the training corpora for **Co-LMLM: Continuous-Query Limited Memory Language Models**.
Co-LMLM is trained on text in which factual spans are wrapped in <FACT>...</FACT> tags. Producing those tags with a frontier LLM is far too expensive to run over a pretraining-scale corpus, so this model distills the span-marking half of that annotation into a ModernBERT-large token classifier: it labels every token O / B / I and thereby marks the fact spans directly.
It is the first stage of a two-stage annotation pipeline. The second stage, CoLMLM-Question-Generator, writes a question and a paraphrased answer for each span this model marks.
Model details
Usage
For more details and the full annotation pipeline, see the code repository:
👉 [github.com/lil-lab/Co-LMLM](https://github.com/lil-lab/Co-LMLM)
Standalone, the model is an ordinary token classifier. Decoding its BIO output back into character spans:
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer
model_id = "lil-lab/CoLMLM-Fact-Span-Annotator"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForTokenClassification.from_pretrained(
model_id, dtype=torch.bfloat16,
# attn_implementation="flash_attention_2", # faster, if flash-attn is installed
).eval()
text = "Marie Curie was born in Warsaw in 1867 and won two Nobel Prizes."
encoded = tokenizer(text, return_tensors="pt", return_offsets_mapping=True,
truncation=True, max_length=4096)
offsets = encoded.pop("offset_mapping")[0].tolist()
with torch.no_grad():
predictions = model(**encoded).logits[0].argmax(-1).tolist()
spans, start, end = [], None, None
for (char_start, char_end), label in zip(offsets, predictions):
if char_start == char_end: # special token
continue
tag = model.config.id2label[label]
if tag == "B":
if start is not None:
spans.append((start, end))
start, end = char_start, char_end
elif tag == "I" and start is not None:
end = char_end
else:
if start is not None:
spans.append((start, end))
start = None
if start is not None:
spans.append((start, end))
print([text[s:e].strip() for s, e in spans])
# ['Warsaw', '1867', 'two Nobel Prizes']This model is part of the **Co-LMLM** collection.
Citation
@misc{feldman2026colmlmcontinuousquerylimitedmemory,
title={Co-LMLM: Continuous-Query Limited Memory Language Models},
author={Yair Feldman and Linxi Zhao and Nathan Godey and Dongyoung Go and Yilun Hua and Kilian Q. Weinberger and Jennifer J. Sun and Yoav Artzi},
year={2026},
eprint={2607.07707},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2607.07707},
}