AbstractPhil/bulk-cc12m-features

bulk-cc12m-features — ten teacher towers over CC12M, plus their consensus

Precomputed image-tower features for 10,968,539 CC12M images (all 2,176 shards of pixparse/cc12m-wds) from ten independent teacher extractions — eight CLIP variants across three pretraining corpora and two model scales, SigLIP, and DINOv3 — plus one derived consensus target. About 110 million feature vectors, roughly 130 GPU-hours of extraction, so that a student can be distilled against any of these teachers (or a consensus of them) with zero teacher inference at training time.

Captions are included with the clip_b16_laion2b config and join to every other tower by key.

The towers

`consensus_gpa` is a derived target, not an extraction. It is the generalized-Procrustes mean shape of the five B-tier CLIP towers (clip_b16_laion2b, clip_b32_openai, clip_b32_laion2b, clip_b32_datacomp, clip_b16_openai): each tower L2-normalized, then five iterations of {orthogonally align every tower to the running mean on a fixed 10,000-row subsample via fp64 SVD, rotate the full tower, re-mean, renormalize}, converging at mean alignment ≈ 0.91. Rows are already unit norm. This is the exact tensor the published consensus-distilled students were trained against — see clip-vitb-mini-distilled. Note that the mean has no privileged frame: a student trained on it lands in the consensus frame, not any teacher's, which is why that model ships a fitted rotation alongside its weights.

`dinov3_l16` is different in kind. DINOv3 has no projection head and no text tower: its 1024-d vector is the CLS hidden state, not a language-aligned projection. It is a full, valid extraction over the same images, but it is not a drop-in member of a CLIP consensus and has no text side to evaluate against. Fusing it with the CLIP towers needs an explicit cross-space alignment (learned projectors, or a whitened-Procrustes fit), not a plain average.

All extracted towers are stored fp16 and UNNORMALIZED — L2-normalize at load if you want unit vectors. (consensus_gpa is the exception: it is already unit norm by construction.)

Join by key, never by position

Every tower covers the identical key set, shard for shard (verified on first, middle and last shards). But row order differs between towers: extraction ran decode-completion-ordered, so shard n of two towers holds the same 5,041 images in different order.

# correct
order = {k: i for i, k in enumerate(tower_a["keys"])}
b_aligned = tower_b["emb"][[order[k] for k in tower_b["keys"]]]

# WRONG — silently misaligns every row
pairs = zip(tower_a["emb"], tower_b["emb"])

Forms

The ten extracted towers carry all three forms. consensus_gpa is derived rather than extracted, so it has no per-wds-shard form — parquet and concat only.

from datasets import load_dataset
ds = load_dataset("AbstractPhil/bulk-cc12m-features",
                  "clip_l14_laion2b", split="train", streaming=True)
row = next(iter(ds))          # {'key', 'features': [768 floats]}

Parquet stores features as fp32 lists carrying fp16 precision (the convention of the companion COCO bank). The .pt shards and concats load with weights_only=True.

Exact preprocessing (the part banks usually leave undocumented)

Feature parity requires bit-level preprocessing agreement, and it is per tower, not per family — every extraction was gated before spending compute: the fast decode path against the tower's reference processor, and reproduction of that tower's stored COCO features from the companion bank, both at cosine ≥ 0.9999.

Other constants that matter: plain GELU, not QuickGELU, for the CLIP towers (the openclip default for some checkpoints differs, and the mismatch reads as cosine ≈ 0.975 against stored features — close enough to miss); fp32 compute; readout `getimagefeatures` (CLS → visual projection) for CLIP/SigLIP and **CLS** for DINOv3. Two checkpoints ship openclip weights only (LAION B/16, DataComp B/32) and were converted deterministically to transformers format; hub-hosted conversions of openclip checkpoints do **not** reproduce these banks — convert from the openclip original.

Provenance and quality

• —Extracted 2026-07-27..31. The primary tower ran as a single 8.1-hour streaming pass (download → decode → embed → discard; the source imagery was never resident) at ~376 img/s; the remaining towers followed on rented A40s.
• —Zero decode errors and zero download failures across all 2,176 shards of the primary pass; per-shard counts in each tower's ledger.jsonl.
• —Full-bank integrity sweep on the primary: every file loads, key/caption/ embedding counts agree with the ledger, no non-finite values, feature norms stable across the run.
• —Shards hold 5,040–5,041 samples each.

Licensing note

These are derived features and captions, not images. CC12M imagery remains the property of its owners; captions and the underlying URL list come from Google's Conceptual 12M under its stated terms, and the image snapshot mirrored by pixparse/cc12m-wds. Intended for research use.

References

• —Changpinyo et al., Conceptual 12M — https://arxiv.org/abs/2102.08981
• —Radford et al., CLIP — https://arxiv.org/abs/2103.00020
• —Cherti et al., Reproducible scaling laws for contrastive language-image learning (OpenCLIP) — https://arxiv.org/abs/2212.07143
• —Gadre et al., DataComp — https://arxiv.org/abs/2304.14108
• —Zhai et al., SigLIP — https://arxiv.org/abs/2303.15343
• —Siméoni et al., DINOv3 — https://arxiv.org/abs/2508.10104
• —Companion COCO bank (34 towers): AbstractPhil/bulk-coco-features
• —First consumer of this bank: AbstractPhil/clip-vitb-mini-distilled