kylebrodeur/embedding-eval-results

Embedding Eval Results

The committed results from a real embedding-model benchmark that embarrassed a leaderboard's recommendation.

What's in here

Four CSV files representing four evaluation runs on a personal Obsidian vault:

What the data shows

Round 1 on 54 notes: all 7 models looked great. recall@5 above 90% for every candidate. The "winner" by leaderboard-style ranking was nomic-embed-text.

Stage 1 on 995 notes: the ranking inverted. The mid-pack model embeddinggemma took first on every metric. The round-one "winner" sank to mid-pack. The default the system had been quietly using, all-minilm, dropped to last.

Same code. Same models. Same scoring. The only thing that changed was the size of the pile.

Columns

Each CSV has the same columns:

name,provider,model,dim,recall@1,recall@5,recall@10,mrr@10,q_latency_ms,approx_cost_usd

• —name: human-readable label (matches the model except for the legacy all-minilm row, which is annotated)
• —provider: ollama (local) or modal (a GPU embedding service hosted on Modal)
• —model: the actual model name passed to the embedder
• —dim: embedding dimension
• —recall@1, recall@5, recall@10: fraction of queries where the source note landed in the top-k
• —mrr@10: mean reciprocal rank
• —q_latency_ms: query-side embedding latency (note embedding happens separately)
• —approx_cost_usd: zero for local models; the cheapest paid model cost roughly a few cents per run

Source

The full eval harness is in the modal-embedding-server repo at eval/run_eval.py: multi-provider (Ollama local + Cloud, Hugging Face Inference, a Modal GPU service), a uv script with no setup, and the harness that produced these numbers. The full eval pipeline is part of a five-repo Modal ecosystem from the same author; the operator CLI for the fleet is modal-toolkit.

The simplified standalone version of the harness, written for the blog post, is at kylebrodeur/embed-eval-on-your-vault. It is a single-file Python script with no dependencies beyond python3 that runs against any vault of .md or .txt notes.

The companion blog post is "Whose data picked the winner?".

Reproducibility

Each CSV is timestamped (UTC). The harness commits the result on every run. Re-running with the same model/provider/dataset should produce the same numbers within rounding. Embedding models do drift across versions, so if you re-run after a model bump, expect ±1-2% on the metrics.

Citation

If you use these numbers in another piece, cite the model row and the run timestamp. The full provenance is in the modal-embedding-server repo commit log (eval/ directory, 2026-06-18 to 2026-06-20 range).

License

MIT. Numbers are facts; the framing is the author's.