RiverRider/motherlode-code-search

Motherlode code search

Index a public GitHub repository inside the page and search it in plain English. The encoder, motherlode-code-small-en-v0.1, runs in your browser through transformers.js on WebGPU, or on WebAssembly where WebGPU is missing. The page asks GitHub for the files and the Hub for the model, and nothing else.

How it works:

• —The page lists the repository's source and text files through GitHub's API and fetches them from raw.githubusercontent.com: up to 800 files and 4 MB on WebGPU, source files first, and 40 files and 150 KB on WebAssembly, which is far slower: nanoGPT's 111 passages took 46.5 s there against 0.5 s on WebGPU, on an M2 Ultra in Chrome 154.
• —Files are cut as Black Window's engine cuts source: windows of about 700 characters at line ends, each labelled with the file, the nearest definition above it and its first line. Each window is embedded with CLS pooling, at 16-bit precision where the GPU supports it.
• —Search ranks windows by cosine after the model's shipped mean (centring/mean_shipped.json in the model repository) is subtracted from every vector, the query's included.

The same encoder is the default reader of Sunstone, which indexes the folders you open in VS Code, and of blackwindow.xyz.

This page is Apache-2.0. The model is licensed BUSL-1.1; its card states the terms.