commoncrawl/web-graph-knn

Web graph kNN spot-check

Type a hostname, get its nearest neighbours in the learned 128-d Common Crawl host-graph embedding. Companion to the 2D viewer: that one shows where hosts land after a UMAP projection, this one answers who is actually nearest in the real space.

Running locally

uv sync --extra faiss --extra viewer
wgl knn build --hf-repo commoncrawl/web-graph-embeddings --shards 8 --row-groups 1 \
    --out data/knn/cc_deg8_poc
WGL_KNN_INDEX=data/knn/cc_deg8_poc uv run python knn-viewer/app.py

Running as a Space

The app reads its index from $WGL_KNN_INDEX, defaulting to /data — the mount path for a HuggingFace volume. Point a private bucket at it:

hf buckets create commoncrawl/web-graph-knn-index --private
hf sync data/knn/cc_deg8_full hf://buckets/commoncrawl/web-graph-knn-index/cc_deg8 --delete
hf spaces volumes set commoncrawl/web-graph-knn \
    -v hf://buckets/commoncrawl/web-graph-knn-index:/data

app.py needs wgl.knn.index, which pulls only numpy, faiss and tldextract, so wgl is vendored next to app.py on the Space rather than installed. Only the import closure of app.py ships — 13 modules of 100, so the Space holds the code it runs and not the training tree. KnnIndex stays a wgl module rather than being copied into the app on purpose: it reads the artifact format wgl.knn.build writes, wgl knn query reads the same one, and tests/test_knn_index.py round-trips build against load to keep them honest. Deploying does the vendoring for you:

make knn-diff     # which files the Space is missing or has stale (changes nothing)
make knn-deploy   # upload the app + vendored src/wgl, skipping what the Space already has
make knn-serve    # run it locally against $KNN_INDEX first