Datasetpublic

CohereLabs/BinaryVectorDB

Pre-build Binary Vector Databases Here we ship different pre-build Binary Vector databases. See the Github readme what Binary Vector Database is and why it can save you a lot of memory (and money) when you need to scale to tens or hundred millions of embeddings. Available Datasets We have Wikipedia available in all 300+ languages. See Files for the different files. These are simple .zip files you can download and unzip locally. Note for the English Wikipedia:… See the full description on the dataset page: https://huggingface.co/datasets/CohereLabs/BinaryVectorDB.

sourceHugging Faceupdated 3y agoView on Hugging Face

8likes205downloads

Dataset Card

Pre-build Binary Vector Databases

Here we ship different pre-build Binary Vector databases.

See the Github readme what Binary Vector Database is and why it can save you a lot of memory (and money) when you need to scale to tens or hundred millions of embeddings.

Comparison

Available Datasets

We have Wikipedia available in all 300+ languages.

See Files for the different files. These are simple .zip files you can download and unzip locally.

Note for the English Wikipedia: English Wikipedia is due to its size distributed as 2 pars: wikipedia-2023-11-en-part-1.zip wikipedia-2023-11-en-part-2.zip

Unzip both to be able to load the Binary Vector Database for the English Wikipedia (42M embeddings).