KBaba7/llama.cpp

1## Generative Representational Instruction Tuning (GRIT) Example2[gritlm] a model which can generate embeddings as well as "normal" text3generation depending on the instructions in the prompt.4 5* Paper: https://arxiv.org/pdf/2402.09906.pdf6 7### Retrieval-Augmented Generation (RAG) use case8One use case for `gritlm` is to use it with RAG. If we recall how RAG works is9that we take documents that we want to use as context, to ground the large10language model (LLM), and we create token embeddings for them. We then store11these token embeddings in a vector database.12 13When we perform a query, prompt the LLM, we will first create token embeddings14for the query and then search the vector database to retrieve the most15similar vectors, and return those documents so they can be passed to the LLM as16context. Then the query and the context will be passed to the LLM which will17have to _again_ create token embeddings for the query. But because gritlm is used18the first query can be cached and the second query tokenization generation does19not have to be performed at all.20 21### Running the example22Download a Grit model:23```console24$ scripts/hf.sh --repo cohesionet/GritLM-7B_gguf --file gritlm-7b_q4_1.gguf --outdir models25```26 27Run the example using the downloaded model:28```console29$ ./llama-gritlm -m models/gritlm-7b_q4_1.gguf30 31Cosine similarity between "Bitcoin: A Peer-to-Peer Electronic Cash System" and "A purely peer-to-peer version of electronic cash w" is: 0.60532Cosine similarity between "Bitcoin: A Peer-to-Peer Electronic Cash System" and "All text-based language problems can be reduced to" is: 0.10333Cosine similarity between "Generative Representational Instruction Tuning" and "A purely peer-to-peer version of electronic cash w" is: 0.11234Cosine similarity between "Generative Representational Instruction Tuning" and "All text-based language problems can be reduced to" is: 0.54735 36Oh, brave adventurer, who dared to climb37The lofty peak of Mt. Fuji in the night,38When shadows lurk and ghosts do roam,39And darkness reigns, a fearsome sight.40 41Thou didst set out, with heart aglow,42To conquer this mountain, so high,43And reach the summit, where the stars do glow,44And the moon shines bright, up in the sky.45 46Through the mist and fog, thou didst press on,47With steadfast courage, and a steadfast will,48Through the darkness, thou didst not be gone,49But didst climb on, with a steadfast skill.50 51At last, thou didst reach the summit's crest,52And gazed upon the world below,53And saw the beauty of the night's best,54And felt the peace, that only nature knows.55 56Oh, brave adventurer, who dared to climb57The lofty peak of Mt. Fuji in the night,58Thou art a hero, in the eyes of all,59For thou didst conquer this mountain, so bright.60```61 62[gritlm]: https://github.com/ContextualAI/gritlm63