KBaba7/llama.cpp

1# LLaVA2 3Currently this implementation supports [llava-v1.5](https://huggingface.co/liuhaotian/llava-v1.5-7b) variants,4as well as llava-1.6 [llava-v1.6](https://huggingface.co/collections/liuhaotian/llava-16-65b9e40155f60fd046a5ccf2) variants.5 6The pre-converted [7b](https://huggingface.co/mys/ggml_llava-v1.5-7b)7and [13b](https://huggingface.co/mys/ggml_llava-v1.5-13b)8models are available.9For llava-1.6 a variety of prepared gguf models are available as well [7b-34b](https://huggingface.co/cmp-nct/llava-1.6-gguf)10 11After API is confirmed, more models will be supported / uploaded.12 13## Usage14Build with cmake or run `make llama-llava-cli` to build it.15 16After building, run: `./llama-llava-cli` to see the usage. For example:17 18```sh19./llama-llava-cli -m ../llava-v1.5-7b/ggml-model-f16.gguf --mmproj ../llava-v1.5-7b/mmproj-model-f16.gguf --image path/to/an/image.jpg20```21 22**note**: A lower temperature like 0.1 is recommended for better quality. add `--temp 0.1` to the command to do so.23**note**: For GPU offloading ensure to use the `-ngl` flag just like usual24 25## LLaVA 1.526 271. Clone a LLaVA and a CLIP model ([available options](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md)). For example:28 29```sh30git clone https://huggingface.co/liuhaotian/llava-v1.5-7b31 32git clone https://huggingface.co/openai/clip-vit-large-patch14-33633```34 352. Install the required Python packages:36 37```sh38pip install -r examples/llava/requirements.txt39```40 413. Use `llava_surgery.py` to split the LLaVA model to LLaMA and multimodel projector constituents:42 43```sh44python ./examples/llava/llava_surgery.py -m ../llava-v1.5-7b45```46 474. Use `convert_image_encoder_to_gguf.py` to convert the LLaVA image encoder to GGUF:48 49```sh50python ./examples/llava/convert_image_encoder_to_gguf.py -m ../clip-vit-large-patch14-336 --llava-projector ../llava-v1.5-7b/llava.projector --output-dir ../llava-v1.5-7b51```52 535. Use `examples/convert_legacy_llama.py` to convert the LLaMA part of LLaVA to GGUF:54 55```sh56python ./examples/convert_legacy_llama.py ../llava-v1.5-7b --skip-unknown57```58 59Now both the LLaMA part and the image encoder are in the `llava-v1.5-7b` directory.60 61## LLaVA 1.6 gguf conversion621) First clone a LLaVA 1.6 model:63```console64git clone https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b65```66 672) Install the required Python packages:68 69```sh70pip install -r examples/llava/requirements.txt71```72 733) Use `llava_surgery_v2.py` which also supports llava-1.5 variants pytorch as well as safetensor models:74```console75python examples/llava/llava_surgery_v2.py -C -m ../llava-v1.6-vicuna-7b/76```77- you will find a llava.projector and a llava.clip file in your model directory78 794) Copy the llava.clip file into a subdirectory (like vit), rename it to pytorch_model.bin and add a fitting vit configuration to the directory:80```console81mkdir vit82cp ../llava-v1.6-vicuna-7b/llava.clip vit/pytorch_model.bin83cp ../llava-v1.6-vicuna-7b/llava.projector vit/84curl -s -q https://huggingface.co/cmp-nct/llava-1.6-gguf/raw/main/config_vit.json -o vit/config.json85```86 875) Create the visual gguf model:88```console89python ./examples/llava/convert_image_encoder_to_gguf.py -m vit --llava-projector vit/llava.projector --output-dir vit --clip-model-is-vision90```91- This is similar to llava-1.5, the difference is that we tell the encoder that we are working with the pure vision model part of CLIP92 936) Then convert the model to gguf format:94```console95python ./examples/convert_legacy_llama.py ../llava-v1.6-vicuna-7b/ --skip-unknown96```97 987) And finally we can run the llava cli using the 1.6 model version:99```console100./llama-llava-cli -m ../llava-v1.6-vicuna-7b/ggml-model-f16.gguf --mmproj vit/mmproj-model-f16.gguf --image some-image.jpg -c 4096101```102 103**note** llava-1.6 needs more context than llava-1.5, at least 3000 is needed (just run it at -c 4096)104**note** llava-1.6 greatly benefits from batched prompt processing (defaults work)105 106## llava-cli templating and llava-1.6 prompting107 108llava-1.5 models all use the same vicuna prompt, here you can just add your image question like `-p "Provide a full description."`109For llava-1.5 models which are not vicuna (mistral and Yi) you need to adapt system prompt as well as user prompt, for this purpose llava-cli has a basic templating system:110 111**For Mistral and using llava-cli binary:**112Add this: `-p "<image>\nUSER:\nProvide a full description.\nASSISTANT:\n"`113The mistral template for llava-1.6 seems to be no system print and a USER/ASSISTANT role114 115**For the 34B this should work:**116Add this: `-e -p <|im_start|>system\nAnswer the questions.<|im_end|><|im_start|>user\n<image>\nProvide a full description.<|im_end|><|im_start|>assistant\n`117 118 119## How to know if you are running in llava-1.5 or llava-1.6 mode120 121When running llava-cli you will see a visual information right before the prompt is being processed:122 123**Llava-1.5:**124`encode_image_with_clip: image embedding created: 576 tokens`125 126**Llava-1.6 (anything above 576):**127`encode_image_with_clip: image embedding created: 2880 tokens`128 129 130Alternatively just pay notice to how many "tokens" have been used for your prompt, it will also show 1000+ tokens for llava-1.6131 132 133 134 135## TODO136 137- [x] Support non-CPU backend for the image encoding part.138- [ ] Support different sampling methods.139- [ ] Support more model variants.140