Brunobkr/llama.cpp_AlgMor24_github

1# LLaVA2 3Currently this implementation supports [llava-v1.5](https://huggingface.co/liuhaotian/llava-v1.5-7b) variants,4as well as llava-1.6 [llava-v1.6](https://huggingface.co/collections/liuhaotian/llava-16-65b9e40155f60fd046a5ccf2) variants.5 6The pre-converted [7b](https://huggingface.co/mys/ggml_llava-v1.5-7b)7and [13b](https://huggingface.co/mys/ggml_llava-v1.5-13b)8models are available.9For llava-1.6 a variety of prepared gguf models are available as well [7b-34b](https://huggingface.co/cmp-nct/llava-1.6-gguf)10 11After API is confirmed, more models will be supported / uploaded.12 13## Usage14Build the `llama-mtmd-cli` binary.15 16After building, run: `./llama-mtmd-cli` to see the usage. For example:17 18```sh19./llama-mtmd-cli -m ../llava-v1.5-7b/ggml-model-f16.gguf \20    --mmproj ../llava-v1.5-7b/mmproj-model-f16.gguf \21    --chat-template vicuna22```23 24**note**: A lower temperature like 0.1 is recommended for better quality. add `--temp 0.1` to the command to do so.25**note**: For GPU offloading ensure to use the `-ngl` flag just like usual26 27## LLaVA 1.528 291. Clone a LLaVA and a CLIP model ([available options](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md)). For example:30 31```sh32git clone https://huggingface.co/liuhaotian/llava-v1.5-7b33 34git clone https://huggingface.co/openai/clip-vit-large-patch14-33635```36 372. Install the required Python packages:38 39```sh40pip install -r tools/mtmd/requirements.txt41```42 433. Use `llava_surgery.py` to split the LLaVA model to LLaMA and multimodel projector constituents:44 45```sh46python ./tools/mtmd/llava_surgery.py -m ../llava-v1.5-7b47```48 494. Use `convert_image_encoder_to_gguf.py` to convert the LLaVA image encoder to GGUF:50 51```sh52python ./tools/mtmd/convert_image_encoder_to_gguf.py -m ../clip-vit-large-patch14-336 --llava-projector ../llava-v1.5-7b/llava.projector --output-dir ../llava-v1.5-7b53```54 555. Use `examples/convert_legacy_llama.py` to convert the LLaMA part of LLaVA to GGUF:56 57```sh58python ./examples/convert_legacy_llama.py ../llava-v1.5-7b --skip-unknown59```60 61Now both the LLaMA part and the image encoder are in the `llava-v1.5-7b` directory.62 63## LLaVA 1.6 gguf conversion641) First clone a LLaVA 1.6 model:65```console66git clone https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b67```68 692) Install the required Python packages:70 71```sh72pip install -r tools/mtmd/requirements.txt73```74 753) Use `llava_surgery_v2.py` which also supports llava-1.5 variants pytorch as well as safetensor models:76```console77python tools/mtmd/llava_surgery_v2.py -C -m ../llava-v1.6-vicuna-7b/78```79- you will find a llava.projector and a llava.clip file in your model directory80 814) Copy the llava.clip file into a subdirectory (like vit), rename it to pytorch_model.bin and add a fitting vit configuration to the directory:82```console83mkdir vit84cp ../llava-v1.6-vicuna-7b/llava.clip vit/pytorch_model.bin85cp ../llava-v1.6-vicuna-7b/llava.projector vit/86curl -s -q https://huggingface.co/cmp-nct/llava-1.6-gguf/raw/main/config_vit.json -o vit/config.json87```88 895) Create the visual gguf model:90```console91python ./tools/mtmd/convert_image_encoder_to_gguf.py -m vit --llava-projector vit/llava.projector --output-dir vit --clip-model-is-vision92```93- This is similar to llava-1.5, the difference is that we tell the encoder that we are working with the pure vision model part of CLIP94 956) Then convert the model to gguf format:96```console97python ./examples/convert_legacy_llama.py ../llava-v1.6-vicuna-7b/ --skip-unknown98```99 1007) And finally we can run the llava cli using the 1.6 model version:101```console102./llama-mtmd-cli -m ../llava-v1.6-vicuna-7b/ggml-model-f16.gguf --mmproj vit/mmproj-model-f16.gguf103```104 105**note** llava-1.6 needs more context than llava-1.5, at least 3000 is needed (just run it at -c 4096)106 107**note** llava-1.6 greatly benefits from batched prompt processing (defaults work)108 109**note** if the language model in step `6)` is incompatible with the legacy conversion script, the easiest way handle the LLM model conversion is to load the model in transformers, and export only the LLM from the llava next model.110 111```python112import os113import transformers114 115model_path = ...116llm_export_path = ...117 118tokenizer = transformers.AutoTokenizer.from_pretrained(model_path)119model = transformers.AutoModelForImageTextToText.from_pretrained(model_path)120 121tokenizer.save_pretrained(llm_export_path)122model.language_model.save_pretrained(llm_export_path)123```124 125Then, you can convert the LLM using the `convert_hf_to_gguf.py` script, which handles more LLM architectures.126 127## Chat template128 129For llava-1.5 and llava-1.6, you need to use `vicuna` chat template. Simply add `--chat-template vicuna` to activate this template.130 131 132## How to know if you are running in llava-1.5 or llava-1.6 mode133 134When running llava-cli you will see a visual information right before the prompt is being processed:135 136**Llava-1.5:**137`encode_image_with_clip: image embedding created: 576 tokens`138 139**Llava-1.6 (anything above 576):**140`encode_image_with_clip: image embedding created: 2880 tokens`141 142 143Alternatively just pay notice to how many "tokens" have been used for your prompt, it will also show 1000+ tokens for llava-1.6144