KBaba7/llama.cpp

1# GLMV-EDGE2 3Currently this implementation supports [glm-edge-v-2b](https://huggingface.co/THUDM/glm-edge-v-2b) and [glm-edge-v-5b](https://huggingface.co/THUDM/glm-edge-v-5b).4 5## Usage6Build with cmake or run `make llama-llava-cli` to build it.7 8After building, run: `./llama-llava-cli` to see the usage. For example:9 10```sh11./llama-llava-cli -m model_path/ggml-model-f16.gguf --mmproj model_path/mmproj-model-f16.gguf --image img_path/image.jpg -p "<|system|>\n system prompt <image><|user|>\n prompt <|assistant|>\n"12```13 14**note**: A lower temperature like 0.1 is recommended for better quality. add `--temp 0.1` to the command to do so.15**note**: For GPU offloading ensure to use the `-ngl` flag just like usual16 17## GGUF conversion18 191. Clone a GLMV-EDGE model ([2B](https://huggingface.co/THUDM/glm-edge-v-2b) or [5B](https://huggingface.co/THUDM/glm-edge-v-5b)). For example:20 21```sh22git clone https://huggingface.co/THUDM/glm-edge-v-5b or https://huggingface.co/THUDM/glm-edge-v-2b23```24 252. Use `glmedge-surgery.py` to split the GLMV-EDGE model to LLM and multimodel projector constituents:26 27```sh28python ./examples/llava/glmedge-surgery.py -m ../model_path29```30 314. Use `glmedge-convert-image-encoder-to-gguf.py` to convert the GLMV-EDGE image encoder to GGUF:32 33```sh34python ./examples/llava/glmedge-convert-image-encoder-to-gguf.py -m ../model_path --llava-projector ../model_path/glm.projector --output-dir ../model_path35```36 375. Use `examples/convert_hf_to_gguf.py` to convert the LLM part of GLMV-EDGE to GGUF:38 39```sh40python convert_hf_to_gguf.py ../model_path41```42 43Now both the LLM part and the image encoder are in the `model_path` directory.44