KBaba7/llama.cpp

1## MiniCPM-V 2.62 3### Prepare models and code4 5Download [MiniCPM-V-2_6](https://huggingface.co/openbmb/MiniCPM-V-2_6) PyTorch model from huggingface to "MiniCPM-V-2_6" folder.6 7Clone llama.cpp:8```bash9git clone git@github.com:OpenBMB/llama.cpp.git10cd llama.cpp11git checkout minicpmv-main12```13 14### Usage of MiniCPM-V 2.615 16Convert PyTorch model to gguf files (You can also download the converted [gguf](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) by us)17 18```bash19python ./examples/llava/minicpmv-surgery.py -m ../MiniCPM-V-2_620python ./examples/llava/minicpmv-convert-image-encoder-to-gguf.py -m ../MiniCPM-V-2_6 --minicpmv-projector ../MiniCPM-V-2_6/minicpmv.projector --output-dir ../MiniCPM-V-2_6/ --image-mean 0.5 0.5 0.5 --image-std 0.5 0.5 0.5 --minicpmv_version 321python ./convert_hf_to_gguf.py ../MiniCPM-V-2_6/model22 23# quantize int4 version24./llama-quantize ../MiniCPM-V-2_6/model/ggml-model-f16.gguf ../MiniCPM-V-2_6/model/ggml-model-Q4_K_M.gguf Q4_K_M25```26 27Build for Linux or Mac28 29```bash30make31make llama-minicpmv-cli32```33 34Inference on Linux or Mac35```36# run f16 version37./llama-minicpmv-cli -m ../MiniCPM-V-2_6/model/ggml-model-f16.gguf --mmproj ../MiniCPM-V-2_6/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image xx.jpg -p "What is in the image?"38 39# run quantized int4 version40./llama-minicpmv-cli -m ../MiniCPM-V-2_6/model/ggml-model-Q4_K_M.gguf --mmproj ../MiniCPM-V-2_6/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image xx.jpg  -p "What is in the image?"41 42# or run in interactive mode43./llama-minicpmv-cli -m ../MiniCPM-V-2_6/model/ggml-model-Q4_K_M.gguf --mmproj ../MiniCPM-V-2_6/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image xx.jpg -i44```45 46### Video47Install FFmpeg48```49brew install ffmpeg50brew install pkg-config51```52 53### Android54 55#### Build on Android device using Termux56We found that build on Android device would bring better runtime performance, so we recommend to build on device.57 58[Termux](https://github.com/termux/termux-app#installation) is a terminal app on Android device (no root required).59 60Install tools in Termux:61```62apt update && apt upgrade -y63apt install git make cmake64```65 66It's recommended to move your model inside the `~/` directory for best performance:67```68cd storage/downloads69mv model.gguf ~/70```71 72#### Building the Project using Android NDK73Obtain the [Android NDK](https://developer.android.com/ndk) and then build with CMake.74 75Execute the following commands on your computer to avoid downloading the NDK to your mobile. Alternatively, you can also do this in Termux:76 77```bash78mkdir build-android79cd build-android80export NDK=/your_ndk_path81cmake -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-23 -DCMAKE_C_FLAGS=-march=armv8.4a+dotprod ..82make83```84 85Install [termux](https://github.com/termux/termux-app#installation) on your device and run `termux-setup-storage` to get access to your SD card (if Android 11+ then run the command twice).86 87Finally, copy these built `llama` binaries and the model file to your device storage. Because the file permissions in the Android sdcard cannot be changed, you can copy the executable files to the `/data/data/com.termux/files/home/bin` path, and then execute the following commands in Termux to add executable permission:88 89(Assumed that you have pushed the built executable files to the /sdcard/llama.cpp/bin path using `adb push`)90```91$cp -r /sdcard/llama.cpp/bin /data/data/com.termux/files/home/92$cd /data/data/com.termux/files/home/bin93$chmod +x ./*94```95 96Download models and push them to `/sdcard/llama.cpp/`, then move it to `/data/data/com.termux/files/home/model/`97 98```99$mv /sdcard/llama.cpp/ggml-model-Q4_K_M.gguf /data/data/com.termux/files/home/model/100$mv /sdcard/llama.cpp/mmproj-model-f16.gguf /data/data/com.termux/files/home/model/101```102 103Now, you can start chatting:104```105$cd /data/data/com.termux/files/home/bin106$./llama-minicpmv-cli -m ../model/ggml-model-Q4_K_M.gguf --mmproj ../model/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image xx.jpg  -p "What is in the image?"107```108