KBaba7/llama.cpp

1 2# Android3 4## Build on Android using Termux5 6[Termux](https://termux.dev/en/) is an Android terminal emulator and Linux environment app (no root required). As of writing, Termux is available experimentally in the Google Play Store; otherwise, it may be obtained directly from the project repo or on F-Droid.7 8With Termux, you can install and run `llama.cpp` as if the environment were Linux. Once in the Termux shell:9 10```11$ apt update && apt upgrade -y12$ apt install git cmake13```14 15Then, follow the [build instructions](https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md), specifically for CMake.16 17Once the binaries are built, download your model of choice (e.g., from Hugging Face). It's recommended to place it in the `~/` directory for best performance:18 19```20$ curl -L {model-url} -o ~/{model}.gguf21```22 23Then, if you are not already in the repo directory, `cd` into `llama.cpp` and:24 25```26$ ./build/bin/llama-cli -m ~/{model}.gguf -c {context-size} -p "{your-prompt}"27```28 29Here, we show `llama-cli`, but any of the executables under `examples` should work, in theory. Be sure to set `context-size` to a reasonable number (say, 4096) to start with; otherwise, memory could spike and kill your terminal.30 31To see what it might look like visually, here's an old demo of an interactive session running on a Pixel 5 phone:32 33https://user-images.githubusercontent.com/271616/225014776-1d567049-ad71-4ef2-b050-55b0b3b9274c.mp434 35## Cross-compile using Android NDK36It's possible to build `llama.cpp` for Android on your host system via CMake and the Android NDK. If you are interested in this path, ensure you already have an environment prepared to cross-compile programs for Android (i.e., install the Android SDK). Note that, unlike desktop environments, the Android environment ships with a limited set of native libraries, and so only those libraries are available to CMake when building with the Android NDK (see: https://developer.android.com/ndk/guides/stable_apis.)37 38Once you're ready and have cloned `llama.cpp`, invoke the following in the project directory:39 40```41$ cmake \42  -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \43  -DANDROID_ABI=arm64-v8a \44  -DANDROID_PLATFORM=android-28 \45  -DCMAKE_C_FLAGS="-march=armv8.7a" \46  -DCMAKE_CXX_FLAGS="-march=armv8.7a" \47  -DGGML_OPENMP=OFF \48  -DGGML_LLAMAFILE=OFF \49  -B build-android50```51 52Notes:53  - While later versions of Android NDK ship with OpenMP, it must still be installed by CMake as a dependency, which is not supported at this time54  - `llamafile` does not appear to support Android devices (see: https://github.com/Mozilla-Ocho/llamafile/issues/325)55 56The above command should configure `llama.cpp` with the most performant options for modern devices. Even if your device is not running `armv8.7a`, `llama.cpp` includes runtime checks for available CPU features it can use.57 58Feel free to adjust the Android ABI for your target. Once the project is configured:59 60```61$ cmake --build build-android --config Release -j{n}62$ cmake --install build-android --prefix {install-dir} --config Release63```64 65After installing, go ahead and download the model of your choice to your host system. Then:66 67```68$ adb shell "mkdir /data/local/tmp/llama.cpp"69$ adb push {install-dir} /data/local/tmp/llama.cpp/70$ adb push {model}.gguf /data/local/tmp/llama.cpp/71$ adb shell72```73 74In the `adb shell`:75 76```77$ cd /data/local/tmp/llama.cpp78$ LD_LIBRARY_PATH=lib ./bin/llama-simple -m {model}.gguf -c {context-size} -p "{your-prompt}"79```80 81That's it!82 83Be aware that Android will not find the library path `lib` on its own, so we must specify `LD_LIBRARY_PATH` in order to run the installed executables. Android does support `RPATH` in later API levels, so this could change in the future. Refer to the previous section for information about `context-size` (very important!) and running other `examples`.84