Brunobkr/llama.cpp_AlgMor24_github

1# Development and Testing2 3## Development4 5### Code Generation6 7The backend uses code generation from YAML configuration:8 9```bash10# Regenerate protocol code11cd ggml-virtgpu/12python regenerate_remoting.py13```14 15### Adding New Operations16 171. Add function definition to `ggmlremoting_functions.yaml`182. Regenerate code with `regenerate_remoting.py`193. Implement guest-side forwarding in `virtgpu-forward-*.cpp`204. Implement host-side handling in `backend-dispatched-*.cpp`21 22## Testing23 24This document provides instructions for building and testing the GGML-VirtGPU backend on macOS with containers.25 26### Prerequisites27 28The testing setup requires:29 30- macOS host system31- Container runtime with `libkrun` provider (podman machine)32- Access to development patchset for VirglRenderer33 34### Required Patchsets35 36The backend requires patches that are currently under review:37 38- **Virglrenderer APIR upstream PR**: https://gitlab.freedesktop.org/virgl/virglrenderer/-/merge_requests/1590 (for reference)39- **MacOS Virglrenderer (for krunkit)**: https://gitlab.freedesktop.org/kpouget/virglrenderer/-/tree/main-macos40- **Linux Virglrenderer (for krun)**: https://gitlab.freedesktop.org/kpouget/virglrenderer/-/tree/main-linux41 42### Build Instructions43 44#### 1. Build ggml-virtgpu-backend (Host-side, macOS)45 46```bash47# Build the backend that runs natively on macOS48mkdir llama.cpp49cd llama.cpp50git clone https://github.com/ggml-org/llama.cpp.git src51cd src52 53LLAMA_MAC_BUILD=$PWD/build/ggml-virtgpu-backend54 55cmake -S . -B $LLAMA_MAC_BUILD \56      -DGGML_NATIVE=OFF \57      -DLLAMA_CURL=ON \58      -DGGML_VIRTGPU=ON \59      -DGGML_VIRTGPU_BACKEND=ONLY \60      -DGGML_METAL=ON61 62TARGETS="ggml-metal"63cmake --build $LLAMA_MAC_BUILD --parallel 8 --target $TARGETS64 65# Build additional tools for native benchmarking66EXTRA_TARGETS="llama-run llama-bench"67cmake --build $LLAMA_MAC_BUILD --parallel 8 --target $EXTRA_TARGETS68```69 70#### 2. Build virglrenderer (Host-side, macOS)71 72```bash73# Build virglrenderer with APIR support74mkdir virglrenderer75cd virglrenderer76git clone https://gitlab.freedesktop.org/kpouget/virglrenderer -b main-macos src77cd src78 79VIRGL_BUILD_DIR=$PWD/build80 81# -Dvenus=true and VIRGL_ROUTE_VENUS_TO_APIR=1 route the APIR requests via the Venus backend, for easier testing without a patched hypervisor82 83meson setup $VIRGL_BUILD_DIR \84      -Dvenus=true \85      -Dapir=true86 87ninja -C $VIRGL_BUILD_DIR88```89 90#### 3. Build ggml-virtgpu (Guest-side, Linux)91 92Option A: Build from a script:93 94```bash95# Inside a Linux container96mkdir llama.cpp97git clone https://github.com/ggml-org/llama.cpp.git src98cd src99 100LLAMA_LINUX_BUILD=$PWD/build-virtgpu101 102cmake -S . -B $LLAMA_LINUX_BUILD \103      -DGGML_VIRTGPU=ON104 105ninja -C $LLAMA_LINUX_BUILD106```107 108Option B: Build container image with frontend:109 110```bash111cat << EOF > remoting.containerfile112FROM quay.io/fedora/fedora:43113USER 0114 115WORKDIR /app/remoting116 117ARG LLAMA_CPP_REPO="https://github.com/ggml-org/llama.cpp.git"118ARG LLAMA_CPP_VERSION="master"119ARG LLAMA_CPP_CMAKE_FLAGS="-DGGML_VIRTGPU=ON"120ARG LLAMA_CPP_CMAKE_BUILD_FLAGS="--parallel 4"121 122RUN dnf install -y git cmake gcc gcc-c++ libcurl-devel libdrm-devel123 124RUN git clone "\${LLAMA_CPP_REPO}" src \\125 && git -C src fetch origin \${LLAMA_CPP_VERSION} \\126 && git -C src reset --hard FETCH_HEAD127 128RUN mkdir -p build \\129 && cd src \\130 && set -o pipefail \\131 && cmake -S . -B ../build \${LLAMA_CPP_CMAKE_FLAGS} \\132 && cmake --build ../build/ \${LLAMA_CPP_CMAKE_BUILD_FLAGS}133 134ENTRYPOINT ["/app/remoting/src/build/bin/llama-server"]135EOF136 137mkdir -p empty_dir138podman build -f remoting.containerfile ./empty_dir -t localhost/llama-cpp.virtgpu139```140 141### Environment Setup142 143#### Set krunkit Environment Variables144 145```bash146# Define the base directories (adapt these paths to your system)147VIRGL_BUILD_DIR=$HOME/remoting/virglrenderer/build148LLAMA_MAC_BUILD=$HOME/remoting/llama.cpp/build-backend149 150# For krunkit to load the custom virglrenderer library151export DYLD_LIBRARY_PATH=$VIRGL_BUILD_DIR/src152 153# For Virglrenderer to load the ggml-remotingbackend library154export VIRGL_APIR_BACKEND_LIBRARY="$LLAMA_MAC_BUILD/bin/libggml-virtgpu-backend.dylib"155 156# For llama.cpp remotingbackend to load the ggml-metal backend157export APIR_LLAMA_CPP_GGML_LIBRARY_PATH="$LLAMA_MAC_BUILD/bin/libggml-metal.dylib"158export APIR_LLAMA_CPP_GGML_LIBRARY_REG=ggml_backend_metal_reg159```160 161#### Launch Container Environment162 163```bash164# Set container provider to libkrun165export CONTAINERS_MACHINE_PROVIDER=libkrun166podman machine start167```168 169#### Verify Environment170 171Confirm that krunkit is using the correct virglrenderer library:172 173```bash174lsof -c krunkit | grep virglrenderer175# Expected output:176# krunkit 50574 user  txt  REG  1,14  2273912  10849442 ($VIRGL_BUILD_DIR/src)/libvirglrenderer.1.dylib177```178 179### Running Tests180 181#### Launch Test Container182 183```bash184# Optional model caching185mkdir -p models186PODMAN_CACHE_ARGS="-v models:/models --user root:root --cgroupns host --security-opt label=disable -w /models"187 188podman run $PODMAN_CACHE_ARGS -it --rm --device /dev/dri localhost/llama-cpp.virtgpu189```190 191#### Test llama.cpp in Container192 193```bash194 195# Run performance benchmark196/app/remoting/build/bin/llama-bench -m ./llama3.2197```198 199Expected output (performance may vary):200```201| model                          |       size |     params | backend    | ngl |          test |                  t/s |202| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |203| llama 3B Q4_K - Medium         |   1.87 GiB |     3.21 B | ggml-virtgpu |  99 |         pp512 |        991.30 ± 0.66 |204| llama 3B Q4_K - Medium         |   1.87 GiB |     3.21 B | ggml-virtgpu |  99 |         tg128 |         85.71 ± 0.11 |205```206 207### Troubleshooting208 209#### SSH Environment Variable Issues210 211⚠️ **Warning**: Setting `DYLD_LIBRARY_PATH` from SSH doesn't work on macOS. Here is a workaround:212 213**Workaround 1: Replace system library**214```bash215VIRGL_BUILD_DIR=$HOME/remoting/virglrenderer/build  # ⚠️ adapt to your system216BREW_VIRGL_DIR=/opt/homebrew/Cellar/virglrenderer/0.10.4d/lib217VIRGL_LIB=libvirglrenderer.1.dylib218 219cd $BREW_VIRGL_DIR220mv $VIRGL_LIB ${VIRGL_LIB}.orig221ln -s $VIRGL_BUILD_DIR/src/$VIRGL_LIB222```223