KBaba7/llama.cpp
0
1# Setting Up CUDA on Fedora2 3In this guide we setup [Nvidia CUDA](https://docs.nvidia.com/cuda/) in a toolbox container. This guide is applicable for:4 5- [Fedora Workstation](https://fedoraproject.org/workstation/)6- [Atomic Desktops for Fedora](https://fedoraproject.org/atomic-desktops/)7- [Fedora Spins](https://fedoraproject.org/spins)8- [Other Distributions](https://containertoolbx.org/distros/), including `Red Hat Enterprise Linux >= 8.5`, `Arch Linux`, and `Ubuntu`.9 10## Table of Contents11 12- [Prerequisites](#prerequisites)13- [Using the Fedora 41 CUDA Repository](#using-the-fedora-41-cuda-repository)14- [Creating a Fedora Toolbox Environment](#creating-a-fedora-toolbox-environment)15- [Installing Essential Development Tools](#installing-essential-development-tools)16- [Adding the CUDA Repository](#adding-the-cuda-repository)17- [Installing `nvidia-driver-libs`](#installing-nvidia-driver-libs)18- [Manually Resolving Package Conflicts](#manually-resolving-package-conflicts)19- [Finalizing the Installation of `nvidia-driver-libs`](#finalizing-the-installation-of-nvidia-driver-libs)20- [Installing the CUDA Meta-Package](#installing-the-cuda-meta-package)21- [Configuring the Environment](#configuring-the-environment)22- [Verifying the Installation](#verifying-the-installation)23- [Conclusion](#conclusion)24- [Troubleshooting](#troubleshooting)25- [Additional Notes](#additional-notes)26- [References](#references)27 28## Prerequisites29 30- **Toolbox Installed on the Host System** `Fedora Silverblue` and `Fedora Workstation` both have toolbox by default, other distributions may need to install the [toolbox package](https://containertoolbx.org/install/).31- **NVIDIA Drivers and Graphics Card installed on Host System (recommended)** To run CUDA program, such as `llama.cpp`, the host should be setup to access your NVIDIA hardware. Fedora Hosts can use the [RPM Fusion Repository](https://rpmfusion.org/Howto/NVIDIA).32- **Internet connectivity** to download packages.33 34### Using the Fedora 41 CUDA Repository35 36The latest release is 41.37 38- [Fedora 41 CUDA Repository](https://developer.download.nvidia.com/compute/cuda/repos/fedora41/x86_64/)39 40**Note:** We recommend using a toolbox environment to prevent system conflicts.41 42## Creating a Fedora Toolbox Environment43 44This guide focuses on Fedora hosts, but with small adjustments, it can work for other hosts. Using the Fedora Toolbox allows us to install the necessary packages without affecting the host system.45 46**Note:** Toolbox is available for other systems, and even without Toolbox, it is possible to use Podman or Docker.47 481. **Create a Fedora 41 Toolbox:**49 50 ```bash51 toolbox create --image registry.fedoraproject.org/fedora-toolbox:41 --container fedora-toolbox-41-cuda52 ```53 542. **Enter the Toolbox:**55 56 ```bash57 toolbox enter --container fedora-toolbox-41-cuda58 ```59 60 Inside the toolbox, you have root privileges and can install packages without affecting the host system.61 62## Installing Essential Development Tools63 641. **Synchronize the DNF Package Manager:**65 66 ```bash67 sudo dnf distro-sync68 ```69 702. **Install the Default Text Editor (Optional):**71 72 ```bash73 sudo dnf install vim-default-editor --allowerasing74 ```75 76 The `--allowerasing` flag will allow the removal of the conflicting `nano-default-editor` package.77 783. **Install Development Tools and Libraries:**79 80 ```bash81 sudo dnf install @c-development @development-tools cmake82 ```83 84 This installs essential packages for compiling software, including `gcc`, `make`, and other development headers.85 86## Adding the CUDA Repository87 88Add the NVIDIA CUDA repository to your DNF configuration:89 90```bash91sudo dnf config-manager addrepo --from-repofile=https://developer.download.nvidia.com/compute/cuda/repos/fedora41/x86_64/cuda-fedora41.repo92```93 94After adding the repository, synchronize the package manager again:95 96```bash97sudo dnf distro-sync98```99 100## Installing `nvidia-driver-libs` and `nvidia-driver-cuda-libs`101 102We need to detect if the host is supplying the [NVIDIA driver libraries into the toolbox](https://github.com/containers/toolbox/blob/main/src/pkg/nvidia/nvidia.go).103 104```bash105ls -la /usr/lib64/libcuda.so.1106```107 108**Explanation:**109 110- `nvidia-driver-libs` and `nvidia-driver-cuda-libs` contains necessary NVIDIA driver libraries required by CUDA,111 on hosts with NVIDIA drivers installed the Fedora Container will supply the host libraries.112 113### Install Nvidia Driver Libraries on Guest (if `libcuda.so.1` was NOT found).114 115```bash116sudo dnf install nvidia-driver-libs nvidia-driver-cuda-libs117```118 119### Manually Updating the RPM database for host-supplied NVIDIA drivers (if `libcuda.so.1` was found).120 121If the installation fails due to conflicts, we'll manually download and install the required packages, excluding conflicting files.122 123#### 1. Download `nvidia-driver-libs` and `nvidia-driver-cuda-libs` RPM's (with dependencies)124 125```bash126sudo dnf download --destdir=/tmp/nvidia-driver-libs --resolve --arch x86_64 nvidia-driver-libs nvidia-driver-cuda-libs127```128 129#### 2. Update the RPM database to assume the installation of these packages.130 131```bash132sudo rpm --install --verbose --hash --justdb /tmp/nvidia-driver-libs/*133```134 135**Note:**136 137- The `--justdb` option only updates the RPM database, without touching the filesystem.138 139#### Finalizing the Installation of `nvidia-driver-libs` and `nvidia-driver-cuda-libs`140 141After manually installing the dependencies, run:142 143```bash144sudo dnf install nvidia-driver-libs nvidia-driver-cuda-libs145```146 147You should receive a message indicating the package is already installed:148 149```150Updating and loading repositories:151Repositories loaded.152Package "nvidia-driver-libs-3:570.86.10-1.fc41.x86_64" is already installed.153Package "nvidia-driver-cuda-libs-3:570.86.10-1.fc41.x86_64" is already installed.154 155Nothing to do.156```157 158## Installing the CUDA Meta-Package159 160Now that the driver libraries are installed, proceed to install CUDA:161 162```bash163sudo dnf install cuda164```165 166This installs the CUDA toolkit and associated packages.167 168## Configuring the Environment169 170To use CUDA, add its binary directory to your system's `PATH`.171 1721. **Create a Profile Script:**173 174 ```bash175 sudo sh -c 'echo "export PATH=\$PATH:/usr/local/cuda/bin" >> /etc/profile.d/cuda.sh'176 ```177 178 **Explanation:**179 180 - We add to `/etc/profile.d/` as the `/etc/` folder is unique to this particular container, and is not shared with other containers or the host system.181 - The backslash `\` before `$PATH` ensures the variable is correctly written into the script.182 1832. **Make the Script Executable:**184 185 ```bash186 sudo chmod +x /etc/profile.d/cuda.sh187 ```188 1893. **Source the Script to Update Your Environment:**190 191 ```bash192 source /etc/profile.d/cuda.sh193 ```194 195 **Note:** This command updates your current shell session with the new `PATH`. The `/etc/profile.d/cuda.sh` script ensures that the CUDA binaries are available in your `PATH` for all future sessions.196 197## Verifying the Installation198 199To confirm that CUDA is correctly installed and configured, check the version of the NVIDIA CUDA Compiler (`nvcc`):200 201```bash202nvcc --version203```204 205You should see output similar to:206 207```208nvcc: NVIDIA (R) Cuda compiler driver209Copyright (c) 2005-2025 NVIDIA Corporation210Built on Wed_Jan_15_19:20:09_PST_2025211Cuda compilation tools, release 12.8, V12.8.61212Build cuda_12.8.r12.8/compiler.35404655_0213```214 215This output confirms that the CUDA compiler is accessible and indicates the installed version.216 217## Conclusion218 219You have successfully set up CUDA on Fedora within a toolbox environment using the Fedora 41 CUDA repository. By manually updating the RPM db and configuring the environment, you can develop CUDA applications without affecting your host system.220 221## Troubleshooting222 223- **Installation Failures:**224 225 - If you encounter errors during installation, carefully read the error messages. They often indicate conflicting files or missing dependencies.226 - You may use the `--excludepath` option with `rpm` to exclude conflicting files during manual RPM installations.227 228- **Rebooting the Container:**229 230 - Sometimes there may be a bug in the NVIDIA driver host passthrough (such as missing a shared library). Rebooting the container may solve this issue:231 232 ```bash233 # on the host system234 podman container restart --all235 ```236 237- **Environment Variables Not Set:**238 - If `nvcc` is not found after installation, ensure that `/usr/local/cuda/bin` is in your `PATH`.239 - Run `echo $PATH` to check if the path is included.240 - Re-source the profile script or open a new terminal session.241 242## Additional Notes243 244- **Updating CUDA in the Future:**245 246 - Keep an eye on the official NVIDIA repositories for updates to your Fedora version.247 - When an updated repository becomes available, adjust your `dnf` configuration accordingly.248 249- **Building `llama.cpp`:**250 251 - With CUDA installed, you can follow these [build instructions for `llama.cpp`](https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md) to compile it with CUDA support.252 - Ensure that any CUDA-specific build flags or paths are correctly set in your build configuration.253 254- **Using the Toolbox Environment:**255 - The toolbox environment is isolated from your host system, which helps prevent conflicts.256 - Remember that system files and configurations inside the toolbox are separate from the host. By default the home directory of the user is shared between the host and the toolbox.257 258---259 260**Disclaimer:** Manually installing and modifying system packages can lead to instability of the container. The above steps are provided as a guideline and may need adjustments based on your specific system configuration. Always back up important data before making significant system changes, especially as your home folder is writable and shared with he toolbox.261 262**Acknowledgments:** Special thanks to the Fedora community and NVIDIA documentation for providing resources that assisted in creating this guide.263 264## References265 266- [Fedora Toolbox Documentation](https://docs.fedoraproject.org/en-US/fedora-silverblue/toolbox/)267- [NVIDIA CUDA Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html)268- [Podman Documentation](https://podman.io/get-started)269 270---271