KBaba7/llama.cpp

1# Add a new model architecture to `llama.cpp`2 3Adding a model requires few steps:4 51. Convert the model to GGUF62. Define the model architecture in `llama.cpp`73. Build the GGML graph implementation8 9After following these steps, you can open PR.10 11Also, it is important to check that the examples and main ggml backends (CUDA, METAL, CPU) are working with the new architecture, especially:12- [main](/examples/main/)13- [imatrix](/examples/imatrix/)14- [quantize](/examples/quantize/)15- [server](/examples/server/)16 17### 1. Convert the model to GGUF18 19This step is done in python with a `convert` script using the [gguf](https://pypi.org/project/gguf/) library.20Depending on the model architecture, you can use either [convert_hf_to_gguf.py](/convert_hf_to_gguf.py) or [examples/convert_legacy_llama.py](/examples/convert_legacy_llama.py) (for `llama/llama2` models in `.pth` format).21 22The convert script reads the model configuration, tokenizer, tensor names+data and converts them to GGUF metadata and tensors.23 24The required steps to implement for an HF model are:25 261. Define the model `Model.register` annotation in a new `Model` subclass, example:27 28```python29@Model.register("MyModelForCausalLM")30class MyModel(Model):31    model_arch = gguf.MODEL_ARCH.MYMODEL32```33 342. Define the layout of the GGUF tensors in [constants.py](/gguf-py/gguf/constants.py)35 36Add an enum entry in `MODEL_ARCH`, the model human friendly name in `MODEL_ARCH_NAMES` and the GGUF tensor names in `MODEL_TENSORS`.37 38Example for `falcon` model:39```python40    MODEL_ARCH.FALCON: [41        MODEL_TENSOR.TOKEN_EMBD,42        MODEL_TENSOR.OUTPUT_NORM,43        MODEL_TENSOR.OUTPUT,44        MODEL_TENSOR.ATTN_NORM,45        MODEL_TENSOR.ATTN_NORM_2,46        MODEL_TENSOR.ATTN_QKV,47        MODEL_TENSOR.ATTN_OUT,48        MODEL_TENSOR.FFN_DOWN,49        MODEL_TENSOR.FFN_UP,50    ]51```52 533. Map the original tensor names to the standardize equivalent in GGUF54 55As a general rule, before adding a new tensor name to GGUF, be sure the equivalent naming does not already exist.56 57Once you have found the GGUF tensor name equivalent, add it to the [tensor_mapping.py](/gguf-py/gguf/tensor_mapping.py) file.58 59If the tensor name is part of a repetitive layer/block, the key word `bid` substitutes it.60 61Example for the normalization tensor in attention layers:62 63```python64block_mappings_cfg: dict[MODEL_TENSOR, tuple[str, ...]] = {65        # Attention norm66        MODEL_TENSOR.ATTN_NORM: (67            "gpt_neox.layers.{bid}.input_layernorm",                # gptneox68            "transformer.h.{bid}.ln_1",                             # gpt2 gpt-j refact qwen69            "transformer.blocks.{bid}.norm_1",                      # mpt70            ...71        )72}73```74 75`transformer.blocks.{bid}.norm_1` will be mapped to `blk.{bid}.attn_norm` in GGUF.76 77Depending on the model configuration, tokenizer, code and tensors layout, you will have to override:78- `Model#set_gguf_parameters`79- `Model#set_vocab`80- `Model#write_tensors`81 82NOTE: Tensor names must end with `.weight` or `.bias` suffixes, that is the convention and several tools like `quantize` expect this to proceed the weights.83 84### 2. Define the model architecture in `llama.cpp`85 86The model params and tensors layout must be defined in `llama.cpp`:871. Define a new `llm_arch`882. Define the tensors layout in `LLM_TENSOR_NAMES`893. Add any non-standard metadata in `llm_load_hparams`904. Create the tensors for inference in `llm_load_tensors`915. If the model has a RoPE operation, add the rope type in `llama_rope_type`92 93NOTE: The dimensions in `ggml` are typically in the reverse order of the `pytorch` dimensions.94 95### 3. Build the GGML graph implementation96 97This is the funniest part, you have to provide the inference graph implementation of the new model architecture in `llama_build_graph`.98 99Have a look at existing implementations like `build_llama`, `build_dbrx` or `build_bert`.100 101Some `ggml` backends do not support all operations. Backend implementations can be added in a separate PR.102 103Note: to debug the inference graph: you can use [llama-eval-callback](/examples/eval-callback/).104 105## GGUF specification106 107https://github.com/ggerganov/ggml/blob/master/docs/gguf.md108 109## Resources110 111- YaRN RoPE scaling https://github.com/ggerganov/llama.cpp/pull/2268112- support Baichuan serial models https://github.com/ggerganov/llama.cpp/pull/3009113- support attention bias https://github.com/ggerganov/llama.cpp/pull/4283114- Mixtral support https://github.com/ggerganov/llama.cpp/pull/4406115- BERT embeddings https://github.com/ggerganov/llama.cpp/pull/5423116- Grok-1 support https://github.com/ggerganov/llama.cpp/pull/6204117- Command R Plus support https://github.com/ggerganov/llama.cpp/pull/6491118- support arch DBRX https://github.com/ggerganov/llama.cpp/pull/6515119- How to convert HuggingFace model to GGUF format https://github.com/ggerganov/llama.cpp/discussions/2948120