KBaba7/llama.cpp

1 2# llama-gguf-hash3 4CLI to hash GGUF files to detect difference on a per model and per tensor level.5 6**Command line options:**7 8- `--help`: display help message9- `--xxh64`: use xhash 64bit hash mode (default)10- `--sha1`: use sha111- `--uuid`: use uuid12- `--sha256`: use sha25613- `--all`: use all hash14- `--no-layer`: exclude per layer hash15- `--uuid`: generate UUIDv5 ID16- `-c`, `--check <manifest>`:  verify against a manifest17 18## About19 20While most POSIX systems already have hash checking programs like sha256sum, it21is designed to check entire files. This is not ideal for our purpose if we want22to check for consistency of the tensor data even if the metadata content of the23gguf KV store has been updated.24 25This program is designed to hash a gguf tensor payload on a 'per tensor layer'26in addition to a 'entire tensor model' hash. The intent is that the entire27tensor layer can be checked first but if there is any detected inconsistencies,28then the per tensor hash can be used to narrow down the specific tensor layer29that has inconsistencies.30 31For Maintainers:32- Detection of tensor inconsistency during development and automated tests33    - This is served by xxh64 which is fast34    - This is also served by having per tensor layer to assist in narrowing down35      the location of the faulty tensor layer36    - This is also served by sha1 which is much slower but more widely supported37 38For Model Creators:39- Optional consistent UUID generation based on model tensor content40    - This is served by UUIDv5 which is useful for databases keys41        - llama.cpp UUIDv5 Namespace: `ef001206-dadc-5f6d-a15f-3359e577d4e5`42            - Made via UUIDv5 URL namespace of `en.wikipedia.org/wiki/Llama.cpp`43 44For Model Users:45- Assurance of tensor layer integrity even if metadata was updated46    - This is served by sha256 which is still considered very secure as of 202447 48### Design Note49 50- The default behavior of this program if no arguments is provided is to hash51  using xxhash's xxh32 mode because it is very fast and is primarily targeted52  towards maintainers who may want to use this in automated tests.53- xxhash support xxh32 and xxh128 for 32bit hash and 128bit hash respectively54  however we picked 64bit xxhash as most computers are 64bit as of 2024 and thus55  would have a better affinity to calculating hash that is 64bit in size.56 57## Compile Example58 59```bash60cmake -B build -DCMAKE_BUILD_TYPE=Debug -DLLAMA_FATAL_WARNINGS=ON61make -C build clean62make -C build llama-gguf-hash VERBOSE=163./build/bin/llama-gguf-hash test.gguf64./build/bin/llama-gguf-hash --xxh64 test.gguf65./build/bin/llama-gguf-hash --sha1 test.gguf66./build/bin/llama-gguf-hash --uuid test.gguf67./build/bin/llama-gguf-hash --sha256 test.gguf68```69 70## Generation and Verification Example71 72To generate we may use this command73 74```bash75./llama-gguf-hash --all test.gguf > test.gguf.manifest76```77 78Which would generate a manifest that looks like below, which contains multiple hash type and per tensor layer hashes as well79(This excludes UUID as that is an ID not a hash)80 81```bash82xxh64     f66e9cd66a4396a0  test.gguf:tensor_083sha1      59f79ecefd8125a996fdf419239051a7e99e5f20  test.gguf:tensor_084sha256    c0510d38fa060c46265e0160a85c7243096b01dd31c2f355bdbb5516b20de1bd  test.gguf:tensor_085xxh64     7d3a1f9ac04d0537  test.gguf:tensor_186sha1      4765f592eacf096df4628ba59476af94d767080a  test.gguf:tensor_187sha256    8514cbcc73692a2c56bd7a33a022edd5ff819614bd23b19915d7224387f397a7  test.gguf:tensor_188xxh64     a0af5d700049693b  test.gguf:tensor_289sha1      25cbfbad4513cc348e2c95ebdee69d6ff2fd8753  test.gguf:tensor_290sha256    947e6b36e20f2cc95e1d2ce1c1669d813d574657ac6b5ac5196158d454d35180  test.gguf:tensor_291xxh64     e83fddf559d7b6a6  test.gguf:tensor_392sha1      a9cba73e2d90f2ee3dae2548caa42bef3fe6a96c  test.gguf:tensor_393sha256    423b044e016d8ac73c39f23f60bf01bedef5ecb03c0230accd824c91fe86f1a1  test.gguf:tensor_394xxh64     1257733306b7992d  test.gguf:tensor_495sha1      d7bc61db93bb685ce9d598da89717c66729b7543  test.gguf:tensor_496sha256    79737cb3912d4201384cf7f16a1a37ff7823f23ea796cb205b6ca361ab9e3ebf  test.gguf:tensor_497xxh64     d238d16ba4711e58  test.gguf:tensor_598sha1      0706566c198fe1072f37e0a5135b4b5f23654c52  test.gguf:tensor_599sha256    60949be8298eced0ecdde64487643d018407bd261691e061d9e9c3dbc9fd358b  test.gguf:tensor_5100xxh64     3fbc3b65ab8c7f39  test.gguf:tensor_6101sha1      73922a0727226a409049f6fc3172a52219ca6f00  test.gguf:tensor_6102sha256    574f4c46ff384a3b9a225eb955d2a871847a2e8b3fa59387a8252832e92ef7b0  test.gguf:tensor_6103xxh64     c22021c29854f093  test.gguf:tensor_7104sha1      efc39cece6a951188fc41e354c73bbfe6813d447  test.gguf:tensor_7105sha256    4c0410cd3c500f078ae5b21e8dc9eb79e29112713b2ab58a882f82a3868d4d75  test.gguf:tensor_7106xxh64     936df61f5d64261f  test.gguf:tensor_8107sha1      c2490296d789a4f34398a337fed8377d943d9f06  test.gguf:tensor_8108sha256    c4401313feeba0261275c3b25bd2d8fe40ce04e0f440c2980ed0e9674c30ff01  test.gguf:tensor_8109xxh64     93fd20c64421c081  test.gguf:tensor_9110sha1      7047ce1e78437a6884337a3751c7ee0421918a65  test.gguf:tensor_9111sha256    23d57cf0d7a6e90b0b3616b41300e0cd354781e812add854a5f95aa55f2bc514  test.gguf:tensor_9112xxh64     5a54d3aad816f302  test.gguf113sha1      d15be52c4ff213e823cb6dd13af7ee2f978e7042  test.gguf114sha256    7dd641b32f59b60dbd4b5420c4b0f6321ccf48f58f6ae201a3dbc4a58a27c6e4  test.gguf115```116 117We can then use the normal check command which will by default check for the highest security strength hash and verify against that:118 119```bash120$ ./llama-gguf-hash --check test.gguf.manifest test.gguf121manifest  test.gguf.manifest  sha256  sha1  xxh64122sha256    c0510d38fa060c46265e0160a85c7243096b01dd31c2f355bdbb5516b20de1bd  test.gguf:tensor_0  -  Ok123sha256    8514cbcc73692a2c56bd7a33a022edd5ff819614bd23b19915d7224387f397a7  test.gguf:tensor_1  -  Ok124sha256    947e6b36e20f2cc95e1d2ce1c1669d813d574657ac6b5ac5196158d454d35180  test.gguf:tensor_2  -  Ok125sha256    423b044e016d8ac73c39f23f60bf01bedef5ecb03c0230accd824c91fe86f1a1  test.gguf:tensor_3  -  Ok126sha256    79737cb3912d4201384cf7f16a1a37ff7823f23ea796cb205b6ca361ab9e3ebf  test.gguf:tensor_4  -  Ok127sha256    60949be8298eced0ecdde64487643d018407bd261691e061d9e9c3dbc9fd358b  test.gguf:tensor_5  -  Ok128sha256    574f4c46ff384a3b9a225eb955d2a871847a2e8b3fa59387a8252832e92ef7b0  test.gguf:tensor_6  -  Ok129sha256    4c0410cd3c500f078ae5b21e8dc9eb79e29112713b2ab58a882f82a3868d4d75  test.gguf:tensor_7  -  Ok130sha256    c4401313feeba0261275c3b25bd2d8fe40ce04e0f440c2980ed0e9674c30ff01  test.gguf:tensor_8  -  Ok131sha256    23d57cf0d7a6e90b0b3616b41300e0cd354781e812add854a5f95aa55f2bc514  test.gguf:tensor_9  -  Ok132sha256    7dd641b32f59b60dbd4b5420c4b0f6321ccf48f58f6ae201a3dbc4a58a27c6e4  test.gguf  -  Ok133 134Verification results for test.gguf.manifest - Success135```136 137Or we may explicitly ask for a faster hash like:138 139```bash140$ ./llama-gguf-hash --check test.gguf.manifest --xxh64 test.gguf141manifest  test.gguf.manifest  sha256  sha1  xxh64142xxh64     f66e9cd66a4396a0  test.gguf:tensor_0  -  Ok143xxh64     7d3a1f9ac04d0537  test.gguf:tensor_1  -  Ok144xxh64     a0af5d700049693b  test.gguf:tensor_2  -  Ok145xxh64     e83fddf559d7b6a6  test.gguf:tensor_3  -  Ok146xxh64     1257733306b7992d  test.gguf:tensor_4  -  Ok147xxh64     d238d16ba4711e58  test.gguf:tensor_5  -  Ok148xxh64     3fbc3b65ab8c7f39  test.gguf:tensor_6  -  Ok149xxh64     c22021c29854f093  test.gguf:tensor_7  -  Ok150xxh64     936df61f5d64261f  test.gguf:tensor_8  -  Ok151xxh64     93fd20c64421c081  test.gguf:tensor_9  -  Ok152xxh64     5a54d3aad816f302  test.gguf  -  Ok153 154Verification results for test.gguf.manifest - Success155```156 157Or maybe we want to just check that all the hash is valid:158 159```bash160$./llama-gguf-hash --check test.gguf.manifest --all test.gguf.manifest161manifest  test.gguf.manifest  sha256  sha1  xxh64162xxh64     f66e9cd66a4396a0  test.gguf:tensor_0  -  Ok163sha1      59f79ecefd8125a996fdf419239051a7e99e5f20  test.gguf:tensor_0  -  Ok164sha256    c0510d38fa060c46265e0160a85c7243096b01dd31c2f355bdbb5516b20de1bd  test.gguf:tensor_0  -  Ok165xxh64     7d3a1f9ac04d0537  test.gguf:tensor_1  -  Ok166sha1      4765f592eacf096df4628ba59476af94d767080a  test.gguf:tensor_1  -  Ok167sha256    8514cbcc73692a2c56bd7a33a022edd5ff819614bd23b19915d7224387f397a7  test.gguf:tensor_1  -  Ok168xxh64     a0af5d700049693b  test.gguf:tensor_2  -  Ok169sha1      25cbfbad4513cc348e2c95ebdee69d6ff2fd8753  test.gguf:tensor_2  -  Ok170sha256    947e6b36e20f2cc95e1d2ce1c1669d813d574657ac6b5ac5196158d454d35180  test.gguf:tensor_2  -  Ok171xxh64     e83fddf559d7b6a6  test.gguf:tensor_3  -  Ok172sha1      a9cba73e2d90f2ee3dae2548caa42bef3fe6a96c  test.gguf:tensor_3  -  Ok173sha256    423b044e016d8ac73c39f23f60bf01bedef5ecb03c0230accd824c91fe86f1a1  test.gguf:tensor_3  -  Ok174xxh64     1257733306b7992d  test.gguf:tensor_4  -  Ok175sha1      d7bc61db93bb685ce9d598da89717c66729b7543  test.gguf:tensor_4  -  Ok176sha256    79737cb3912d4201384cf7f16a1a37ff7823f23ea796cb205b6ca361ab9e3ebf  test.gguf:tensor_4  -  Ok177xxh64     d238d16ba4711e58  test.gguf:tensor_5  -  Ok178sha1      0706566c198fe1072f37e0a5135b4b5f23654c52  test.gguf:tensor_5  -  Ok179sha256    60949be8298eced0ecdde64487643d018407bd261691e061d9e9c3dbc9fd358b  test.gguf:tensor_5  -  Ok180xxh64     3fbc3b65ab8c7f39  test.gguf:tensor_6  -  Ok181sha1      73922a0727226a409049f6fc3172a52219ca6f00  test.gguf:tensor_6  -  Ok182sha256    574f4c46ff384a3b9a225eb955d2a871847a2e8b3fa59387a8252832e92ef7b0  test.gguf:tensor_6  -  Ok183xxh64     c22021c29854f093  test.gguf:tensor_7  -  Ok184sha1      efc39cece6a951188fc41e354c73bbfe6813d447  test.gguf:tensor_7  -  Ok185sha256    4c0410cd3c500f078ae5b21e8dc9eb79e29112713b2ab58a882f82a3868d4d75  test.gguf:tensor_7  -  Ok186xxh64     936df61f5d64261f  test.gguf:tensor_8  -  Ok187sha1      c2490296d789a4f34398a337fed8377d943d9f06  test.gguf:tensor_8  -  Ok188sha256    c4401313feeba0261275c3b25bd2d8fe40ce04e0f440c2980ed0e9674c30ff01  test.gguf:tensor_8  -  Ok189xxh64     93fd20c64421c081  test.gguf:tensor_9  -  Ok190sha1      7047ce1e78437a6884337a3751c7ee0421918a65  test.gguf:tensor_9  -  Ok191sha256    23d57cf0d7a6e90b0b3616b41300e0cd354781e812add854a5f95aa55f2bc514  test.gguf:tensor_9  -  Ok192xxh64     5a54d3aad816f302  test.gguf  -  Ok193sha1      d15be52c4ff213e823cb6dd13af7ee2f978e7042  test.gguf  -  Ok194sha256    7dd641b32f59b60dbd4b5420c4b0f6321ccf48f58f6ae201a3dbc4a58a27c6e4  test.gguf  -  Ok195 196Verification results for test.gguf.manifest - Success197```198 199 200## Crypto/Hash Libraries Used201 202These micro c libraries dependencies was installed via the [clib c package manager](https://github.com/clibs)203 204- https://github.com/Cyan4973/xxHash205- https://github.com/clibs/sha1/206- https://github.com/jb55/sha256.c207