Mungert/OpenMath-CodeLlama-13b-Python-hf-GGUF
0870
1---2license: llama23base_model:4- codellama/CodeLlama-13b-Python-hf5datasets:6- nvidia/OpenMathInstruct-17language:8- en9tags:10- nvidia11- code12- math13---14 15# <span style="color: #7FFF7F;">OpenMath-CodeLlama-13b-Python-hf GGUF Models</span>16 17 18## <span style="color: #7F7FFF;">Model Generation Details</span>19 20This model was generated using [llama.cpp](https://github.com/ggerganov/llama.cpp) at commit [`e743cddb`](https://github.com/ggerganov/llama.cpp/commit/e743cddb60dc3a8815b9de7dd7d5c491e61b2259).21 22 23 24 25 26---27 28## <span style="color: #7FFF7F;">Quantization Beyond the IMatrix</span>29 30I've been experimenting with a new quantization approach that selectively elevates the precision of key layers beyond what the default IMatrix configuration provides.31 32In my testing, standard IMatrix quantization underperforms at lower bit depths, especially with Mixture of Experts (MoE) models. To address this, I'm using the `--tensor-type` option in `llama.cpp` to manually "bump" important layers to higher precision. You can see the implementation here: 33👉 [Layer bumping with llama.cpp](https://github.com/Mungert69/GGUFModelBuilder/blob/main/model-converter/tensor_list_builder.py)34 35While this does increase model file size, it significantly improves precision for a given quantization level.36 37### **I'd love your feedback—have you tried this? How does it perform for you?**38 39 40 41 42---43 44<a href="https://readyforquantum.com/huggingface_gguf_selection_guide.html" style="color: #7FFF7F;">45 Click here to get info on choosing the right GGUF model format46</a>47 48---49 50 51 52<!--Begin Original Model Card-->53 54 55 56# OpenMath-CodeLlama-13b-Python-hf57 58OpenMath models were designed to solve mathematical problems by integrating text-based reasoning with code blocks59executed by Python interpreter. The models were trained on [OpenMathInstruct-1](https://huggingface.co/datasets/nvidia/OpenMathInstruct-1),60a math instruction tuning dataset with 1.8M problem-solution pairs generated using permissively licensed61[Mixtral-8x7B](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) model.62 63<table border="1">64 <tr>65 <td></td>66 <td colspan="2" style="text-align: center;">greedy</td>67 <td colspan="2" style="text-align: center;">majority@50</td>68 </tr>69 <tr>70 <td style="text-align: center;">model</td>71 <td style="text-align: center;">GSM8K</td>72 <td style="text-align: center;">MATH</td>73 <td style="text-align: center;">GMS8K</td>74 <td style="text-align: center;">MATH</td>75 </tr>76 <tr>77 <td style="text-align: right;">OpenMath-CodeLlama-7B (<a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-7b-Python">nemo</a> | <a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-7b-Python-hf">HF</a>)</td>78 <td style="text-align: center;">75.9</td>79 <td style="text-align: center;">43.6</td>80 <td style="text-align: center;">84.8</td>81 <td style="text-align: center;">55.6</td>82 </tr>83 <tr>84 <td style="text-align: right;">OpenMath-Mistral-7B (<a href="https://huggingface.co/nvidia/OpenMath-Mistral-7B-v0.1">nemo</a> | <a href="https://huggingface.co/nvidia/OpenMath-Mistral-7B-v0.1-hf">HF</a>)</td>85 <td style="text-align: center;">80.2</td>86 <td style="text-align: center;">44.5</td>87 <td style="text-align: center;">86.9</td>88 <td style="text-align: center;">57.2</td>89 </tr>90 <tr>91 <td style="text-align: right;">OpenMath-CodeLlama-13B (<a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-13b-Python">nemo</a> | <a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-13b-Python-hf">HF</a>)</td>92 <td style="text-align: center;">78.8</td>93 <td style="text-align: center;">45.5</td>94 <td style="text-align: center;">86.8</td>95 <td style="text-align: center;">57.6</td>96 </tr>97 <tr>98 <td style="text-align: right;">OpenMath-CodeLlama-34B (<a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-34b-Python">nemo</a> | <a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-34b-Python-hf">HF</a>)</td>99 <td style="text-align: center;">80.7</td>100 <td style="text-align: center;">48.3</td>101 <td style="text-align: center;">88.0</td>102 <td style="text-align: center;">60.2</td>103 </tr>104 <tr>105 <td style="text-align: right;">OpenMath-Llama2-70B (<a href="https://huggingface.co/nvidia/OpenMath-Llama-2-70b">nemo</a> | <a href="https://huggingface.co/nvidia/OpenMath-Llama-2-70b-hf">HF</a>)</td>106 <td style="text-align: center;"><b>84.7</b></td>107 <td style="text-align: center;">46.3</td>108 <td style="text-align: center;">90.1</td>109 <td style="text-align: center;">58.3</td>110 </tr>111 <tr>112 <td style="text-align: right;">OpenMath-CodeLlama-70B (<a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-70b-Python">nemo</a> | <a href="https://huggingface.co/nvidia/OpenMath-CodeLlama-70b-Python-hf">HF</a>)</td>113 <td style="text-align: center;">84.6</td>114 <td style="text-align: center;"><b>50.7</b></td>115 <td style="text-align: center;"><b>90.8</b></td>116 <td style="text-align: center;"><b>60.4</b></td>117 </tr>118</table>119 120The pipeline we used to produce these models is fully open-sourced!121 122- [Code](https://github.com/Kipok/NeMo-Skills)123- [Models](https://huggingface.co/collections/nvidia/openmath-65c5619de2ba059be0775014)124- [Dataset](https://huggingface.co/datasets/nvidia/OpenMathInstruct-1)125 126See our [paper](https://arxiv.org/abs/2402.10176) for more details!127 128# How to use the models?129 130Try to [run inference with our models](https://github.com/Kipok/NeMo-Skills/blob/main/docs/inference.md) with just a few commands!131 132# Reproducing our results133 134We provide [all instructions](https://github.com/Kipok/NeMo-Skills/blob/main/docs/reproducing-results.md) to fully reproduce our results.135 136# Improving other models137 138To improve other models or to learn more about our code, read through the docs below.139 140- [NeMo-Skills Pipeline](https://github.com/Kipok/NeMo-Skills)141 - [Generating synthetic data](https://github.com/Kipok/NeMo-Skills/blob/main/docs/synthetic-data-generation.md)142 - [Finetuning models](https://github.com/Kipok/NeMo-Skills/blob/main/docs/finetuning.md)143 - [Evaluating models](https://github.com/Kipok/NeMo-Skills/blob/main/docs/evaluation.md)144 145In our pipeline we use [NVIDIA NeMo](https://www.nvidia.com/en-us/ai-data-science/generative-ai/nemo-framework/),146an end-to-end, cloud-native framework to build, customize, and deploy generative AI models anywhere.147It includes training and inferencing frameworks, guardrailing toolkits, data curation tools, and pretrained models,148offering enterprises an easy, cost-effective, and fast way to adopt generative AI.149 150# Citation151 152If you find our work useful, please consider citing us!153 154```bibtex155@article{toshniwal2024openmath,156 title = {OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset},157 author = {Shubham Toshniwal and Ivan Moshkov and Sean Narenthiran and Daria Gitman and Fei Jia and Igor Gitman},158 year = {2024},159 journal = {arXiv preprint arXiv: Arxiv-2402.10176}160}161```162 163# License164 165The use of this model is governed by the [Llama 2 Community License Agreement](https://ai.meta.com/llama/license/)166 167<!--End Original Model Card-->168 169---170 171# <span id="testllm" style="color: #7F7FFF;">🚀 If you find these models useful</span>172 173Help me test my **AI-Powered Quantum Network Monitor Assistant** with **quantum-ready security checks**: 174 175👉 [Quantum Network Monitor](https://readyforquantum.com/?assistant=open&utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface_repo_readme) 176 177 178The full Open Source Code for the Quantum Network Monitor Service available at my github repos ( repos with NetworkMonitor in the name) : [Source Code Quantum Network Monitor](https://github.com/Mungert69). You will also find the code I use to quantize the models if you want to do it yourself [GGUFModelBuilder](https://github.com/Mungert69/GGUFModelBuilder)179 180💬 **How to test**: 181 Choose an **AI assistant type**: 182 - `TurboLLM` (GPT-4.1-mini) 183 - `HugLLM` (Hugginface Open-source models) 184 - `TestLLM` (Experimental CPU-only) 185 186### **What I’m Testing** 187I’m pushing the limits of **small open-source models for AI network monitoring**, specifically: 188- **Function calling** against live network services 189- **How small can a model go** while still handling: 190 - Automated **Nmap security scans** 191 - **Quantum-readiness checks** 192 - **Network Monitoring tasks** 193 194🟡 **TestLLM** – Current experimental model (llama.cpp on 2 CPU threads on huggingface docker space): 195- ✅ **Zero-configuration setup** 196- ⏳ 30s load time (slow inference but **no API costs**) . No token limited as the cost is low.197- 🔧 **Help wanted!** If you’re into **edge-device AI**, let’s collaborate! 198 199### **Other Assistants** 200🟢 **TurboLLM** – Uses **gpt-4.1-mini** :201- **It performs very well but unfortunatly OpenAI charges per token. For this reason tokens usage is limited. 202- **Create custom cmd processors to run .net code on Quantum Network Monitor Agents**203- **Real-time network diagnostics and monitoring**204- **Security Audits**205- **Penetration testing** (Nmap/Metasploit) 206 207🔵 **HugLLM** – Latest Open-source models: 208- 🌐 Runs on Hugging Face Inference API. Performs pretty well using the lastest models hosted on Novita.209 210### 💡 **Example commands you could test**: 2111. `"Give me info on my websites SSL certificate"` 2122. `"Check if my server is using quantum safe encyption for communication"` 2133. `"Run a comprehensive security audit on my server"`2144. '"Create a cmd processor to .. (what ever you want)" Note you need to install a [Quantum Network Monitor Agent](https://readyforquantum.com/Download/?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface_repo_readme) to run the .net code on. This is a very flexible and powerful feature. Use with caution!215 216### Final Word217 218I fund the servers used to create these model files, run the Quantum Network Monitor service, and pay for inference from Novita and OpenAI—all out of my own pocket. All the code behind the model creation and the Quantum Network Monitor project is [open source](https://github.com/Mungert69). Feel free to use whatever you find helpful.219 220If you appreciate the work, please consider [buying me a coffee](https://www.buymeacoffee.com/mahadeva) ☕. Your support helps cover service costs and allows me to raise token limits for everyone.221 222I'm also open to job opportunities or sponsorship.223 224Thank you! 😊225