monsterapi/llama2-7b-tiny-codes-code-generation

1---2license: apache-2.03library_name: peft4tags:5- llama26- llama2-7b7- code generation8- code-generation9- code10- instruct11- instruct-code12- code-alpaca13- alpaca-instruct14- alpaca15- llama7b16- gpt217datasets:18- nampdn-ai/tiny-codes19base_model: meta-llama/Llama-2-7b-hf20---21## Training procedure22We finetuned [Llama 2 7B model](https://huggingface.co/meta-llama/Llama-2-7b-hf) from Meta on [nampdn-ai/tiny-codes](https://huggingface.co/datasets/nampdn-ai/tiny-codes) for ~ 10,000 steps using [MonsterAPI](https://monsterapi.ai) no-code [LLM finetuner](https://docs.monsterapi.ai/fine-tune-a-large-language-model-llm).23 24This dataset contains **1.63 million rows** and is a collection of short and clear code snippets that can help LLM models learn how to reason with both natural and programming languages. The dataset covers a wide range of programming languages, such as Python, TypeScript, JavaScript, Ruby, Julia, Rust, C++, Bash, Java, C#, and Go. It also includes two database languages: Cypher (for graph databases) and SQL (for relational databases) in order to study the relationship of entities. 25 26The finetuning session got completed in 193 minutes and costed us only ~ `$7.5` for the entire finetuning run!27 28#### Hyperparameters & Run details:29- Model Path: meta-llama/Llama-2-7b-hf30- Dataset: nampdn-ai/tiny-codes31- Learning rate: 0.000232- Number of epochs: 1 (10k steps)33- Data split: Training: 90% / Validation: 10%34- Gradient accumulation steps: 135 36### Framework versions37 38- PEFT 0.4.039 40### Loss metrics:41![training loss](train-loss.png "Training loss")