mo7amed-3bdalla7/tinyllama-python-lora

1---2license: apache-2.03base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.04tags:5  - tinyllama6  - lora7  - peft8  - python9  - code10  - fine-tuning11model_type: causal-lm12library_name: transformers13pipeline_tag: text-generation14---15 16# 🐍 TinyLLaMA LoRA - Fine-tuned on Python Code17 18This is a **LoRA fine-tuned version** of [`TinyLlama/TinyLlama-1.1B-Chat-v1.0`](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0) using a subset of Python code from the `codeparrot` dataset. It is trained to generate Python functions and code snippets based on natural language or code-based prompts.19 20## 🔧 Training Details21 22- **Base model**: `TinyLlama/TinyLlama-1.1B-Chat-v1.0`23- **Adapter type**: LoRA (PEFT)24- **Dataset**: `codeparrot/codeparrot-clean-valid[:1000]`25- **Tokenized max length**: 51226- **Trained on**: Apple M3 Pro (MPS backend)27- **Epochs**: 128- **Batch size**: 1 (with gradient accumulation)29 30## 💡 Example Usage31 32```python33from transformers import AutoModelForCausalLM, AutoTokenizer34from peft import PeftModel35 36base_model = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"37adapter_model = "your-username/tinyllama-python-lora"38 39tokenizer = AutoTokenizer.from_pretrained(base_model)40model = AutoModelForCausalLM.from_pretrained(base_model)41model = PeftModel.from_pretrained(model, adapter_model)42 43prompt = "<|python|>\ndef fibonacci(n):"44inputs = tokenizer(prompt, return_tensors="pt")45outputs = model.generate(**inputs, max_new_tokens=100)46print(tokenizer.decode(outputs[0], skip_special_tokens=True))47```48 49## 🧠 Intended Use50Code completion for Python51 52Teaching LLMs Python function structure53 54Experimentation with LoRA on small code datasets55 56##⚠️ Limitations57Trained on a small subset of data (1,000 samples)58 59May hallucinate or generate syntactically incorrect code60 61Not suitable for production use without further fine-tuning and evaluation62 63## 📜 License64Apache 2.0 — same as the base model.