darwinkernelpanic/deepseek-coder-6.7b-instruct-luau

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/> <details><summary>See axolotl config</summary>

axolotl version: 0.13.0.dev0

base_model: deepseek-ai/deepseek-coder-6.7b-instruct
hub_model_id: darwinkernelpanic/deepseek-coder-6.7b-instruct-luau
hub_strategy: end
trust_remote_code: true

load_in_8bit: false
load_in_4bit: true

datasets:
  - path: darwinkernelpanic/luau_corpus_axolotl
    type: completion
    field_instruction: prompt
    field_output: completion

dataset_prepared_path:
val_set_size: 0.05
output_dir: ./outputs/deepseek-luau-finetune

sequence_len: 3072
sample_packing: true
eval_sample_packing: true

adapter: qlora
lora_model_dir:
lora_r: 32
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true

wandb_project: deepseek-luau-finetune
wandb_entity:
wandb_watch:
wandb_name: deepseek-coder-6.7b-luau
wandb_log_model:

gradient_accumulation_steps: 2
micro_batch_size: 6
num_epochs: 3
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 0.0002
bf16: auto
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false

resume_from_checkpoint:
logging_steps: 10
flash_attention: true
warmup_ratio: 0.1
evals_per_epoch: 4
saves_per_epoch: 1
weight_decay: 0.01

fsdp: []
fsdp_config: {}

special_tokens:
  pad_token: "<|EOT|>"

</details><br>

deepseek-coder-6.7b-instruct-luau

This model is a fine-tuned version of deepseek-ai/deepseek-coder-6.7b-instruct on the darwinkernelpanic/luaucorpusaxolotl dataset. It achieves the following results on the evaluation set:

• —Loss: 1.6346
• —Ppl: 5.1272
• —Memory/max Active (gib): 10.65
• —Memory/max Allocated (gib): 10.65
• —Memory/device Reserved (gib): 11.93

Model description

The model was fine-tuned on the Roblox/luau_corpus dataset which was converted to have the "prompt" collum replaced by "text" for compatibility reasons. It was fine-tuned for improved knowledge and performance on Luau code (Roblox's Lua dialect, see luau.org), which should end up improving code quality for Luau and Roblox projects.

Intended uses & limitations

This model is intended for use within applications that use the Luau programming language, including but not limited to

• —Roblox projects
• —Standalone Luau projects (Lune?)

It may have limitations for projects that

• —Use alternative languages
• —Use Lua
• —Non programming related projects

Training and evaluation data

N/A

Training procedure

Trained on 1x RTX 6000Ada

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 0.0002
• —trainbatchsize: 6
• —evalbatchsize: 6
• —seed: 42
• —gradientaccumulationsteps: 2
• —totaltrainbatch_size: 12
• —optimizer: Use OptimizerNames.ADAMWTORCHFUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• —lrschedulertype: cosine
• —lrschedulerwarmup_steps: 16
• —training_steps: 162

Training results

Framework versions

• —PEFT 0.18.0
• —Transformers 4.57.1
• —Pytorch 2.8.0+cu128
• —Datasets 4.4.1
• —Tokenizers 0.22.1