Modelpublic

ruby2022/NYCU-IAlI-DL2026-LLM3-GRPO

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes20downloads

Model Card

NYCU IAI-DL 2026 LLM3 GRPO — Qwen3-14B LoRA Adapter

This repository contains:

—LoRA adapter fine-tuned from Qwen/Qwen3-14B via GRPO reinforcement learning
—Training data (train-reasoning-v2.csv) used for GRPO training

📊 Results

Submission	Inference	Public	Private
Qwen3-14B + GRPO	Majority Vote N=8	0.694	0.698
Qwen3-14B + GRPO	Greedy (N=1)	0.687	0.685

🏆 Competition

Kaggle: nycu-i-al-i-dl-2026-llm-3-grpo

Task: Fine-tune a Chinese-released LLM using GRPO so that it correctly answers Traditional Chinese single-choice questions (A/B/C/D).

🔧 LoRA Configuration

Parameter	Value
Base model	`Qwen/Qwen3-14B`
Rank (r)	16
Alpha	16
Dropout	0.0
Target modules	qproj, kproj, vproj, oproj, gateproj, upproj, down_proj
Trainable params	~1.5%
Training method	GRPO (Group Relative Policy Optimization)

📁 Files

File	Description
`adapter_config.json`	LoRA adapter configuration
`adapter_model.safetensors`	LoRA adapter weights (~245 MB)
`tokenizer.json`	Tokenizer
`tokenizer_config.json`	Tokenizer configuration
`chat_template.jinja`	Qwen3 chat template with thinking enabled
`train-reasoning-v2.csv`	Training dataset (3,199 samples)
`kaggle_test_set_792.csv`	Test set (792 questions, no labels)

🚀 Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model_name = "Qwen/Qwen3-14B"
adapter_name = "ruby2022/NYCU-IAlI-DL2026-LLM3-GRPO"

tokenizer = AutoTokenizer.from_pretrained(adapter_name)
model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_name)
model = model.merge_and_unload()

🏗️ Training Details

—Algorithm: GRPO (Group Relative Policy Optimization)
—Key innovation: Option Shuffling augmentation to prevent reward collapse
—Reward functions: correctness_reward (+2.0/−1.0) + format_reward (+0.5)
—Inference: Majority Voting (N=8, temperature=0.6)
—Thinking mode: Qwen3 native <think>...</think> enabled throughout

See the GitHub repository for full source code, training scripts, and logs.

⚙️ Hardware

—GPU: NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM)
—Training: Single GPU, ~500 steps GRPO