Modelpublic

Codemaster67/Olmo_10M_tok_unichem_fineweb

sourceHugging Faceapache-2.0updated 11d agoView on Hugging Face

0likes41downloads

Model Card

OLMo-7B QLoRA Adapter — Chemistry SMILES CPT

Model Description

This is a QLoRA (Quantized LoRA) adapter trained on top of allenai/OLMo-1B-hf for chemistry SMILES language modelling using the Codemaster67/Unichem_smiles-fineweb-10M dataset.

The base model was loaded in 4-bit precision (NF4 quantization via bitsandbytes with double quantization) and LoRA adapter matrices were trained on top in bfloat16. This is the most memory-efficient training configuration compared to full LoRA and full fine-tuning.

The base model's tokenizer was pre-extended with ~300 SPE (SMILES Pair Encoding) chemistry tokens plus <|start_of_smiles|> / <|end_of_smiles|> special tokens. The embed_tokens and lm_head layers are saved as full (non-LoRA) trainable copies via modules_to_save because they were resized during tokenizer extension.

QLoRA / Quantization Configuration

Parameter	Value
Quantization	NF4 (4-bit)
Double Quantization	True
Compute dtype	bfloat16
Rank (r)	64
Alpha	128
Effective Scaling	2.0
Target Modules	all-linear
Dropout	0.01
RSLoRA	True (rank-stabilized)
Modules to Save	embedtokens, lmhead

Training Details

Parameter	Value
Method	QLoRA (4-bit base + LoRA adapters)
Epochs	1
Learning Rate	3e-05
Optimizer	AdamW 8-bit
Batch Size (per device)	32
Gradient Accumulation	1
Max Sequence Length	512
Warmup Ratio	0.1
Weight Decay	0.01
Effective Batch Size (Batch Size 32 x Gradient Accumulation 1)	32
Scheduler	Cosine
Precision	bf16 (adapters) / 4-bit NF4 (base)
Gradient Checkpointing	True
Training Data	Full dataset (train+test merged), no validation split
Packed Training Sequences	19829

Training Results

Metric	Value
Training Loss	1.3341

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
base_model = AutoModelForCausalLM.from_pretrained(
    "allenai/OLMo-1B-hf", quantization_config=bnb_config, trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, "Codemaster67/Olmo_10M_tok_unichem_fineweb")
tokenizer = AutoTokenizer.from_pretrained("Codemaster67/Olmo_10M_tok_unichem_fineweb", trust_remote_code=True)

smiles_input = "<|start_of_smiles|>CC(=O)Oc1ccccc1C(=O)O<|end_of_smiles|>"
inputs = tokenizer(smiles_input, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Intended Use

Chemistry-domain language modelling, SMILES generation and completion, and downstream molecular property prediction via fine-tuning.

Limitations

—QLoRA adapters only; requires the base model allenai/OLMo-1B-hf loaded in 4-bit to use.
—Trained primarily on SMILES strings; natural-language instruction-following ability may degrade compared to the base OLMo checkpoint.
—No validation set was used during training, so no held-out metrics are reported.