Modelpublic

ARM-Development/Llama-3.1-8B-tabular-1.0

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes63downloads

Model Card

Model Card for `sciencebase-metadata-llama3-8b` (v 1.0)

Model Details

Field	Value
Developed by	Quan Quy, Travis Ping, Tudor Garbulet, Chirag Shah, Austin Aguilar
Contact	quyqm@ornl.gov • pingts@ornl.gov • garbuletvt@ornl.gov • shahch@ornl.gov • aguilaral@ornl.gov
Funded by	U.S. Geological Survey (USGS) & Oak Ridge National Laboratory – ARM Data Center
Model type	Autoregressive LLM, instruction-tuned for structured → metadata generation
Base model	`meta-llama/Llama-3.1-8B-Instruct`
Languages	English (metadata vocabulary)
Finetuned from	`unsloth/Meta-Llama-3.1-8B-Instruct`

Model Description

Fine-tuned on ≈ 9 000 ScienceBase “data → metadata” pairs to automate creation of FGDC/ISO-style metadata records for scientific datasets.

Model Sources

Resource	Link
Repository	<https://huggingface.co/ARM-Development/Llama-3.1-8B-tabular-1.0>
Demo	<https://colab.research.google.com/drive/1saCEFhkBYDhQWkdTwnwiE_-AiWmD6p0f#scrollTo=WeniLP-Ah1QL>

Uses

Direct Use

Generate schema-compliant metadata text from a JSON/CSV representation of a ScienceBase item.

Downstream Use

Integrate as a micro-service in data-repository pipelines.

Out-of-Scope

Open-ended content generation, or any application outside metadata curation.

Bias, Risks, and Limitations

—Domain-specific bias toward ScienceBase field names.
—Possible hallucination of fields when prompts are underspecified.

Training Details

Training Data

—~9 k ScienceBase records with curated metadata.

Training Procedure

Hyper-parameter	Value
Max sequence length	100 000
Precision	fp16 / bf16 (auto)
Quantisation	4-bit QLoRA (`load_in_4bit=True`)
LoRA rank / α	16 / 16
Target modules	qproj, kproj, vproj, oproj, gateproj, upproj, down_proj
Optimiser	`adamw_8bit`
LR / schedule	2 × 10⁻⁴, linear
Epochs	1
Effective batch	4 (1 GPU × grad-acc 4)
Trainer	`trl` SFTTrainer + `peft` 0.15.2

Hardware & Runtime

Field	Value
GPU	1 × NVIDIA A100 80 GB
Total training hours	~120 hours
Cloud/HPC provider	ARM Cumulus HPC

Software Stack

Package	Version
Python	3.12.9
PyTorch	2.6.0 + CUDA 12.4
Transformers	4.51.3
Accelerate	1.6.0
PEFT	0.15.2
Unsloth	2025.3.19
BitsAndBytes	0.45.5
TRL	0.15.2
Xformers	0.0.29.post3
Datasets	3.5.0
…

Evaluation

Evaluation still in progress.

Technical Specifications

Architecture & Objective

QLoRA-tuned Llama-3.1-8B-Instruct; causal-LM objective with structured-to-text instruction prompts.

Model Card Authors

Quan Quy, Travis Ping, Tudor Garbulet, Chirag Shah, Austin Aguilar

Model Card for sciencebase-metadata-llama3-8b (v 1.0)