Modelpublic

model-rampage/BareTorch-500M-Base

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes222downloads

Model Card

🐻🔥 BareTorch-500M-Base

BareTorch-500M Base is a foundational sub-quadratic language model built under a pure GEMM-compliant, kernel-free paradigm. The model combines CS-LRAD (Chunk-Segmented Low-Rank Associative Delta Engine) recurrent layers with standard Transformer multi-head self-attention in a 3:1 interleaved hybrid topology ($3\times\text{CS-LRAD} \to 1\times\text{Transformer}$).

By structuring sub-quadratic state updates into block-parallel chunk segments ($C=32$), BareTorch bypasses the compilation and hardware lock-in of custom CUDA or Triton kernels, running with $O(N)$ execution and memory efficiency natively across NVIDIA CUDA, Apple Silicon MLX, WebGPU, and TPUs.

📐 Model Architecture Specifications

—Parameters: ~500M (498.2M active parameters)
—Hidden Dimension ($d_{model}$): 1,152
—Total Layers: 24 (Interleaved 18x CS-LRAD + 6x Transformer)
—Attention Heads: 16 ($d_{head} = 72$)
—CS-LRAD Subspace Rank ($r$): 8
—Chunk Size ($C$): 32 tokens
—Tokenizer: HuggingFaceTB/SmolLM2-360M (Vocab size: 49,152)
—Context Window: Up to 32,768 tokens

🏋️ Pre-Training Runway Specs & Loss Convergence

—Training Runway: 100 Billion Tokens ($190,735$ optimization steps)
—Hardware Cluster: $4\times$ NVIDIA H100 SXM (80GB VRAM)
—Global Batch Size: $524,288$ tokens/step ($256$ sequences of length $2048$)
—Optimizer & LR: AdamW ($ ext{LR}_{peak} = 6 \times 10^{-4}$, weight decay $0.1$, cosine decay scheduler with $2,000$ warmup steps)

Pre-Training Evaluation Loss Milestones

Optimization Step	Tokens Processed	Train Loss	Eval Loss
Step 1,000	~0.52 Billion	5.2876	--
Step 10,000	~5.24 Billion	2.7632	2.6943
Step 50,000	~26.21 Billion	2.5275	2.4836
Step 100,000	~52.43 Billion	2.4324	2.3965
Step 150,000	~78.64 Billion	2.3355	2.3054
Step 190,735 (Final)	100.0 Billion	2.2901	2.2690

📊 Zero-Shot Downstream Benchmarks

Task / Benchmark	Metric	Score
HellaSwag	Acc (Norm)	43.69%
ARC Easy	Acc (Norm)	53.58%
ARC Challenge	Acc (Norm)	28.92%
WinoGrande	Accuracy	51.30%
MMLU (Overall 57-Subject Average)	Accuracy	24.70%
├─ MMLU STEM	Accuracy	23.53%
├─ MMLU Humanities	Accuracy	24.87%
├─ MMLU Social Sciences	Accuracy	24.86%
└─ MMLU Other	Accuracy	25.46%

⚡ Long-Context Inference Hardware Scaling (32,768 Context)

BareTorch replaces context-dependent KV-caches with constant-sized $O(1)$ recurrent state updates, eliminating memory bus bottlenecks and out-of-memory crashes on long-context workloads.

1. Discrete CUDA GPU (NVIDIA RTX 4090 - 24GB)

Baseline Pair	Context	Prefill Latency	Local GPU Decode	Peak VRAM	Advantage
Qwen3 0.6B Match	32,768	314.89 ms vs 4,343.26 ms	164.49 tok/s vs 13.15 tok/s	1.66 GB vs 8.58 GB	12.51x Faster (-80.6% VRAM)
SmolLM2 1.7B Match	32,768	1,134.20 ms vs 4,292.54 ms	98.92 tok/s vs 15.79 tok/s	4.39 GB vs 15.57 GB	6.26x Faster (-71.8% VRAM)
Llama 3.2 1B Match	32,768	598.36 ms vs 2,960.03 ms	169.24 tok/s vs 24.44 tok/s	3.02 GB vs 4.67 GB	6.92x Faster (-35.4% VRAM)
Gemma 2 2B Match	32,768	1,049.72 ms (Baseline: 💥 OOM)	101.51 tok/s (Baseline: 💥 OOM)	5.96 GB (Baseline: 💥 OOM)	Prevents OOM Crashes

2. Apple Silicon Unified Memory (M1 MacBook Pro 16GB - Native MLX)

Baseline Pair	Context	Prefill Latency	Local GPU Decode	Peak VRAM	Advantage
SmolLM2 1.7B Match	32,768	29.56 s vs 59.03 s	29.69 tok/s vs 0.66 tok/s	3.78 GB vs 9.97 GB	44.98x Faster (-62.1% VRAM)
Llama 3.2 1B Match	32,768	18.69 s vs 41.06 s	28.42 tok/s vs 3.29 tok/s	2.81 GB vs 3.71 GB	8.64x Faster (-24.4% VRAM)
Qwen3 0.6B Match	32,768	7.06 s (Baseline: 💥 OOM)	79.55 tok/s (Baseline: 💥 OOM)	1.37 GB (Baseline: 💥 OOM)	Prevents OOM Crashes

💻 Usage & Code Example

python

import torch
from transformers import AutoTokenizer
from baretorch.integration.configuration_baretorch import BareTorchConfig
from baretorch.integration.modeling_baretorch import BareTorchForCausalLM

model_id = "model-rampage/BareTorch-500M-Base"
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-360M")
model = BareTorchForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).cuda()

prompt = "The key innovation of pure GEMM sub-quadratic architectures is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=100)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

📜 Citation

bibtex

@article{kovacevic2026baretorch,
  title={BareTorch: Challenging State-of-The-Art Sequence Mixing Topologies via Kernel-Free, Pure GEMM-Compliant Architectures},
  author={Kovacevic Buvinic, Martin Ignacio},
  journal={BareTorch Framework Laboratory Technical Report},
  year={2026}
}