Team Ai
Datasetpublic

uv-scripts/transformers-inference

Transformers Continuous Batching Scripts GPU inference scripts using transformers' native continuous batching (CB). No vLLM dependency required. Why transformers CB? Instant new model support - works with any model supported by transformers, including newly released architectures. No waiting for vLLM to add support. No dependency headaches - no vLLM, flashinfer, or custom wheel indexes. Just transformers + accelerate. Simple HF Jobs setup - no Docker image… See the full description on the dataset page: https://huggingface.co/datasets/uv-scripts/transformers-inference.

sourceHugging Faceupdated 7mo agoView on Hugging Face
2likes15downloads
Dataset Card

Transformers Continuous Batching Scripts

GPU inference scripts using transformers' native continuous batching (CB). No vLLM dependency required.

Why transformers CB?

  • —Instant new model support - works with any model supported by transformers, including newly released architectures. No waiting for vLLM to add support.
  • —No dependency headaches - no vLLM, flashinfer, or custom wheel indexes. Just transformers + accelerate.
  • —Simple HF Jobs setup - no Docker image needed. Just hf jobs uv run.
  • —~95% of vLLM throughput - uses PagedAttention and continuous scheduling for near-vLLM performance.

Available Scripts

generate-responses.py

Generate responses for prompts in a dataset. Supports chat messages and plain text prompts.

Quick Start
bash
# Local (requires GPU)
uv run generate-responses.py \
    username/input-dataset \
    username/output-dataset \
    --prompt-column question

# HF Jobs (single GPU)
hf jobs uv run --flavor l4x1 -s HF_TOKEN \
    https://huggingface.co/datasets/uv-scripts/transformers-inference/raw/main/generate-responses.py \
    username/input-dataset \
    username/output-dataset \
    --prompt-column question \
    --max-tokens 1024

# HF Jobs (multi-GPU for larger models)
hf jobs uv run --flavor l4x4 -s HF_TOKEN \
    https://huggingface.co/datasets/uv-scripts/transformers-inference/raw/main/generate-responses.py \
    username/input-dataset \
    username/output-dataset \
    --model-id Qwen/Qwen3-30B-A3B-Instruct-2507 \
    --messages-column messages \
    --max-batch-tokens 2048 \
    --max-tokens 4096
Example with SmolTalk2
bash
# Generate responses for SmolTalk2 chat data
hf jobs uv run --flavor l4x1 -s HF_TOKEN \
    https://huggingface.co/datasets/uv-scripts/transformers-inference/raw/main/generate-responses.py \
    HuggingFaceTB/smoltalk2 username/smoltalk2-responses \
    --subset SFT \
    --split OpenHermes_2.5_no_think \
    --messages-column messages \
    --max-tokens 256
Parameters
ParameterDefaultDescription
--model-idQwen/Qwen3-4B-Instruct-2507Any HF causal LM model
--messages-columnmessagesColumn with chat messages
--prompt-column-Column with plain text prompts (alternative to messages)
--output-columnresponseName for the generated response column
--temperature0.7Sampling temperature
--top-p0.8Top-p (nucleus) sampling
--top-k20Top-k sampling
--max-tokens4096Maximum tokens to generate per response
--repetition-penalty1.0Repetition penalty
--max-batch-tokens512Token budget per scheduling step (see below)
--dtypebfloat16Model precision (bfloat16, float16, float32)
--attn-implementation`pagedsdpa`Attention backend (`pagedsdpa or paged\flashattention2`)
--max-samplesallLimit to N samples (useful for testing)
--hf-token-HF token (or use HF_TOKEN env var)
--skip-long-promptsTrueSkip prompts exceeding context length
Tuning --max-batch-tokens

This is the key performance parameter. It controls how many tokens the continuous batching scheduler processes per step:

  • —Too low (e.g., 128): GPU underutilized, slow throughput
  • —Too high (e.g., 8192): May cause out-of-memory errors
  • —Default 512: Conservative, works on most GPUs
  • —Recommended for A100/H100: 2048-4096
  • —Recommended for L4: 512-1024

If you hit OOM errors, reduce this value or switch to --dtype float16.

Current Limitations

  • —Single GPU only - device_map="auto" (pipeline parallelism) doesn't work with CB's PagedAttention cache. Transformers does have tensor parallelism (tp_plan="auto") for supported models, but it requires torchrun and is undocumented with CB. For now, use a model that fits on one GPU (e.g., 8B in bf16 on A10G/L4 with 24GB).
  • —Text-only - no vision-language model support yet.

When to use this vs vLLM

Transformers CBvLLM
Best forNew/niche models, simple setup, avoiding dependency issuesMaximum throughput, production serving
Model supportAny transformers model, immediatelyPopular models, may lag on new architectures
Dependenciestransformers + acceleratevllm + flashinfer + custom indexes
Docker imageNot neededvllm/vllm-openai recommended
Multi-GPUSingle GPU only (for now)Tensor parallelism
Performance~95% of vLLM for text generationFastest for supported models
VLM supportNot yetYes

Rule of thumb: Use transformers CB when you want simplicity and broad model support. Use vLLM when you need maximum throughput with well-supported models.