uv-scripts/vllm

vLLM Inference Scripts

Ready-to-run UV scripts for GPU-accelerated inference using vLLM.

These scripts use UV's inline script metadata to automatically manage dependencies - just run with uv run and everything installs automatically!

📋 Available Scripts

vlm-classify.py

Vision Language Model (VLM) image classification with structured output constraints.

Features:

• —🖼️ Process images through state-of-the-art VLMs (Qwen2-VL)
• —🎯 Structured classification using vLLM's GuidedDecodingParams
• —📐 Automatic image resizing to optimize token usage
• —💾 Memory-efficient lazy batch processing
• —🏷️ Simple CLI interface for defining classes
• —🤗 Direct integration with Hugging Face datasets

Usage:

# Basic classification
uv run vlm-classify.py \
    username/input-dataset \
    username/output-dataset \
    --classes "document,photo,diagram,other"

# With custom prompt and image resizing
uv run vlm-classify.py \
    username/input-dataset \
    username/output-dataset \
    --classes "index-card,manuscript,title-page,other" \
    --prompt "What type of historical document is this?" \
    --max-size 768

# Quick test with sample limit
uv run vlm-classify.py \
    davanstrien/sloane-index-cards \
    username/test-output \
    --classes "index,content,other" \
    --max-samples 10

HF Jobs execution:

hf jobs uv run \
    --flavor a10g \
    --image vllm/vllm-openai \
    -s HF_TOKEN \
    https://huggingface.co/datasets/uv-scripts/vllm/raw/main/vlm-classify.py \
    username/input-dataset \
    username/output-dataset \
    --classes "title-page,content,index,other" \
    --max-size 768

Key Parameters:

• —--classes: Comma-separated list of classification categories (required)
• —--prompt: Custom classification prompt (optional, auto-generated if not provided)
• —--max-size: Maximum image dimension in pixels for resizing (reduces token count)
• —--model: VLM model to use (default: Qwen/Qwen2-VL-7B-Instruct)
• —--batch-size: Number of images to process at once (default: 8)
• —--max-samples: Limit number of samples for testing

classify-dataset.py

Batch text classification using BERT-style encoder models (e.g., BERT, RoBERTa, DeBERTa, ModernBERT) with vLLM's optimized inference engine.

Note: This script is specifically for encoder-only classification models, not generative LLMs.

Features:

• —🚀 High-throughput batch processing
• —🏷️ Automatic label mapping from model config
• —📊 Confidence scores for predictions
• —🤗 Direct integration with Hugging Face Hub

Usage:

# Local execution (requires GPU)
uv run classify-dataset.py \
    davanstrien/ModernBERT-base-is-new-arxiv-dataset \
    username/input-dataset \
    username/output-dataset \
    --inference-column text \
    --batch-size 10000

HF Jobs execution:

hf jobs uv run \
    --flavor l4x1 \
    --image vllm/vllm-openai \
    https://huggingface.co/datasets/uv-scripts/vllm/resolve/main/classify-dataset.py \
    davanstrien/ModernBERT-base-is-new-arxiv-dataset \
    username/input-dataset \
    username/output-dataset \
    --inference-column text \
    --batch-size 100000

generate-responses.py

Generate responses for prompts using generative LLMs (e.g., Llama, Qwen, Mistral) with vLLM's high-performance inference engine.

Features:

• —💬 Automatic chat template application
• —📝 Support for both chat messages and plain text prompts
• —🔀 Multi-GPU tensor parallelism support
• —📏 Smart filtering for prompts exceeding context length
• —📊 Comprehensive dataset cards with generation metadata
• —⚡ HF Transfer enabled for fast model downloads
• —🎛️ Full control over sampling parameters
• —🎯 Sample limiting with --max-samples for testing

Usage:

# With chat-formatted messages (default)
uv run generate-responses.py \
    username/input-dataset \
    username/output-dataset \
    --messages-column messages \
    --max-tokens 1024

# With plain text prompts (NEW!)
uv run generate-responses.py \
    username/input-dataset \
    username/output-dataset \
    --prompt-column question \
    --max-tokens 1024 \
    --max-samples 100

# With custom model and parameters
uv run generate-responses.py \
    username/input-dataset \
    username/output-dataset \
    --model-id meta-llama/Llama-3.1-8B-Instruct \
    --prompt-column text \
    --temperature 0.9 \
    --top-p 0.95 \
    --max-model-len 8192

HF Jobs execution (multi-GPU):

hf jobs uv run \
    --flavor l4x4 \
    --image vllm/vllm-openai \
    -e UV_PRERELEASE=if-necessary \
    -s HF_TOKEN \
    https://huggingface.co/datasets/uv-scripts/vllm/raw/main/generate-responses.py \
    davanstrien/cards_with_prompts \
    davanstrien/test-generated-responses \
    --model-id Qwen/Qwen3-30B-A3B-Instruct-2507 \
    --gpu-memory-utilization 0.9 \
    --max-tokens 600 \
    --max-model-len 8000

Multi-GPU Tensor Parallelism

• —Auto-detects available GPUs by default
• —Use --tensor-parallel-size to manually specify
• —Required for models larger than single GPU memory (e.g., 30B+ models)

Handling Long Contexts

The generate-responses.py script includes smart prompt filtering:

• —Default behavior: Skips prompts exceeding maxmodellen
• —Use `--max-model-len`: Limit context to reduce memory usage
• —Use `--no-skip-long-prompts`: Fail on long prompts instead of skipping
• —Skipped prompts receive empty responses and are logged

📚 About vLLM

vLLM is a high-throughput inference engine optimized for:

• —Fast model serving with PagedAttention
• —Efficient batch processing
• —Support for various model architectures
• —Seamless integration with Hugging Face models

🔧 Technical Details

UV Script Benefits

• —Zero setup: Dependencies install automatically on first run
• —Reproducible: Locked dependencies ensure consistent behavior
• —Self-contained: Everything needed is in the script file
• —Direct execution: Run from local files or URLs

Dependencies

Scripts use UV's inline metadata for automatic dependency management:

# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "datasets",
#     "flashinfer-python",
#     "huggingface-hub[hf_transfer]",
#     "torch",
#     "transformers",
#     "vllm",
# ]
# ///

For bleeding-edge features, use the UV_PRERELEASE=if-necessary environment variable to allow pre-release versions when needed.

Docker Image

For HF Jobs, we recommend the official vLLM Docker image: vllm/vllm-openai

This image includes:

• —Pre-installed CUDA libraries
• —vLLM and all dependencies
• —UV package manager
• —Optimized for GPU inference

Environment Variables

• —HF_TOKEN: Your Hugging Face authentication token (auto-detected if logged in)
• —UV_PRERELEASE=if-necessary: Allow pre-release packages when required
• —HF_HUB_ENABLE_HF_TRANSFER=1: Automatically enabled for faster downloads

🔗 Resources

• —vLLM Documentation
• —UV Documentation
• —UV Scripts Organization