LostGentoo/hf-inference-endpoint-benchmarks

Raw benchmark result files

Raw JSON outputs from the sessions described in benchmarking-methodology.md. Model: Qwen3.5-4B family, hf-endpoints deployed via the configs documented in cli-and-api.md.

Short-prompt decode comparison (valid metric — prompt negligible vs output, see trap #1 in methodology doc)

Long-context (~6.8K prompt tokens) — FLAWED METRIC, kept for reference only

Do not use these for conclusions. aggregate_tps_wallclock = completion_tokens / total_wall_s conflates prefill and decode time — see trap #1. Led to an incorrect "llama.cpp beats vLLM at long context" conclusion that was retracted. Kept only so the mistake and its diagnosis are traceable.

Long-context (~5.5-6.8K prompt tokens) — corrected TTFT/decode-tok/s split, streaming-based

These are the trustworthy long-context numbers (directional — n=1 per point, see trap #3 in methodology doc).

Field reference

• —ttft_s / avg_ttft_s: time from request start to first streamed content token (prefill-dominated).
• —decode_tps / avg_decode_tps_per_stream: tokens/sec between first and last streamed token for a single stream (steady-state decode).
• —sum_decode_tps_all_streams / aggregate_tps_wallclock: naive sum or wall-clock-derived aggregate — see which metric each file actually used before comparing across files, they are not all the same formula (short-prompt files use completion_tokens/wall_s per stream summed; v2 files separate TTFT from decode cleanly).

Embedding models — Qwen3-Embedding-8B, single-shot latency/throughput, L4

Different task shape from everything above: embeddings are a single prefill forward pass (no decode loop), so there's no TTFT/decode split — the harness measures whole-request wall-clock latency and usage.prompt_tokens / wall_s throughput directly. Both endpoints served via /v1/embeddings, exact-token-length prompts built with the real Qwen3 tokenizer (distinct content per request, nonce-prefixed — avoids prefix-cache skew, trap #4), concurrency capped at 4 to match llama.cpp's -np 4 hard slot limit (trap #5), 3-repeat waves per cell (trap #3). Pooling verified correct before trusting results: within-endpoint paraphrase-vs-unrelated discrimination PASS on both engines, cross-endpoint cosine similarity on an identical sentence = 0.97 (confirms both use equivalent last-token pooling despite vLLM's --convert embed having no explicit pooler override).

• —llama.cpp: Qwen/Qwen3-Embedding-8B-GGUF Q4KM, mode: embeddings, pooling: last, -np 4 -c 131072 -ctk q8_0 -ctv q8_0 (q8_0 KV cache — needed to fit 4×32760 ctx in 24GB, an uncontrolled variable vs. vLLM's default KV dtype, not intrinsic to the model).
• —vLLM: drawais/Qwen3-Embedding-8B-AWQ-INT4 (compressed-tensors AWQ, architecture tagged Qwen3ForCausalLM — not a registered embedding arch, requires --convert embed to activate pooling; --task embed is deprecated/removed as of the pinned vllm/vllm-openai:v0.23.0 image and fails at argparse), --max-model-len 32768, tensorParallelSize: 1.
• —TEI: Qwen/Qwen3-Embedding-8B unquantized fp16 — TEI has no quantization support at all (confirmed via OpenAPI schema: TeiContainer has no quantize field, CLI --dtype only accepts float16/float32; unlike TGI it doesn't implement AWQ/GPTQ/compressed-tensors/bnb kernels). So this is not a bit-width-matched comparison to the other two — it's TEI's real ceiling, included as a "purpose-built engine, no quant" reference point instead. Image ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 (generic CUDA build covers Ampere/Ada/Hopper/Blackwell — only Turing needs a separate tag), --max-batch-tokens 32768. Pooling not overridden — repo ships 1_Pooling/config.json and TEI's native FlashQwen3 handler defaults correctly despite logging "Could not find a Sentence Transformers config" at boot (verified via the cosine-discrimination check below, not just trusted).
• —Gotcha (llama.cpp + vLLM): requesting exactly 32768 input tokens fails — the server prepends a BOS token, producing 32769 actual tokens and a 400 (exceeds context size / exceeds max_model_len). Used target 32760 as the "full context" tier instead of the nominal 32768 (Qwen3-Embedding-8B's documented native limit).
• —Gotcha (TEI): --max-batch-tokens sizes the warmup allocation directly — setting it to 4 × 32760 (to match the other two engines' concurrency=4/full-ctx cell) OOM'd at warmup on L4 (16GB fp16 weights + oversized batch buffer > 24GB). Capped at 32768 (one full-context request) instead; concurrency beyond that just queues within TEI's dynamic batcher rather than erroring, which is what's actually being measured in the conc=2/4 full-ctx cells.
• —Pooling verification: cosine-similarity discrimination check (paraphrase vs. unrelated sentence) run against all three before trusting numbers — llama.cpp 0.89/0.32, vLLM 0.88/0.34, TEI 0.89/0.33 (all PASS, all closely matched — confirms equivalent last-token pooling across engines despite two of the three having no explicit pooler config).
• —Audited after the fact and root-caused — TEI wasn't batching concurrent requests, and it's a deliberate upstream design choice, not a config miss. Two things found by re-reading TEI's own per-request queue_time/inference_time logs: (1) the harness sent "model": "tei" to TEI, which it doesn't recognize — logged a benign warning on every single TEI request but fell through to the loaded model correctly each time (confirmed via adjacent success log lines), no effect on the numbers; (2) queue_time showed a clean serial staircase at every concurrency ≥2 (e.g. full-ctx/conc=4: 0.8s → 16s → 32s → 48s → 64s, exact multiples of the ~16s inference_time) — requests were being processed close to one-at-a-time, not batched together. Redeployed with --max-batch-requests 16 explicit and fixed the model field, re-ran 4K/8K — numbers unchanged within noise, so that wasn't the lever.
• —Root cause, confirmed via TEI's own source (`core/src/infer.rs`, `core/src/queue.rs`) and the maintainer discussion on [PR #799](https://github.com/huggingface/text-embeddings-inference/pull/799)/[issue #798](https://github.com/huggingface/text-embeddings-inference/issues/798): the batching task dispatches a batch immediately on being notified of any new request — grabbing whatever's already in the internal queue at that instant, with no deliberate collection/accumulation delay (unlike e.g. Triton's dynamic batcher). The output channel to the backend is bounded to capacity 1 (// Bound channel to 1 to be able to prefetch one batch), so at most one batch can be prepared ahead of what's currently executing. Whether concurrent requests land in the same formed batch is a race against per-connection network/HTTP-parsing jitter, not something max-batch-requests governs — that flag only raises a ceiling that was never the binding constraint.
• —This is intentional: a TEI maintainer explicitly rejected the PR that tried to widen this, arguing that on a single GPU, deliberately delaying dispatch to accumulate bigger batches wouldn't help throughput anyway (compute-bound, and genuinely-parallel execution of multiple batches loses to L2 cache eviction — "less than 50% of the speed" per matmul) — it would only hurt p50 latency and break HTTP-cancellation semantics. Not fixable via any client-side or deploy-time config; it's an upstream architectural stance, open-but-effectively-closed as of this writing.
• —Result shape: vLLM has meaningfully higher aggregate throughput than llama.cpp at every context tier and wins decisively at concurrency ≥2. TEI sits in between and is essentially flat across concurrency 1→4 at every context tier (~2.6-2.9K tok/s @ 4K, ~2.5-2.8K tok/s @ 8K, ~2.0K tok/s @ 32760) — confirmed (see audit note above) to be mostly-serial request processing rather than batching-driven scaling, but also no tail-latency cliff like llama.cpp's full-ctx/conc=4 cell (one isolated 503 at conc=1/32760, not reproduced at conc=2 or conc=4 run right after — treated as transient noise, not a systemic issue). llama.cpp's conc=4/32760-token cell had 2/12 requests exceed the 300s client timeout — a real tail-latency cliff under full-context+full-slot load, not a bug; not retried with a longer timeout since >300s per single embedding call is already impractical.
• —Directional only (n=3 per cell, single session) — see trap #3.

Privacy

Live Hugging Face Inference Endpoint hostnames were redacted from embed_bench_results.json before upload (endpoint_host_redacted: true). Throughput/latency numbers are unchanged.