litert-community/CLIPSeg-rd64-LiteRT

Measured on device (edge-compat, clipsegdecoder): Galaxy S26 · LiteRT 2.2.0 · GPU (ML Drift) · 3.25 ms p50 (2026-08-25); Galaxy S26 · LiteRT 2.2.0 · NPU (QNN/HTP) · 1.67 ms p50 (2026-08-25); Raspberry Pi 5 · LiteRT 2.2.0.dev20260804 · CPU/XNNPACK, 4 threads · 19.0 ms p50 (2026-08-31); browser · Chromium 151 on M4 Max · LiteRT.js 2.5.3 · WebGPU · 4.11 ms p50 · output matches CPU (2026-08-11). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/clipseg-rd64clipsegdecoder/CARD.md

Measured on device (edge-compat, clipsegtextfp16): Galaxy S26 · LiteRT 2.2.0 · GPU (ML Drift) · 6.20 ms p50 (2026-08-26); Galaxy S26 · LiteRT 2.2.0 · NPU (QNN/HTP) · 2.10 ms p50 (2026-08-26); Raspberry Pi 5 · LiteRT 2.2.0.dev20260804 · CPU/XNNPACK, 4 threads · 69.9 ms p50 (2026-08-31); browser · Chromium 151 on M4 Max · LiteRT.js 2.5.3 · WebGPU · 8.71 ms p50 · output matches CPU (2026-08-11). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/clipseg-rd64_clipsegtext_fp16/CARD.md

Measured on device (edge-compat, clipsegvisionfp16): Galaxy S26 · LiteRT 2.2.0 · GPU (ML Drift) · 35.5 ms p50 (2026-08-26); Galaxy S26 · LiteRT 2.2.0 · NPU (QNN/HTP) · 15.9 ms p50 (2026-08-26); Raspberry Pi 5 · LiteRT 2.2.0.dev20260804 · CPU/XNNPACK, 4 threads · 857 ms p50 (2026-08-31); browser · Chromium 151 on M4 Max · LiteRT.js 2.5.3 · WebGPU · 41.0 ms p50 · output differs from CPU (max rel diff 3.7) (2026-08-11). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/clipseg-rd64_clipsegvision_fp16/CARD.md

CLIPSeg rd64 — LiteRT on-device text-prompted segmentation

CLIPSeg (CVPR 2022, Apache-2.0) re-authored for LiteRT: type what you want to segment ("a cat", "the sky") and get a mask — no fixed class list. Three graphs — CLIP text and vision encoders on the CompiledModel GPU, the tiny 3-layer decoder on CPU (its 4-head/headdim-16 attention fp16-miscomputes on the Mali delegate; the 12-head/headdim-64 vision encoder survives at 0.998).

[image]

Input | prompt `"a dog"` | prompt `"the grass"` — the same image, two prompts, masks from the on-device model. Photo: ["Lily the Golden Retriever in the grass"](https://commons.wikimedia.org/wiki/File:Lily_the_Golden_Retriever_in_the_grass.jpg) (Wikimedia Commons, Public Domain).

Verified on a Pixel 8a: text 761/761 GPU (~8.7 ms) + vision 613/613 GPU (~8.2 ms) + decoder CPU (exact); end-to-end device-vs-PyTorch logits corr 0.99998, mask IoU 0.9986.

Files

Minimal usage (Python)

import numpy as np, torch
from PIL import Image
from transformers import CLIPSegProcessor
from ai_edge_litert.interpreter import Interpreter

proc = CLIPSegProcessor.from_pretrained("CIDAS/clipseg-rd64-refined")
img = proc(images=Image.open("photo.jpg"), return_tensors="pt")["pixel_values"].numpy()  # [1,3,352,352]

vis = Interpreter("clipseg_vision_fp16.tflite"); vis.allocate_tensors()
vis.set_tensor(vis.get_input_details()[0]["index"], img); vis.invoke()
t = [vis.get_tensor(o["index"]) for o in sorted(vis.get_output_details(), key=lambda o: o["index"])]  # t3,t6,t9

# cond[512] from the text encoder (token-emb lookup -> text graph -> EOT row @ text_projection)
dec = Interpreter("clipseg_decoder.tflite"); dec.allocate_tensors()   # CPU (exact)
ins = dec.get_input_details()
for d, arr in zip(ins, [t[0], t[1], t[2], cond]):   # cond: [1,512] float32
    dec.set_tensor(d["index"], arr.astype(np.float32))
dec.invoke()
mask = 1 / (1 + np.exp(-dec.get_tensor(dec.get_output_details()[0]["index"])[0]))   # sigmoid, [352,352]

Kotlin (Android)

// vision + text: Accelerator.GPU; decoder: Accelerator.CPU
val vis = CompiledModel.create(File(dir,"clipseg_vision_fp16.tflite").path, CompiledModel.Options(Accelerator.GPU), null)
val dec = CompiledModel.create(File(dir,"clipseg_decoder.tflite").path, CompiledModel.Options(Accelerator.CPU), null)
// vision: image[1,3,352,352] -> t3,t6,t9   decoder: (t3,t6,t9,cond[512]) -> logits[1,352,352]
// text graph + host BPE/emb-lookup/text_projection produce cond; see ClipSeg.kt in the LiteRT sample.
val logits = decOut[0].readFloat()   // sigmoid -> mask

Conversion

Re-authored with litert-torch: qkv-3D-BMM attention, quick-GELU, baked interpolated pos-embed (14²→22² @352), host-side token-embedding lookup, safe_ln_up (up-scaled LayerNorm keeping the eps fp16-normal), convT4x4 (exact non-overlapping ConvTranspose as 1×1-conv + 4-D interleave). The decoder ships on CPU because its small-head-dim attention fp16-miscomputes on the Mali GPU delegate (re-authoring is exact — desktop fp16 corr 0.999996).

Upstream

CIDAS/clipseg-rd64-refined (Apache-2.0). Please cite Lüddecke & Ecker, Image Segmentation Using Text and Image Prompts (CVPR 2022).

Performance

Measured on a Pixel 8a (Tensor G3, Android 16) with the standard TFLite `benchmark_model` tool — 10 warm-up runs then 50 timed runs, reported as the tool's mean.

Any on-device figure recorded when this model shipped came from a different runtime. It was taken through LiteRT's own CompiledModel accelerator (logcat reports it as LITERT_CL), which is the path the Kotlin sample app and the LiteRT API use, and it appears elsewhere on this card. The rows above are the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. The two are not comparable, so read the rows above as a reproducible floor rather than as this model's speed on LiteRT.

On this delegate the CPU is the faster choice for clipseg_decoder.tflite (11.7 ms on CPU against 96.2 ms on GPU), clipseg_text_fp16.tflite (53.8 ms on CPU against 154.4 ms on GPU) — worth knowing before you reach for the GPU on a mid-range phone.

Note that the GPU does not take the whole graph here (44 / 196 in clipseg_decoder.tflite, 32 / 761 in clipseg_text_fp16.tflite, 51 / 613 in clipseg_vision_fp16.tflite); the remainder runs on the CPU and the split costs a per-partition round trip.

Snapdragon NPU (Hexagon)

• —clipseg_decoder.tflite — the NPU is 1.95x faster than the GPU (1.67 ms against 3.25 ms) and loads 7.72x faster (102 ms against 784 ms).
• —clipseg_text_fp16.tflite — the NPU is 2.95x faster than the GPU (2.10 ms against 6.20 ms) and loads 7.55x faster (138 ms against 1041 ms).
• —clipseg_vision_fp16.tflite — the NPU is 2.23x faster than the GPU (15.90 ms against 35.51 ms) and loads 8.17x faster (192 ms against 1569 ms).

Measured on a Samsung Galaxy S26 (Snapdragon 8 Elite Gen 5 / SM8850, Hexagon v81, Android 16) with LiteRT CompiledModel 2.2.0, one accelerator per process, 5 warm-up runs then N=50 timed runs, median reported. Every run held thermal status NONE throughout. Headroom 0.71–0.74, where 1.0 is the throttling threshold.

The NPU rows ran the published file unchanged. LiteRT compiled it for the Hexagon on the device at first load. Those first compiles took 1.0 s to 18 s here. The load column above is the cached load every later run pays. Recipe and the runtime libraries it needs: NPU guide.

GPU wiring: GPU guide.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with the LiteRT `benchmark_model` tool from litert-cli-nightly 0.2.0.dev20260805: CPU inference (XNNPACK, 4 threads), 3 invocations per file of 10 warm-up plus 50 timed runs (the tool caps a phase at 150 s, so very slow graphs run fewer — the Runs column is the actual timed total). The latency is the median across invocations; the spread is the min–max over all timed runs. No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0).