mlboydaisuke/Florence-2-base-ExecuTorch

Florence-2-base — ExecuTorch (vision + text encoder + text decoder)

One set of weights that captions, detects, reads text and grounds phrases, with the task chosen by the prompt you send. 0.23 B parameters. Three .pte files, split the way Whisper is on this shelf and for the same reason: the vision tower and the text encoder run once per image, the decoder runs once per generated token.

vision : pixel_values (1,3,768,768)                        -> image_features (1,577,768)
encoder: (image_features, input_ids (1,32), mask (1,32))   -> hidden (1,609,768)
decoder: (hidden, mask (1,32), decoder_input_ids (1,128))  -> logits (1,128,51328)

Every file takes and returns fp32 tensors, so a precision is a file swap, and the three parts can be mixed — an fp16 vision tower pairs with an fp32 encoder.

Three sets: 1242 MB all-fp32, 574 MB all-int8, 543 MB Core ML. The int8 set is the one to reach for on Android — it is smaller than fp32 by more than half and no slower, and it returns the same captions (below). The int8 recipe is dynamic quantisation, which reaches the linear layers and leaves the 51328×768 token embedding table in fp32; that table is 158 MB of the decoder's file, which is why the decoder halves rather than quarters.

\*Mac arm64, single process, median of 10 — a reference point for relative cost, not a device number. Torch eager fp32 on the same machine: vision 466.0 ms, encoder 30.8 ms, decoder 23.5 ms.

A caption of n tokens costs one vision pass, one encoder pass and n decoder passes: about 0.95 s for a 15-token caption on the fp32 XNNPACK set, 0.88 s on the int8 set, 0.14 s on Core ML, on this Mac. The decoder graph is a fixed 128-token window, so every step costs the same whether it is the first token or the fiftieth.

Running it

1. The image. RGB, divide by 255, ImageNet normalise (mean .485/.456/.406, std .229/.224/.225), bicubic resize to 768×768. No crop. Run the vision .pte.

2. The prompt. Florence-2's task tokens are shorthand the processor expands into a sentence before tokenising — the model never sees <CAPTION>:

Tokenise that sentence as <s> sentence </s> with the repo's tokenizer, right-pad to 32 with the pad id (1), and build an attention_mask that is 1 on the real tokens and 0 on the padding. Run the encoder .pte with (image_features, input_ids, mask).

The 577 image tokens are handled inside the graphs. The prompt sequence the original model sees is <image>×577 + <s> prompt </s>, and because the image tokens are a contiguous prefix, the encoder here concatenates the vision features in front of the text embeddings instead of scattering them into placeholder positions. Same arithmetic, no data-dependent mask, and nothing for the caller to line up.

3. Greedy decoding. Fill a (1,128) int64 window with the pad id, write the decoder start token (2) at position 0, then for step t = 0, 1, 2, ...:

logits = decoder(hidden, mask, window)      # mask is the same one the encoder took
ban every token that would repeat a 3-gram already in the output   # <- see below
next   = argmax(logits[0, t])
if next == 2: stop                          # </s>
window[0, t + 1] = next

Detokenise the collected ids with the repo's tokenizer. There is no KV cache: the decoder is a plain static graph over the window, which is what makes it a single .pte with no state to carry between calls.

Implement the 3-gram ban. no_repeat_ngram_size: 3 is in the model's own generation_config.json, and it is not decoration: Florence-2-large's decoder returns <s> as its argmax three times in a row and only that ban moves it on to the caption — without it the loop emits <s> forever and returns an empty string. This base model happens to move on by itself on every image tested here, so a plain argmax loop gives the same five captions. That is luck, not a contract.

4. Reading a detection answer. Grounded tasks answer with <loc_N> tokens, N in 0..999. Four in a row are a box, and each coordinate is (N + 0.5) × side / 1000 in the original image's pixels — side being the image's width for x and its height for y, not 768. That is the same arithmetic the processor's own parser does.

Verification

The three wrappers reproduce Florence2ForConditionalGeneration exactly: composition max_abs_diff 0.000e+00 against the full model's logits on the same image and prompt.

End to end through the three .pte files, greedy <CAPTION> on five photographs against the same decoding in eager PyTorch: 5/5 captions identical, character for character. Correlation is measured per part in the table above; the caption test is what says the split, the prompt layout and the decode agree.

A black and white photo of a person playing a piano.
A wooden walkway leading to the ocean on a sunny day.
A group of dead trees in a forest under a cloudy sky.
A long empty road in the middle of a forest.
A couple of wooden benches sitting on top of a field of leaves.

The int8 set answers with the same five sentences, character for character, with all three parts quantised at once — which is how they would be used together.

python convert/check_florence2.py fp32     # or int8

Not shipped, and why

fp16 for the encoder and the decoder does not export. BART clamps its activations when, and only when, they are half precision:

if hidden_states.dtype == torch.float16 and not torch.isfinite(hidden_states).all():

In fp32 that line short-circuits and never reaches the graph. Halve the model and it becomes a question about values torch.export cannot answer, and export stops with GuardOnDataDependentSymNode. The clamp could be shimmed away, but then the shipped graph would lack the overflow guard the original has, which is not a trade worth making silently. Nothing is lost by it: int8 is both smaller and faster than fp16 here, and the Core ML builds compute in fp16 internally anyway, holding at correlation 0.999993 and 0.999995.

The fp16 vision tower ships and is probably not what you want. It passes both gates — 196.2 MB at correlation 0.999946 — but the int8 tower is 112.5 MB and three times faster (334.7 ms against 1034.5 ms), because XNNPACK has no fp16 kernels here and inserts casts instead. It is in the repo for anyone who needs more fidelity than int8 without carrying the fp32 file.

Conversion notes

Converted from florence-community/Florence-2-base, the transformers-format mirror of microsoft/Florence-2-base — same MIT weights. The original repo predates the in-tree implementation and its weight names do not match it: loading it into Florence2ForConditionalGeneration prints a load report where every key is unexpected and hands back a randomly initialised model without raising.

The checkpoint declares torch_dtype: float16 and transformers honours it, so from_pretrained must be given dtype=torch.float32 explicitly. Without it the "fp32" build serialises fp16 weights — 196 MB for a 90 M parameter tower — and the lowered vision graph returns NaN.

• —Source: microsoft/Florence-2-base (via florence-community mirror)
• —License: MIT

torch.export -> toedgetransformandlower(partitioner) -> .pte (conversion scripts: executorch-models)