diffusers-modular/MiniMax-H3-Pruned-Ref-Delta-Fused-r1024

MiniMax-H3, one transformer for references and keyframes

MiniMax-H3 ships two 37.5 GB transformer partitions: fl2va for first/last-frame conditioning and ref2va for reference conditioning. This is one checkpoint that serves both, so a deployment holds 37.5 GB instead of 75 GB. It is the pruned fl2va partition with a rank-1024 approximation of the ref2va − fl2va weight delta fused into the weights.

[!WARNING] Experimental and mechanically derived — the delta was extracted from two released checkpoints by SVD, not trained. This is not transformer_ref: against the real thing it reaches video-latent cosine 0.875 / 0.897 / 0.691 on three matched reference requests, short of the 0.99 that would make it a drop-in replacement. Use it when one-partition deployment is worth that gap.

[!IMPORTANT] Governed by the MiniMax H3 Community License Agreement, which carries territory exclusions and redistribution conditions. The upstream LICENSE is included and this card is the modification notice.

Inference

Both workflows resolve to the weights in this repo, and each loads only its own slot — pass a workflow=, or the components of both transformer slots get pulled:

import torch
from diffusers import ComponentsManager, ModularPipeline

REPO = "diffusers-modular/MiniMax-H3-Pruned-Ref-Delta-Fused-r1024"

manager = ComponentsManager()
pipe = ModularPipeline.from_pretrained(REPO, workflow="ref2va", components_manager=manager,
                                      trust_remote_code=True)
pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)
manager.enable_auto_cpu_offload(device="cuda")   # 37.5 GB DiT + 62 GB conditioner + two VAEs

References (workflow="ref2va") — image, video and/or audio, in the order they are passed:

from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3ImageReference

out = pipe(
    prompt="The woman from the reference stands in a snowy park at dusk, catching snowflakes and laughing.",
    references=[MiniMaxH3ImageReference.from_file("subject.png")],
    height=544, width=960, num_frames=124, num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(42),
)

Keyframes (workflow="fl2va") — a first and/or last frame:

from PIL import Image

out = pipe(                       # same repo, reloaded with workflow="fl2va"
    prompt="She lifts the teacup and takes a slow sip, then smiles; quiet kitchen room tone.",
    image=Image.open("first.png"), last_image=Image.open("last.png"),
    height=544, width=960, num_frames=124, num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(42),
)
video, audio, rate = out["videos"][0], out["audio"][0], out["sampling_rate"]

Keyframes and references in the same generation

workflow="combined" — this repo ships the blocks for it:

pipe = ModularPipeline.from_pretrained(REPO, workflow="combined", trust_remote_code=True)
pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)

out = pipe(
    prompt="She lifts the teacup and takes a slow sip, then smiles; warm kitchen light",
    references=[MiniMaxH3ImageReference.from_file("subject.png")],   # who / what
    image=Image.open("first.png"), last_image=Image.open("last.png"),  # where it starts and ends
    height=544, width=960, num_frames=124, num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(42),
)

MiniMax-H3 denoises one packed sequence, and that sequence can hold keyframe conditioning rows and reference conditioning rows at the same time. diffusers' shipped blocks cannot express it: their conditional steps dispatch either/or — select_block checks references first — so a request carrying both is accepted and the keyframes are silently dropped, with no error and no warning.

MiniMaxH3CombinedBlocks (in combined_blocks.py) is the stock blockset with three of its conditional steps replaced by ones that know a fourth shape. It reserves [text | keyframe conditions | reference blocks | target audio | target video], and because the references push the target timeline out, the keyframe anchors ride on the timeline their spans leave behind. The t2va, fl2va and ref2va workflows are untouched and take exactly the same path as before; the denoising loop needed no change at all, since the conditioning rows are simply the leading rows of the sequence.

Two things keep it honest: the layout reproduces both shipped diffusers builders bit for bit in their degenerate cases — no keyframes gives the ref2va layout, no references gives the fl2va one, position_ids compared in float64 — and the stock prepare-latents step asserts that the conditioning rows encoded equal the rows the layout reserved, so a wrong layout raises instead of quietly degrading.

Measured on this checkpoint — with this layout, driven through the same underlying steps, before it was packaged as the blockset above: a combined request (first frame + last frame + an image reference) agrees with the real ref2va partition at video-latent cosine 0.972, closer than any reference-only request reaches, because keyframes anchoring both ends leave the delta less to carry. Adding an audio reference on top moves the video only 0.965 while rewriting the soundtrack to 0.403 — each conditioning doing its own job.

Few-step generation — keep the turbo LoRA live

pipe.load_lora_weights("larryvrh/MiniMax-H3-Turbo-Lora", adapter_name="turbo",
                       load_into_transformer_ref=True)   # `load_into_transformer_ref` only for ref2va

Do not fuse it. A distill LoRA's deltas are 2–5e-4 of the weights they modify, and folding them into bf16 keeps only 0.67–0.79 — a third of turbo's AdaLN modulation is lost to rounding. Left live it costs ~7–15% wall time. For the same reason, an AoT-compiled transformer cannot be combined with a live adapter: the graph is captured from the base modules and runs straight past it.

Requires diffusers with MiniMaxH3LoraLoaderMixin (PR #14408) and trust_remote_code=True — the pruned layout ships a MiniMaxH3PrunedTransformer3DModel whose AdaLN is 8 wide, where the stock class expects 2688.

How it was made

1. 1.Take the difference between the twins. ref2va − fl2va, tensor by tensor. Both partitions are architecturally identical, so this is what makes one able to use references.
2. 2.Compress it. ethanfel's randomized-SVD extraction at rank 1024 (9.4 GB), applied at strength 1.0.
3. 3.Apply the parts that cannot be compressed exactly. 267 patches: 211 RMSNorm deltas, 56 biases, and adaln_t_table — the timestep coordinate table, which differs between the partitions and which the earlier rank-256 extractions omit.
4. 4.Fuse. fuse_lora, then unload the adapter, leaving an ordinary checkpoint. The delta is 0.02–1.9 of the weights it lands in, so it survives a bf16 fold at 1.00 (unlike a distill LoRA — see above).
5. 5.Keep the AdaLN affine map. adaln_basis / adaln_mean ship as buffers, so LoRAs trained on the released 2688-wide AdaLN still project onto this 8-wide one.

Verified by re-downloading this repo and generating: bit-identical (torch.equal on video and audio latents) to the local build it was made from.

Measured

Video-latent cosine against the true ref2va partition, generated on the same GPU (identical weights on a different GPU only agree to 0.959–0.987, so cross-machine anchors are not usable at this precision). Three seed-matched requests: photoreal image reference, stylized image reference, video reference.

Rank is what matters, and the bulk is load-bearing: the exact AdaLN half — the tempting 95 MiB shortcut — lands on the no-delta baseline, so the trunk attention and MLP deltas are carrying the result.

Keyframes still work. Against the pristine fl2va partition on the same first+last-frame request, this checkpoint reaches video cosine 0.930 and is visually indistinguishable. Audio is the weaker half (0.662, and it runs louder). Few-step, with turbo live: 64–179 s per 5 s clip against 174–478 s for 20 steps on the true partition, at the VRAM of a single partition.

Credit

Delta extraction ethanfel; approach established by Kijai; pruned repack Comfy-Org; turbo LoRA larryvrh; base model and license MiniMaxAI. Related community work on the same question: lihaoyun6 (exact-diff-only patch), smhfacct (AdaLN block swap), PulpCut (turbo merged into Ref2VA). The packed-sequence order a combined request uses follows ComfyUI's implementation of this model.