Pika-H3-Flex-FP4Attn-QAD

The long-clip speed variant of Pika H3 Flex NVFP4 QAD for NVIDIA Blackwell GPUs. Both the linear layers and attention run in 4-bit: queries and keys go through FlashAttention-4's NVFP4 path, while attention probabilities and values stay in BF16. Both Flex adapters were trained with quantization-aware distillation (QAD) under exactly these numerics.

Pika H3 Flex · FastVideo · Inference guide · Training recipe · BF16-attention variant

Variants

Attention dominates the cost of long, high-resolution clips, so the gain from FP4 attention grows with clip length. On one GB200, a 15-second 768p clip is about 12% faster end to end than with the BF16-attention variant; short clips run at about the same speed. Output stays close to the BF16-attention variant, which remains the higher-fidelity choice.

Tasks

Quickstart

Install FastVideo with FlashAttention-4. Enable FP4 attention with the same settings the model was trained with:

FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN FASTVIDEO_FA4=1 \
python examples/inference/basic/basic_pika_h3_flex.py \
  --model-path FastVideo/Pika-H3-Flex-NVFP4-FP4Attn-QAD --nvfp4-dit h3_dit \
  --nvfp4-attention smooth_k+hadamard --nvfp4-attention-bf16-blocks 45,49 \
  --nvfp4-decoder /vae_decoder_nvfp4/decoder-nvfp4.safetensors \
  --task fl2va --image first.png --last-image last.png \
  --prompt "$(cat prompt.txt)" --resolution 768p --duration 15

Instead of the flags, you can set FASTVIDEO_NVFP4_FA4=1, FASTVIDEO_NVFP4_FA4_QK_TRANSFORM=smooth_k+hadamard and FASTVIDEO_NVFP4_FA4_BF16_BLOCKS=45,49. For Ref2VA, use --task ref2va with --reference image:..., video:... or audio:... inputs.

How the FP4 attention works

FlashAttention-4's NVFP4 path stores queries and keys as 4-bit values with one scale per 16 channels. Two transformations that leave the attention output mathematically unchanged make them much easier to quantize:

• Key smoothing. The mean key of each attention head is subtracted before quantization. Softmax is unaffected by a shift that every key shares.
• Hadamard rotation. Queries and keys are rotated by the same orthonormal Hadamard matrix. Their dot products are preserved, and outlier channels are spread across the quantization blocks.

Two of the fifty transformer blocks keep BF16 attention. Starting from the BF16-attention QAD model, the model was then trained under these exact numerics. The training recipe documents the full process.

Contents

The transformers are packed NVFP4 checkpoints and load directly in FastVideo, with no conversion step.

Notes

• NVFP4 kernels and FP4 attention require an NVIDIA Blackwell GPU. On other GPUs, use Pika H3 Flex in BF16.
• Run this model with the FP4-attention settings above. It still works with BF16 attention, but it was tuned for FP4 attention; for BF16 attention, use the BF16-attention variant.
• With reference inputs, the opening frame can follow a first-frame reference slightly less closely than in the BF16-attention variant.
• Quantized output can differ from BF16 output.

License and acknowledgements

This model is a derivative of MiniMax H3 and Pika H3 Flex. It is distributed under the MiniMax H3 Community License Agreement; see NOTICE, THIRD_PARTY_NOTICES.md and licenses/. The Qwen3-VL text encoder is licensed under Apache 2.0.

We thank Pika for open-sourcing Pika H3 Flex, and MiniMax for releasing MiniMax H3.