Flora-Captioner

Description:

Flora-captioner-27B generates one action instruction for a supplied target interval in a robot or human manipulation video. It uses neighboring video context to disambiguate the intended local operation. In the Flora data engine, target intervals come from temporal segmentation and stitching; the captioner supplies the text used for balancing the pretraining data distribution. Flora was developed by NVIDIA as the data engine used for training GR00T-2, and is released to help researchers scale up custom robot datasets.

This model is for research and development only.

License/Terms of Use:

Use of this model is governed by the NVIDIA OneWay Noncommercial License (22 March 2022). This model has been post-trained on Qwen/Qwen3.5-27B, revision fc05daec18b0a78c049392ed2e771dde82bdf654. The Qwen base model is licensed under Apache License 2.0. You are responsible for ensuring that your use of NVIDIA provided models complies with all applicable laws.

Deployment Geography:

Global

Use Case:

Researchers, Academics, Open-Source Community: Annotating robot datasets with language; and research into emerging applications such as physical auto-research and human-to-robot transfer. Developers: Prototyping and evaluating data curation, filtering, and monitoring pipelines for robotics foundation model research.

Release Date:

• Hugging Face: 10/15/2026 via Flora-Captioner.

Intended Use

Flora is intended for non-commercial robotics research and development only, including annotating robot and human demonstration videos. Flora is not intended or supported for commercial or production use. Flora outputs are annotations. It is not intended to directly control robots or to serve as the sole safety monitor for a physical system.

References(s):

• GR00T Website: NVIDIA Isaac GR00T
• GR00T N1 White Paper: https://arxiv.org/abs/2503.14734
• Base Model: Qwen3.5-27B
• Benchmarks: WGO-bench

Model Architecture:

Architecture Type: Transformer

Network Architecture: Other: Qwen3.5 vision-language model (natively multimodal vision encoder with a decoder-only Large Language Model)

This model was developed based on Qwen3.5-27B.

Number of model parameters:

• Flora-captioner-27B: 2.7 x 10^10

Flora-captioner-27B is trained in two stages. Supervised fine-tuning updates all weights of the language model and visual merger, without Low-Rank Adaptation (LoRA), while keeping the visual patch embedding and visual transformer blocks frozen. The supervised model is then further optimized with reinforcement learning using Group Relative Policy Optimization (GRPO) with a reference-grounded caption-quality reward, with the vision encoder frozen. Sampled video frames are encoded by the vision encoder. Visual tokens from explicitly labeled BEFORE, TARGET, and AFTER blocks are combined with the annotation prompt and passed to the language model decoder. The decoder autoregressively generates one instruction for TARGET; context-only operations are not part of the requested output.

Input(s):

Input Type(s):

• Video: Ordered Red, Green, Blue (RGB) frames for a TARGET interval, with optional BEFORE and AFTER context, from one camera or three synchronized cameras
• Text: Prompt identifying the temporal blocks and camera roles and requesting a target-only action instruction; an episode task instruction may also be supplied

Input Format(s):

• Video: Decoded RGB frames from a video file (e.g., .mp4), with their source-frame timing metadata
• Text: String

Input Parameters:

• Video: Four-Dimensional (4D): a sequence of RGB images (frames × height × width × 3), organized into labeled temporal blocks
• Text: One-Dimensional (1D)

Other Properties Related to Input: The native reference inference profile is:

8 Hz is the requested sampling rate, not the recording's source frame rate. Duration caps and source-frame availability can change the effective sampling rate. Preserve each block's source clock and selected source-frame indices in the processor's video metadata; disable additional frame sampling with do_sample_frames=False. Short source spans require the reference frame-planning policy rather than invented observations. Missing context can be omitted; source-limited targets must be handled explicitly.

Target intervals and camera layout

The caller supplies a half-open target interval [start_s, end_s) in the source video's clock. BEFORE and AFTER are the nearest available non-overlapping event intervals on either side, not a fixed number of seconds of padding. After temporal stitching, refresh these context intervals from the reconstructed event timeline. Sample all three blocks independently and ret