fibo-scene-analyzer

Supported tasks

AR is the width divided by the height, rounded to two decimals.

With a scene description (scene_contexts in generate()), expand_object describes the object so that it fits the scene, with relationships to the objects of the scene. The scene must not contain the object itself.

SCENE_JSON is compacted JSON with coordinates and colors as tokens. prompt_json() in scene_analyzer.py writes a SceneDescription in this form, also after post_process().

For the grounding tasks, a named target that is not in the image gives an empty list. Without a target, detect and mask return every object; clothing, footwear and accessories are separate targets named "ITEM on PERSON".

Scene detail levels

expand, refine and edit accept general, short, long and full. expand_object accepts short, long, full and full_mask.

How to get started

Download the code files of this repository, then:

from PIL import Image
from scene_analyzer import SceneAnalyzerModel, SceneAnalyzerTask, DetailLevel

model = SceneAnalyzerModel("briaai/fibo-scene-analyzer")  # Transformers, device_map="auto"
image = Image.open("photo.jpg").convert("RGB")

[description] = model.generate([image], SceneAnalyzerTask.CAPTION, detail_level=DetailLevel.FULL, num_samples=3)
description.post_process()  # coordinate and color tokens to numbers
print(description.general.description)
for object in description.objects or []:
    print(object.kind, object.name, object.bounding_boxes)

[people] = model.generate([image], SceneAnalyzerTask.DETECT, text_inputs=["person"])
[text] = model.generate([image], SceneAnalyzerTask.OCR)

To render an answer as an HTML page with the geometry drawn on the image:

from caption_visualizer import CaptionVisualizer

CaptionVisualizer().save("description.html", image, description)

Harnesses

To make it easier to work with the model and get the best results, we provide harness classes based on the Pydantic scene description format. The three harnesses have the same SceneAnalyzerModel.generate() interface and the same task logic; only the engine differs.

What each harness does:

• It builds the prompt in the trained format. A prompt that the model was not trained on, such as a detail level that a task does not accept, raises ValueError.
• It samples num_samples answers and returns the best valid one. A scene description with the most objects wins; for grounding tasks, the most confident answer wins. An input without a valid answer is sampled again (retries), and None is returned if all attempts fail.
• With num_iterations, it runs revision rounds: the model sees its answer and adds the objects that it missed. This works for `` and the whole-image grounding tasks.

vLLM

from scene_analyzer_vllm import SceneAnalyzerModel

model = SceneAnalyzerModel("briaai/fibo-scene-analyzer", gpu_memory_utilization=0.9)
descriptions = model.generate(images, SceneAn