LBS

Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding

💻 Project · 🤗 Collection

Model Summary

LBS-27B is an agentic vision-language model based on Qwen3.8-27b. It learns to select and inspect image regions through GUI grounding, then transfers this observation policy to general-domain visual question answering and high-resolution perception.

Training uses only synthetic GUI grounding data, with no general-domain data or VQA questions: approximately 10K cold-start SFT examples, followed by 80 GRPO steps with a batch size of 64 and 8 rollouts per prompt.

• Adaptive observation: zoom into relevant regions, refine previous crops, and revisit the original image when needed.
• Compositional zoom-in: request up to two independent crops per turn, with a total budget of three crop calls per episode.
• Transfer beyond screens: apply the learned observation policy to visual QA without general-domain VQA training.

Results

All LBS scores use agentic zoom-in over complete benchmark runs. PerceptionBench uses a self-hosted Deepseekv4-flash-0731 judge, with zero failed or skipped judgments; repeated judgments may vary. Chart references use different inference setups and GPT-oss-120B for PerceptionBench, so comparisons are not controlled. See comparison notes.

More result visualizations are available in the LBS project; we have not performed any additional fine-tuning on general-domain data and welcome further RL experiments in specific subdomains.

Usage

Serve the released checkpoint in a compatible vLLM environment. Adjust the four-device example below for your hardware:

git clone --branch LBS --single-branch https://github.com/inclusionAI/UI-Venus.git
cd UI-Venus/evaluation
bash scripts/serve_vllm.sh start inclusionAI/LBS-27B \
  --name lbs --tp 4 --dp 1 --daemon

LBS emits plain-text crop_image actions. A client must execute the crops and return the images and coordinate mappings for multi-turn inference. Final GUI coordinates use [x.xx,y.yy], normalized to [0,1000] on the original image. Grounding predicts a location; operating a live application requires a separate executor.

The evaluation guide provides the client and benchmark setup. Configure endpoints and credentials in an ignored .env.run copied from .env.example, using the actual model ID from /v1/models. Wrappers default to reasoning_effort=xhigh and thinking_budget=4096 through chat_template_kwargs; direct Python entries can default to high.

License

The model weights and project software are released under Apache 2.0. Third-party datasets and screenshots retain their original licenses and terms.