llm-jp-4-vl

|  |  |

LLM-jp-4-VL 9B is a vision-language model developed by LLM-jp.

It is built upon llm-jp/llm-jp-4-8b-thinking and SigLIP 2 So400m.

Compared with the earlier LLM-jp-4-VL 9B beta, the new model is trained on datasets that have no license or terms-of-use issues for commercial use. It is also enhanced to generate a reasoning trace before a final answer. Our evaluation shows improved performance over the beta version, particularly on text-only tasks and challenging chart-understanding benchmarks such as CharXiv and HakushoBench.

To support the continued development of LLM-jp, we would greatly appreciate it if you could share how you utilize LLM-jp outcomes via the survey form.

Model Architecture

The model architecture is inspired by InternVL3.0. It consists of a language model, a vision encoder, and a lightweight projector.

• LLM: llm-jp/llm-jp-4-8b-thinking (8.6B)
• Vision Encoder: google/siglip2-so400m-patch16-512 (0.4B)
• Projector: 2-layer MLP

Chat Template

We adopt a customized chat template based on an extension of the OpenAI Harmony format.

In this template, a sequence of image tokens is wrapped with special tokens and, and inserted into the user message. Because this is a reasoning model, the assistant may first produce an analysis message (reasoning content) and then a final message (the answer). An example input/output is shown below:

Training Data

The training data is a mixture of four large-scale datasets (≈29.3M samples in total):

• Jagle (9.2M): a Japanese multimodal dataset built by LLM-jp.
• RefinedVision (12.0M): a refined version of FineVision built by LLM-jp. Of the 185 subsets in FineVision, we removed 61 that include images with licensing concerns and poor quality. Of the remaining 124 subsets, we regenerated QA pairs for 58 subsets whose original QA pairs were of low quality or had license/terms-of-use concerns (e.g., QA generated by GPT-4o or other proprietary models): keeping the original images, we regenerated the QA from scratch with Qwen3.5 397B-A17B using per-category generation prompts. The other 66 subsets were used as-is. See the RefinedVision dataset page for details.
• Nemotron-Image-Training-v3 subset (5.0M): an English multimodal dataset built by NVIDIA, from which we removed subsets with license/terms-of-use issues. The exact list of subsets we used is given in nemotron_subsets.md.
• llm-jp-4-thinking-sft-data (3.2M): a text-only Japanese reasoning SFT dataset built by LLM-jp.

Training Details

The language model and vision encoder are initialized from their pretrained weights, while the projector is initialized randomly. We then train the model in a single stage for 120,000 steps, with all three components trainable from the start (no projector-only warmup or frozen-backbone stage).

We use different maximum learning rates for each component: 2e-5 for both the language model and the vision encoder, and 1e-4 for the projector.

The learning rate follows the Warmup–Stable–Decay (WSD) schedule, with a warmup phase of 2,000 steps, followed by stable training, and linear decay starting after 80% of the total training steps.

Evaluation

We evaluate our model and all baselines with simple-evals-mm, our evaluation framework for VLMs, across a broad suite of English and Japanese multimodal and text-only benchmarks. For several Japanese benchmarks, we use the refined versions from the JAMMEval collection.

Evaluation settings

LLM-jp-4-VL 9B is decoded greedily (temperature=0.0) with max_new_tokens=32k and reasoning effort medium. Each baseline uses its own recommended sampling temperature; for the reasoning models Qwen3.5-9B and Gemma-4-12B, we increase max_new_tokens to 64k to avoid truncating long chains of thought. Scores are averaged over 3 runs.

Per-benchmark score vs training step

As training progresses, the model improves on most multimodal benchmarks. On the text-only benchmarks, it preserves the reasoning ability of the base LLM (llm-jp-4-8b-thinking), so adding vision capability does not degrade its language reasoning.

Per-benchmark results