idefics2

[!WARNING] Idefics2 will NOT work with Transformers version between 4.41.0 and 4.43.3 included. See the issue https://github.com/huggingface/transformers/issues/32271 and the fix https://github.com/huggingface/transformers/pull/32275

[!IMPORTANT] As of April 18th, 2024, Idefics2 is part of the 4.40.0 Transformers pypi release. Please upgrade your Transformers version (pip install transformers --upgrade).

Idefics2

Idefics2 is an open multimodal model that accepts arbitrary sequences of image and text inputs and produces text outputs. The model can answer questions about images, describe visual content, create stories grounded on multiple images, or simply behave as a pure language model without visual inputs. It improves upon Idefics1, significantly enhancing capabilities around OCR, document understanding and visual reasoning.

We release under the Apache 2.0 license 2 checkpoints:

• idefics2-8b-base: the base model
• idefics2-8b: the base model fine-tuned on a mixture of supervised and instruction datasets (text-only and multimodal datasets)
• idefics2-8b-chatty: idefics2-8b further fine-tuned on long conversation

Model Summary

• Developed by: Hugging Face
• Model type: Multi-modal model (image+text)
• Language(s) (NLP): en
• License: Apache 2.0
• Parent Models: google/siglip-so400m-patch14-384 and mistralai/Mistral-7B-v0.1
• Resources for more information:
  • Description of OBELICS: OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
  • Paper: What matters when building vision-language models?

Uses

idefics2-8b-base and idefics2-8b can be used to perform inference on multimodal (image + text) tasks in which the input is composed of a text query along with one (or multiple) image(s). Text and images can be arbitrarily interleaved. That includes image captioning, visual question answering, etc. These model does not support image generation.

For optimal results, we recommend fine-tuning idefics2-8b on one's specific use-case and data. In fact, the instruction-fine-tuned model (idefics2-8b) is significantly better at following instructions from users and thus should be preferred when using the models out-of-the-box or as a starting point for fine-tuning.

idefics2-8b usually generates very short answers. For long generations, use idefics2-8b-chatty, which was further fine-tuned on long conversations.

As a starting point, we provide fine-tuning codes that can be adapted for one's particular scenario:

• With the TRL library: Script

Technical summary

Idefics2 exhibits strong performance for a model of its size (8B parameters) when compared to other open multimodal models and is often competitive with closed-source systems. As such, it serves as a strong foundation for various use-case specific fine-tunings.

Idefics2 introduces several carefully abalated improvements over Idefics1:

• We manipulate images in their native resolutions (up to 980 x 980) and native aspect ratios by following the NaViT strategy. That circumvent the need to resize images to fixed-size squares as it has been historically been done in the computer vision community. Additionally, we follow the strategy from SPHINX and (optionally) allow sub-image splitting and passing **