h2ovl-mississippi

[📜 H2OVL-Mississippi Paper] [🤗 HF Demo] [🚀 Quick Start]

The H2OVL-Mississippi-800M is a compact yet powerful vision-language model from H2O.ai, featuring 0.8 billion parameters. Despite its small size, it delivers state-of-the-art performance in text recognition, excelling in the Text Recognition segment of OCRBench and outperforming much larger models in this domain. Built upon the robust architecture of our H2O-Danube language models, the Mississippi-800M extends their capabilities by seamlessly integrating vision and language tasks.

Key Features:

• 0.8 Billion Parameters: Balance between performance and efficiency, making it suitable for OCR and document processing.
• Trained on 19 million image-text pairs, with a focus on OCR, document comprehension, and chart, figure, and table interpretation, the model is optimized for superior OCR performance.

Benchmarks

Performance Comparison of Similar Sized Models Across Multiple Benchmarks - OpenVLM Leaderboard

Quick Start

Install dependencies:

pip install transformers torch torchvision einops timm peft sentencepiece flash_attn

Sample demo:

import torch
from transformers import AutoConfig, AutoModel, AutoTokenizer

# Set up the model and tokenizer
model_path = 'h2oai/h2ovl-mississippi-800m'
config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
config.llm_config._attn_implementation = 'flash_attention_2'
model = AutoModel.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    config=config,
    low_cpu_mem_usage=True,
    trust_remote_code=True).eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, use_fast=False)
generation_config = dict(max_new_tokens=2048, do_sample=True)

# pure-text conversation
question = 'Hello, how are you?'
response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)
print(f'User: {question}\nAssistant: {response}')

# Example for single image
image_file = './examples/image.jpg'
question = '\nRead the text in the image.'
response, history = model.chat(tokenizer, image_file, question, generation_config, history=None, return_history=True)
print(f'User: {question}\nAssistant: {response}')

Prompt Engineering for JSON Extraction

Overview

This guide demonstrates how to create prompts for extracting information and converting it into structured JSON outputs. It starts with basic examples and progresses to more complex JSON structures, including handling data from images of tables and charts. The objective is to help users design effective prompts that can be used in various applications, such as natural language processing, chatbots, or data extraction from visual inputs.

Table of Contents

1. Getting Started
2. Extracting Simple Information
3. Extracting Nested Information
4. Extracting Lists and Arrays
5. Extracting Tables
6. Extracting Charts
7. Best Practices

Getting Started

To get started with JSON extraction from images, it's essential to have a clear understanding of the visual content you want to extract and the structure of the desired JSON