keural-vision-encoder-mid

A 183.5M-parameter image encoder built from scratch — adaptive tokenization, hierarchical concept tokens, and continuous 2D positional encoding. Not a CLIP/ViT fine-tune.

Developed by MKD Co., Ltd. · Code: github.com/mkd-hika/Keural-Vision-Encoder-Mid-level

At a Glance

Measured at 384×384 with token_budget=512, matching training. Accuracy trails same-size peers — see Limitations and Evaluation for why, and for the higher-k retrieval results where the gap closes.

Downstream Results — Vision-Language Model

The encoder is frozen and used as-is inside a bilingual (English + Korean) vision-language model: mkd-hika/keural-mid-vlm-bilingual. Only a projector (8.9M) and LoRA adapters (5.0M) are trained on top; the decoder is Qwen2.5-7B-Instruct.

A vision-language model can score respectably while barely using the image, because many questions are answerable from language priors alone. Each benchmark below is therefore run twice — once normally, once with the visual tokens removed and everything else held constant. The difference is what this encoder contributes, and it is the only figure the language model cannot produce on its own.

The POPE yes-ratio is 0.492 against a gold ratio of 0.500, so the model genuinely discriminates on object-existence questions rather than defaulting to one answer — accuracy alone is not interpretable without that check. The blind VQAv2 score of 39.83% lands where language-prior performance is expected to, which is corroborating evidence that the measurement is sound.

Notably, the encoder reaches these numbers while scoring 37.2% zero-shot on ImageNet. Zero-shot accuracy measures image–text alignment; the 75.0% CIFAR-100 linear probe is the better indicator of representation quality for downstream use, because a trained projector relearns the alignment from scratch.

Model Description

Keural Mid is a mid-scale vision encoder that maps an image to a single 768-dim embedding aligned to a text embedding space, plus a variable-length sequence of semantically-typed tokens. It is trained from random initialization (no CLIP/ViT/DINO backbone) with a combination of sigmoid contrastive learning, hierarchical concept alignment, saliency regularization, and dual-teacher knowledge distillation.

It introduces three mechanisms not found together in existing encoders (CLIP, SigLIP, DINOv2):

Intended Uses & Limitations

Intended uses

• Image feature extraction — a global pooled embedding for retrieval, clustering, or as a frozen backbone for downstream heads (e.g. linear probing, detection, captioning front-ends).
• Research into adaptive tokenization, hierarchical token typing, and resolution-elastic position encoding.

Limitations

• Accuracy trails same-size peers. Zero-shot ImageNet Top-1 is 37.2%, versus 68.6% for CLIP ViT-B/16 (~86M params) and 76.7% for SigLIP ViT-B/16 (~93M params) — both smaller than this model. This is attributed to a training-budget gap (18K steps on ~20M pairs, vs. peers trained on 12M-400M+ pairs over far more steps) rather than an architecture ceiling: training loss was still declining at the final step. See Evaluation.
• Saliency mechanism is real but token placement is still settling. The learned saliency map produces stable, object-tracking contours on images with a clear dominant subject. However, the discrete tokens it samples continued shifting meaningfully until very late in training (only ~50% overlap with the final checkpoint's token positions as of step 13-14K, reaching ~73% by step 16K) — exact token placement had not fully converged even at the final checkpoint.
• Domain/bias. Trained on web image–text pairs; it inherits the coverage and biases of that distribution and is English-text aligned.
• Not a safety-filtered model. No content moderation or de-biasing has been applied.

How to Use

import torch
from PIL import Image
from torchvision import transforms
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "mkd-hika/keural-vision-encoder-mid",
    trust_remote_code=True,
).eval()

transform = transforms.