LFM3-ONNX

src="https://huggingface.co/LiquidAI/LFM3-1.7B/resolve/main/assets/liquid-ai-banner.webp" alt="Liquid AI" style="width: 100%; max-width: 100%; height: auto; display: block; margin: 0;" />

LFM3-1.7B-ONNX

ONNX version of LFM3-1.7B for cross-platform inference with ONNX Runtime.

Find architecture and usage details in the original model card.

Files

fp16, q4f16 and q8f16 run with fp16 activations, the others with fp32. q4 and q4f16 keep the embedding, the LM head and 27 of the 153 layer projections in int8 and the rest in int4, while q4f32 quantizes all 153 projections to int4 and keeps the embedding and LM head in fp32.

Use with ONNX Runtime

The graphs run on ONNX Runtime. Each step takes input_ids, attention_mask, position_ids and logits_to_keep, plus the convolution and attention caches (past_conv.N, past_key_values.N.key and .value). It returns logits and the updated caches (present_conv.N, present.N.key and .value). The fp16, q4f16 and q8f16 graphs keep their caches in fp16.

On GPU, use fp16, q4f16 or q8f16. The CUDA execution provider has no fp32 GroupQueryAttention kernel, so the other graphs run attention on the CPU.

In the browser, use import * as ort from 'onnxruntime-web/webgpu': the default onnxruntime-web import cannot run these graphs. On WebGPU, use model_q4f16, and prefill in chunks of at most 256 tokens.

Recommended sampling: temperature=0.1, top_k=50, repetition_penalty=1.1. For math and science, use temperature=1.0, top_p=0.95, top_k=20 instead. More options in the ONNX guide.