Introducing LFM3: Every Modality for Specialized Intelligence

Tri-hybrid layer mix. LFM3 combines three operators: gated short convolutions, sliding-window attention (SWA), and global attention. Of its 24 layers, 13 are convolutions, 5 are SWA with a 256-token window, and 6 are global attention. The convolutions also use a kernel size of 7, up from 3 in LFM2.

Head-wise gated attention. A sigmoid gate scales each head’s output, so a head can switch off when it has nothing to add. This can help suppress activation outliers that hurt quantization and improve training stability.

Partial RoPE. Global attention layers rotate only the highest-frequency half of each head and leave the low-frequency half unrotated, so those dimensions match on content regardless of distance. SWA layers keep full RoPE.

Sandwich norm. Each block normalizes both its input and its output, which helps keep activations well-scaled.

LFM3 uses the same 128K vocabulary as LFM2.5-8B-A1B, built to tokenize non-Latin scripts more efficiently.

Training. LFM3-1.7B-Base was pre-trained on 22T tokens. Compared with LFM2, the mix adds more scientific papers, repository-level code, a wider range of knowledge sources, and considerably more math. The final 3.5T tokens are a mid-training stage on curated, higher-quality data, with a larger share of agentic traces and diverse tool calls to prepare the model for agentic post-training. This stage also extends the context window to 256K tokens.

The first LFM3 release includes six models, all branched from the same LFM3-1.7B-Base backbone and all around 2B parameters. Four of them cover different input modalities (text, vision, audio, and omni), and the last two are decision models specialized from the vision and omni models. Instead of running one large general-purpose model, you can pick the one that fits your use case, customize it, and run it on-device.