NeoMME: an efficient Multimodal-native and Multilingual Encoder

2026-09-04 · Hugging Face

NeoMME: an efficient Multimodal-native and Multilingual Encoder

Overview

Hcompany introduces NeoMME, a family of 260M and 800M multilingual multimodal encoders. Unlike many generative visual language models, NeoMME does not utilize a separate pretrained vision tower or a causal language model. Instead, a single bidirectional Transformer processes both text tokens and raw image patches, and the entire model is trained from scratch using a masked discrete-diffusion objective.

Key Advantages and Performance

The team fine-tuned NeoMME for visual document retrieval using ColPali's page-image approach. NeoMME-Retriever returns dense and late-interaction embeddings in a single forward pass.

  • Benchmark Performance: Both model sizes lie on the ViDoRe v3 Pareto frontier for nDCG@10 and model size.
  • Processing Speed: At a matched 2048×2048 image input size on an NVIDIA L40S GPU, the 260M model encodes about 51 pages per second, roughly twice the throughput of ColModernVBERT.
  • Storage Optimization: Hierarchical token pooling and asymmetric quantization reduce late-interaction index storage from approximately 1.5 MB to 6 kB per page (255× smaller) while retaining more than 95% of the baseline nDCG@10.

NeoMME is available in Hugging Face Transformers, with all model checkpoints released under the Apache 2.0 license.

Why Another Multimodal Encoder?

Many recent visual document retrievers are adapted from pretrained generative visual language models. These architectures typically involve a separately pretrained vision encoder producing visual features, which a projector maps into the language model's input space, followed by a causal decoder. However, tasks like retrieval, classification, and token labeling do not require autoregressive text generation, making the causal decoder and its parameter and compute overhead unnecessary.

While ModernVBERT applied an efficient bidirectional architecture, it retained a separate pretrained SigLIP2 vision tower. NeoMME pushes this further by designing a multimodal encoder without the overhead of a VLM. It is not based on any existing pretrained vision tower, text encoder, or decoder. By processing image patches and text tokens in one bidirectional Transformer using the same computational path, NeoMME simplifies pretraining, fine-tuning, parallelization, and serving across both modalities.

NeoMME Encoder Backbone

Both the 260M and 800M variants share the same architecture:

  • Native multimodal inputs: Text inputs use factorized token embeddings, while images are divided into a grid of non-overlapping 32×32 patches and projected with a small MLP. Both enter the same Transformer encoder.
  • Dynamic image resolution: Images keep their original aspect ratio and size, allowing the model to allocate more tokens to high-resolution, information-dense pages.
  • Long bidirectional context: Both models feature a context length of 16,384 tokens (enough for up to two standard 4K UHD images). Most layers use symmetric sliding-window attention, while every sixth layer and the final layer use global attention.
  • Modern encoder stack: NeoMME incorporates recent improvements such as grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs.
  • Multilingual text: A BPE tokenizer with a 131k-token vocabulary was trained from scratch on multilingual text, code, mathematics, and machine-produced image transcripts.

Learning from Images through Masked Text

NeoMME is pretrained from scratch as a discrete masked-diffusion text denoiser. For text-only examples, the corruption rate is sampled uniformly between 0 and 1; for multimodal examples, it ranges from 0.3 to 1. Image patches remain visible while the model reconstructs masked text.

Higher text corruption removes language-only shortcuts, forcing NeoMME to rely on visible image evidence. Pretraining mixes multilingual text, code, mathematics, natural images, and document images. Each model processes about 524 billion packed input tokens, including 290 billion from text-only examples. To improve data efficiency given the relatively small text budget compared to ModernBERT, the NorMuon optimizer was chosen.

NeoMME-Retriever

To evaluate the backbone downstream, the team fine-tuned NeoMME for visual document retrieval using ColPali's page-image methodology. NeoMME-Retriever ranks document page screenshots directly, bypassing all OCR preprocessing steps required to extract text from PDFs. Treating pages as images preserves layout, charts, tables, font types, and other visual clues that even a perfect OCR model might fail to capture.

Source