Welcome Muse Spark Open: open-weight frontier multimodal mixture-of-experts model from Meta
2026-10-16 · Hugging Face
Welcome Muse Spark Open: Open-Weight Frontier Multimodal Mixture-of-Experts Model from Meta
Meta has introduced Muse Spark Open, an open-weight multimodal mixture-of-experts (MoE) model. Designed specifically for agentic coding, it processes text, images, audio, and video inputs while featuring a massive context window of up to one million tokens. As the full-size model in the Muse series, it scales up significantly from the smaller Muse Glimmer released earlier.
Overall Capabilities and Architecture
Muse Spark Open is a coding-optimized model trained on real developer workflows, including multi-file refactors and long debugging sessions. It operates as an agentic model capable of planning, acting, observing, and self-correcting over extended tasks, such as browsing the web with images.
Key Features:
- Model size: ~816B total parameters; ~42B active per token.
- Expert routing: 256 experts per MoE layer; 8 selected per token.
- Inputs: Text, images, audio, and video.
- Output: Text.
- Context window: Up to 1M tokens.
- Checkpoint formats: BF16 and NVFP4.
The checkpoint repositories are `Muse-Spark-Open-816B-A42B` and `Muse-Spark-Open-816B-A42B-NVFP4`. Muse Spark Open shares the same tokenizer and vocabulary as Muse Glimmer, with chat template additions for new modalities. Its input embedding and output layer are separate matrices, each containing about 1.66B parameters.
Mixture of Experts Architecture
The language model consists of 62 layers with a hidden dimension of 8192. Each layer utilizes 128 query heads and 16 key-value heads (grouped-query attention), with each head having 64 dimensions. The model employs QK-norm and uses a learned residual gate to combine block inputs and outputs rather than a plain sum.
Replacing the dense feed-forward network found in Muse Glimmer, Muse Spark Open uses an MoE block with 256 experts per layer. A router selects 8 experts per token. These experts hold approximately 97% of the model's total parameters.
Because only 8 experts run per token, the model computes at the speed of a 42B-parameter model, although deployment requires the memory footprint of an 816B-parameter model. The BF16 checkpoint is roughly 1.5 TB. The NVFP4 checkpoint uses 4-bit precision tailored for NVIDIA Blackwell GPUs with negligible accuracy loss. In `transformers`, the BF16 checkpoint runs across 4 nodes of 8 H200 GPUs (TP=32), while NVFP4 runs on a single B200 node. Deployment is supported via llama.cpp, vLLM, and SGLang.
Perception and Audio Encoders
The perception encoder converts images and video frames into tokens. It has 1.8B parameters, 50 layers, and splits images into 14×14 patches to produce 4096-dimensional features. It uses full attention every fourth layer and the last layer, with sparse attention elsewhere.
The audio encoder uses an audio tokenizer, marking positions with `<|audio|>` tokens. It emits one token every 80 ms (12.5 tokens per second). The encoder is a convolutional pyramid with stages repeating for patch sizes of [16, 6, 5, 4], ending with an RMSNorm, linear projection, and RMSNorm block.
Long Context and Inference
The context window supports 1M tokens at inference (`max_position_embeddings` = 1,048,576), unifying text, image, video, and audio tokens. Grouped-query attention minimizes the KV cache, requiring about 250 KB per token and 266 GB for a full 1M-token context in BF16.
For inference support, the `transformers` implementation provides `MuseSparkForConditionalGeneration` for text generation from multimodal inputs.