Welcome Beam: a 501B open model you can own, customize, and scale

2026-10-19 · Hugging Face

Welcome Beam: a 501B open model you can own, customize, and scale

Reflection, a new US lab, has released Beam, its first open-source model, now available on Hugging Face. Beam is a mixture-of-experts (MoE) model with 501B parameters designed for teams wanting to run a frontier-scale model on their own infrastructure. It is specifically focused on coding and agentic use cases and ships with a dedicated open-source Rust-based harness for optimal performance.

Key Features

  • Architecture: 501B-parameter MoE LLM with 23B active parameters per token. It has 52 layers, 256 experts per MoE layer, and routes 8 per token.
  • Context & I/O: Text-only input and output with a massive 1M token context window.
  • Checkpoints: Available in BF16, FP8 (ideal for Hopper and Ampere), and NVFP4 (for NVIDIA Blackwell, keeping some parameters in BF16 for accuracy).
  • Reasoning & Tools: Supports cross-turn reasoning context (`preserve_thinking`) and tool calling with a dedicated grammar. Features five reasoning values: "low", "medium" (default), "high", "xhigh", and "max".
  • Libraries: Day-0 support in `transformers`, `SGLang`, and `vLLM`. Full-parameter RL is supported with Miles.
  • License: Apache 2.0.

Build with Beam

Reflection is offering $10,000 in Hub credits for the best projects using Beam. Developers can build demos, quantize the model, generate synthetic datasets, create RL environments, or benchmark the model. Sharing the project on the model's repo page enters participants for a chance to win the credits.

Architecture Details

Beam is a sparse MoE model, computing each token with a small fraction of the network to run at the speed of a 23B model while retaining 501B capacity.

Attention

The decoder has 52 layers with a hidden size of 5,120. It uses 43 sliding-window layers (attending to the previous 512 tokens) and 9 global layers.

  • Sliding-window layers use RoPE for relative position.
  • Global layers use no positional encoding. Instead, they use scalable-softmax query scaling: `(1 + (pos + 1) / 2048) ** 0.1`, acting as a temperature to maintain softmax precision as context lengthens.
  • Attention uses grouped-query attention (48 query heads, 8 KV heads) with RMSNorm. A learned sigmoid gate scales channels before the output projection.

Mixture of Experts

The first 3 layers use a dense SwiGLU MLP. The remaining 49 use an MoE block of 256 experts (each with ~39M parameters), holding 98.4% of the model's parameters.

  • Routing: Follows DeepSeek-V3's group-limited scheme. Scores 256 experts via sigmoid, splits into 8 groups of 32, keeps the top 4 groups, and picks the top 8 experts. No shared expert is used.
  • Memory: The BF16 checkpoint is about 1 TB. Deployment requires memory for a 501B model but runs at 23B speeds.

Residual Stream

Beam's residual connections are highly distinctive. Instead of adding sublayer outputs with a weight of 1, Beam treats the input embedding as step 1. At step *k*, it adds the sublayer output with weight `1/sqrt(k)` and keeps the existing stream with weight `sqrt((k-1)/k)`. The squares of these weights sum to 1, keeping the stream at a constant scale throughout all 104 additions.

Every sublayer is normalized before and after. Final logits pass through a tanh softcap at ±30 in FP32. Specific operations—router, SwiGLU product, norms, RoPE, query scaling, and softcap—are computed in FP32.

Source