Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem
2026-09-21 · Hugging Face
Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem
Deleting whole transformer blocks, also known as depth pruning, is one of the cheapest and bluntest ways to make a large language model faster. It offers predictable inference speedups and memory savings while stacking cleanly with quantization and low-rank compression. The challenge lies in deciding which blocks to cut. Removing the wrong blocks causes model collapse, and the impact of removing any single block depends on the others removed alongside it. This makes block selection a combinatorial problem rather than a simple ranking problem.
Why picking blocks is a many-body problem
Most existing block-removal methods score each block independently using magnitude, sensitivity, or "block influence" heuristics, then remove the least important ones. In physics, these are akin to mean-field methods, treating each block's contribution as independent. Another shortcut is removing only a single consecutive run of blocks, which keeps the problem small but discards most of the search space.
However, blocks are not independent, much like spins in a real magnet. Whether removing block 20 hurts the model depends on whether block 19 or 24 was also removed—an interaction or coupling between decisions. As models grow deeper and more heterogeneous, ignoring these couplings degrades quality, especially when removing many blocks at once. The number of combinations grows exponentially, making brute force seem hopeless. This is exactly the regime where statistical physics tools excel.
Turning block selection into an energy-minimization problem
The team attaches a binary variable to each transformer block: 0 to keep it, 1 to remove it, like a spin pointing down or up. They perform a second-order Taylor expansion of the model's loss to produce an approximate Hessian matrix. The diagonal of the Hessian represents each block's individual importance, while the off-diagonal entries capture the pairwise couplings—the many-body physics that mean-field methods ignore.
This reformulation turns block selection into a clean optimization: find the set of M blocks whose removal minimizes the energy xᵀH⁰x, subject to removing exactly M of N blocks. Mathematically, this is a constrained binary optimization (CBO) problem; physically, it is an Ising glass with conserved magnetization. The key property established is that this energy is a strong proxy for downstream quality: low-energy states correspond to high-performing pruned models.
Solving it: exact when you can, quantum-inspired when you can't
The practicality of this method lies in its cost. The Hessian is computed just once using a small calibration dataset. Afterward, evaluating any candidate configuration requires only a single cheap energy calculation, eliminating the need to run the actual model or benchmark it. The same Hessian can be reused for different compression targets.
For most models, the configuration space is large but still checkable. Because energy computation is cheap, brute-forcing on a single GPU can check up to tens of billions of spin configurations. For example, removing 8 of Llama-3.3-70B's 80 blocks (around 29 billion configurations) took roughly two days. Beyond exact approaches, the problem can be cast into a QUBO form and handed to highly optimized classical, quantum, and quantum-inspired solvers.
The payoff in deep compression
The payoff in the deep-compression regime is substantial. At 50% compression of Llama-3.3-70B-Instruct, this method gains almost 23 percentage points on MMLU over the best competing block-removal method.