Optimizing Large Language Model Block Pruning Using the Physics-Based Ising Model
Multiverse Computing has introduced a new LLM pruning methodology that models transformer block removal as a constrained binary optimization problem, leveraging the Ising model from physics to capture block interactions.
Traditional methods for pruning large language models often evaluate the importance of individual transformer blocks independently, neglecting the intricate dependencies between them. The newly proposed approach from Multiverse Computing reformulates block removal by treating the retention or deletion of each block as a binary variable. By framing the network as a spin system from statistical physics, the method models block interactions through a coupling matrix derived from the Hessian of the model loss.
Primary source
Open Hugging Face Blog ↗This article summarizes the source. Check the original for full details and conditions.
Comparison
| Aspect | Before / Alternative | After / This |
|---|---|---|
| Block evaluation | Independent scoring of each block in isolation | Coupled scoring capturing interactions between different blocks |
| Optimization framework | Heuristic sensitivity analysis or magnitude-based sorting | Quadratic Unconstrained Binary Optimization (QUBO) solved via Ising solvers |
| Computation scaling | Linear with respect to the number of model layers | Quadratic due to the calculation of off-diagonal Hessian elements |
