Factored One-sided Adam-Moment for Practical and Scalable SOAP

Factored One-sided Adam-Moment for Practical and Scalable SOAP

Second-order optimizers promise faster training than AdamW by using information about the geometry of a model’s loss landscape. Methods such as Shampoo and SOAP can improve convergence, but their additional curvature statistics, matrix operations, and eigendecompositions make them expensive in memory and wall-clock time. The paper’s authors—Jaemyung Yu, Byeongho Heo, Sangdoo Yun, and Dongyoon Han of NAVER AI Lab—ask whether SOAP’s optimization benefits can survive a much leaner implementation.

Their answer is FOAM, or Factored One-sided Adam-Moment. The method combines two reductions to SOAP’s state and computation: it keeps only the input-side curvature factor, and it replaces SOAP’s dense rotated second-moment matrix with a row-and-column factorization. The authors report that this combination preserves SOAP-like validation performance while approaching the practical cost of AdamW or Muon.

Why SOAP is difficult to deploy

AdamW maintains first- and second-moment statistics for each parameter. SOAP adds a more structured approximation to curvature. For a weight matrix, it tracks Kronecker factors on both the input and output sides, periodically eigendecomposes them, and maintains a dense second moment in the resulting rotated coordinates.

That structure is useful, but costly. In the authors’ accounting, SOAP requires roughly 4.5 times AdamW’s optimizer state in its conventional form. Its two factor matrices also require periodic eigendecompositions, and the per-step work increases substantially with layer width. These costs can erase the benefit of reaching a target loss in fewer optimization steps.

How FOAM simplifies SOAP

Keeping the input-side factor

FOAM discards SOAP’s output-side Kronecker factor and retains the input-side factor. This reduces the factor state from both an output-sized and input-sized matrix to the input-side matrix alone, while eliminating one eigendecomposition.

The authors motivate this choice through the curvature of affine layers. Their analysis says that the input features contribute a shared outer-product structure, while the output-side curvature in transformer layers is often close to diagonal or block-diagonal. In that situation, the input axis contains more of the useful non-diagonal structure, making input-only preconditioning a more informative approximation than output-only preconditioning.

This is not merely a theoretical preference in the paper. In layer-wise Gauss–Newton alignment measurements, input-only preconditioning retained substantially more alignment with the measured curvature than output-only preconditioning. The authors report that dropping the output-side rotation cost only about a quarter of the alignment, whereas dropping the input-side rotation cost about 80 percent.

Factoring the rotated second moment

Even one-sided SOAP would still maintain a dense second-moment array with one value for every element of a weight matrix. FOAM instead tracks row and column statistics of the squared, input-rotated gradient. Their outer product reconstructs a rank-one approximation, with a trace-normalization factor used to preserve the overall scale.

Under the paper’s K-FAC assumption, and with a diagonal output-side curvature, the authors show that this factorization exactly recovers the relevant SOAP second moment. In experiments, the assumption was not exact: the reported relative violation was generally around 15–20 percent, with the attention key projection being a notable outlier. However, the authors measured only a small effect on the resulting update and validation loss.

What the experiments show

The main experiments trained GPT-style language models with 124 million, 350 million, and 760 million parameters on FineWeb at a fixed compute-optimal token budget. FOAM’s validation loss was slightly worse than SOAP’s at each scale, by 0.004, 0.008, and 0.005 nats respectively, but remained ahead of AdamW, Muon, and AdaMuon in the reported comparisons.

The resource results are more consequential. FOAM used half of SOAP’s matrix-parameter optimizer state. At 760 million parameters, the reported matrix-state sizes were 11.87 GiB for FOAM and 23.73 GiB for SOAP. FOAM’s per-step time was close to Muon’s and far below SOAP’s. It reached AdamW’s final loss in 10.53 hours, essentially matching AdamW’s 10.55 hours, while SOAP required 22.66 hours in the same comparison.

The authors also report that FOAM remained robust across a three-fold learning-rate range and showed similar tolerance to SOAP across several other hyperparameters. Additional results extended the comparison to a 1.56-billion-parameter GPT-2 XL experiment, ViT-Tiny image classification, larger batch sizes, and two Qwen3 post-training recipes. These results are supportive, but they should not be treated as a demonstration that FOAM is ready for every model or training regime.

Limitations and open questions

The paper’s central limitation is scale. The main study reaches 760 million parameters, with supporting evidence at 1.56 billion, not frontier-scale language models. The authors also state that their implementation does not yet shard the input-side factor and its eigenbasis across devices, an issue that could matter at models of 7 billion parameters or more.

FOAM is also not uniformly accurate across layers. The attention key projection showed the largest departure from the factorization assumption, likely because its gradients interact with rotary position embeddings and softmax. Embeddings and the language-model head remain on AdamW because their vocabulary dimensions make dense factor handling unattractive. These choices mean the reported method is best understood as a matrix-parameter optimizer component, not a complete replacement for every AdamW state.

Practical implications

FOAM appears most useful when SOAP’s quality is attractive but its memory and eigendecomposition costs are prohibitive. It offers a plausible middle ground: more structured curvature information than AdamW, with substantially less state and runtime than bilateral SOAP. The paper’s results also suggest that matrix-preconditioned methods become particularly valuable at large effective batch sizes, where AdamW degraded more sharply in the authors’ tests.

Teams should nevertheless validate the optimizer on their own architecture, batch regime, hardware, and layer mix. Muon or AdaMuon may remain preferable when absolute optimizer-state minimization is the primary constraint, while AdamW retains the advantage of maturity and simplicity. FOAM’s strongest demonstrated claim is narrower but useful: on the tested transformer scales, SOAP-like training quality does not necessarily require SOAP’s full computational overhead.


Sources

AI-assisted research summary. Check the original paper for complete methods, results, and limitations.

Avatar photo

Tech News

Jeff is a passionate blog writer who shares clear, practical insights on technology, digital trends and AI industries. With a focus on simplicity and real-world experience, his writing helps readers understand complex topics in an accessible way. Through his blog, Jeff aims to inform, educate, and inspire curiosity, always valuing clarity, reliability, and continuous learning.

↑ ↓ navigate Enter open Esc close