Imagine spending weeks perfecting a high-performance engine, only to find that your car occasionally drives backward because of a tiny timing glitch in the transmission. That is exactly what happened to Filip Makraduli while optimizing the RMS norm layer in transformer models.
The Simple Version
In a transformer model, the RMS Norm (Root Mean Square Normalization) is like a quality-control inspector. It doesn't do much heavy lifting, but it has to check the data constantly—sometimes up to 33 times in a single decode step.
The problem isn't the math; it's the logistics. GPUs are like world-class chefs who can chop vegetables at light speed but take forever to walk to the fridge or start a new recipe. Every time the GPU starts a new norm operation, it wastes time on 'overhead' rather than actual computation.
How It Actually Works
FlashNorm uses three primary algebraic tricks to eliminate the 'waiting' time that kills GPU performance.
1. Weight Folding
Normally, the model performs a normalization and then applies a 'gain' (a weight) to the result. Weight folding moves this math offline. By folding the gain directly into the projection weights before the model even runs, the GPU can absorb both operations into a single matrix multiplication.
2. Deferred Normalization
In a standard pipeline, the GPU computes the norm, waits for it to finish, and then starts the matrix multiplication. FlashNorm defers the scalar division, allowing the matrix unit and the vector unit to work at the same time.
3. Dropping Redundant Norms
Some newer architectures, such as Gemma 4, perform normalization twice in a row. Because the operation is scale-invariant, the second normalization adds no value. FlashNorm simply drops the pre-normalization step to save time.
Technical Deep Dive
To implement deferred normalization, Makraduli had to move beyond Python and write custom CUDA kernels. This is because standard high-level libraries don't allow the fine-grained control needed to orchestrate different types of GPU cores simultaneously.
Modern GPUs have two distinct processing units: Tensor Cores, which are specialized for high-throughput matrix multiplication (MatMul), and CUDA Cores, which handle general-purpose element-wise operations and reductions (like the square root and mean in RMS Norm).
The goal was to launch the MatMul on the Tensor Cores and the RMS reduction on the CUDA Cores in parallel. However, this introduced a dangerous race condition. As Makraduli explains, the join between these two asynchronous streams was initially implicit. This meant the 'post-scale' operation sometimes began before the MatMul stream had actually finished writing its results to memory.
The result was a bizarre bug where the model began repeating itself with a one-step lag. As Makraduli puts it, the model was reading "outputs from the past" because the post-scale read a stale buffer from an unfinished multiply. To fix this, he had to implement an explicit join, marking the end of each stream and forcing the post-scale operation to wait until both the matrix and vector units had fully completed their work.
Why This Matters
These optimizations provide a 33 to 35 percent speedup for the norm and projection phase, which is significant given how often these layers are called during inference. Because the resulting 'folded' checkpoints are compatible with torch.compile and quantized models, these gains can be integrated into existing production pipelines without rewriting the entire model.
This research also highlights a critical limitation of 'rented' AI endpoints. Most cloud providers offer a black-box API where you cannot perform 'kernel surgery.' To deploy a model with modified CUDA kernels like FlashNorm, developers need an open inference engine—such as Superlinked—that allows them to deploy custom checkpoints and manipulate how the hardware actually executes the math.