The FFN splits into a 'gate' and an 'up' projection multiplied together - that's why you see three Linears and a Mul.
Modern transformer MLPs compute silu(gate_proj(x)) × up_proj(x), then project back down with down_proj - a gated linear unit (SwiGLU when the activation is SiLU). The elementwise multiply lets the network dynamically decide which features to pass, and it consistently beats the classic two-layer ReLU MLP at equal parameter cost. This is why decoder-block MLPs show three Linear layers and a multiply instead of two Linears.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →