A gated, depthwise causal convolution mixes only a few neighboring tokens - a very cheap substitute for attention.
Some hybrid models (LFM2, Nemotron-H) replace most attention layers with short causal convolutions: each position mixes information from just the last few tokens through a depthwise conv, with multiplicative gates deciding what passes through. It is dramatically cheaper than attention and surprisingly capable for local patterns; the few remaining attention layers supply global context.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →