A bank of parallel feed-forward networks; each token runs through only the few the router selected.
Each expert is an independent gated MLP. Because only the top-k experts execute per token, total parameter count and per-token compute decouple: a model can store hundreds of experts' worth of knowledge while running just a handful. Different experts end up specializing - by language, domain, or syntax - although the specialization is learned, not assigned.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →