Scores all experts per token and activates only the top-k - most of the model's weights stay idle each step.
In a mixture-of-experts layer the router is a small linear layer that scores every expert for the current token; only the top-k scorers actually run, and their outputs are combined with the router's weights. This is how models grow to hundreds of billions of parameters while spending the compute of a much smaller dense model per token. Routers are trained with auxiliary objectives (or bias tricks) to keep expert load balanced.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →