An extra mini-block predicts the token after next - used to train richer representations and to speed up decoding.
MTP attaches one or more small extra transformer blocks that predict token t+2 (and beyond) alongside the main next-token head. At training time this densifies the learning signal; at inference the MTP head can draft tokens that the main model then verifies - self-speculative decoding that speeds generation without a separate draft model. DeepSeek V3/V4, GLM and Qwen3.5 all ship MTP blocks.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →