Heads & prediction

LM head (output projection)

Projects the final hidden state onto vocabulary logits. Often reusing the input embedding matrix (‘tied weights’).

First page of Using the Output Embedding to Improve Language ModelsIntroduced inAug 2016Using the Output Embedding to Improve Language ModelsPress & Wolf · arXiv 1608.05859 ↗
LM head: one matrix, read twicetied weights: tokens→vectors at the input, hidden→vocab logits at the outputembedding matrix (shared)'cat'vectorhidden statelogits over the vocab

The LM head is a single linear map from the model’s hidden width to one logit per vocabulary token; softmax over those logits gives next-token probabilities. Many models tie it to the input embedding matrix, saving hidden×vocab parameters (often several hundred million) at negligible quality cost. Common in small and mid-size models, less so at the largest scales.

Adoption over time

Share of new models that have included an LM head over time.

73%2022202320242025202673%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

Qwen/Qwen3-0.6B architecture graphQwen/Qwen3-0.6Btext-generation · ↓ 29.7M · ♡ 1kOpen in visualizer amazon/chronos-2 architecture graphamazon/chronos-2time-series-forecasting · ↓ 29.5M · ♡ 390Open in visualizer sentence-transformers/all-mpnet-base-v2 architecture graphsentence-transformers/all-mpnet-base-v2sentence-similarity · ↓ 25.7M · ♡ 1kOpen in visualizer google-t5/t5-small architecture graphgoogle-t5/t5-smalltranslation · ↓ 25.1M · ♡ 589Open in visualizer FacebookAI/xlm-roberta-base architecture graphFacebookAI/xlm-roberta-basefill-mask · ↓ 20.9M · ♡ 878Open in visualizer Qwen/Qwen3-8B architecture graphQwen/Qwen3-8Btext-generation · ↓ 16.3M · ♡ 1kOpen in visualizer Qwen/Qwen2.5-1.5B-Instruct architecture graphQwen/Qwen2.5-1.5B-Instructtext-generation · ↓ 14.0M · ♡ 790Open in visualizer openai-community/gpt2 architecture graphopenai-community/gpt2text-generation · ↓ 13.8M · ♡ 3kOpen in visualizer autogluon/chronos-bolt-small architecture graphautogluon/chronos-bolt-smalltime-series-forecasting · ↓ 13.5M · ♡ 59Open in visualizer FacebookAI/roberta-large architecture graphFacebookAI/roberta-largefill-mask · ↓ 13.4M · ♡ 318Open in visualizer Qwen/Qwen3.5-9B architecture graphQwen/Qwen3.5-9Bimage-text-to-text · ↓ 12.4M · ♡ 2kOpen in visualizer Qwen/Qwen2.5-7B-Instruct architecture graphQwen/Qwen2.5-7B-Instructtext-generation · ↓ 12.1M · ♡ 1kOpen in visualizer

Browse all 1974 models with this in the catalog →

Related concepts

Multi-token prediction (MTP)

hfviewer renders the full architecture of 3,100+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →