Multimodal

Cross-attention

Queries come from one sequence, keys/values from another. How a decoder reads an encoder’s output.

First page of Neural Machine Translation by Jointly Learning to Align and TranslateIntroduced inSep 2014Neural Machine Translation by Jointly Learning to Align and TranslateBahdanau et al. · arXiv 1409.0473 ↗
Cross-attention: reading a different sequencequeries come from the current stream · keys and values from the other onedecoderencoder output (audio, image, prompt)

In cross-attention the queries belong to the current stream (say, generated text or noisy image latents) while keys and values come from a different one (encoder output, a text prompt, audio features). It is the bridge in encoder-decoder models like Whisper and the conditioning mechanism in diffusion U-Nets, where image latents cross-attend into the text prompt’s embeddings.

Adoption over time

Share of new models that have included cross-attention over time.

2%202220232024202520262%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

openai/whisper-large-v3-turbo architecture graphopenai/whisper-large-v3-turboautomatic-speech-recognition · ↓ 7.7M · ♡ 3kOpen in visualizer openai/whisper-large-v3 architecture graphopenai/whisper-large-v3automatic-speech-recognition · ↓ 4.6M · ♡ 6kOpen in visualizer facebook/bart-large-mnli architecture graphfacebook/bart-large-mnlizero-shot-classification · ↓ 3.2M · ♡ 2kOpen in visualizer microsoft/Florence-2-base architecture graphmicrosoft/Florence-2-baseimage-text-to-text · ↓ 2.8M · ♡ 393Open in visualizer openai/whisper-small architecture graphopenai/whisper-smallautomatic-speech-recognition · ↓ 2.8M · ♡ 585Open in visualizer facebook/sam3 architecture graphfacebook/sam3mask-generation · ↓ 2.2M · ♡ 3kOpen in visualizer openai/whisper-base architecture graphopenai/whisper-baseautomatic-speech-recognition · ↓ 2.0M · ♡ 286Open in visualizer facebook/musicgen-medium architecture graphfacebook/musicgen-mediumtext-to-audio · ↓ 2.0M · ♡ 166Open in visualizer stabilityai/stable-diffusion-xl-base-1.0 architecture graphstabilityai/stable-diffusion-xl-base-1.0text-to-image · ↓ 1.6M · ♡ 8kOpen in visualizer openai/whisper-tiny architecture graphopenai/whisper-tinyautomatic-speech-recognition · ↓ 1.4M · ♡ 441Open in visualizer facebook/bart-large-cnn architecture graphfacebook/bart-large-cnnsummarization · ↓ 1.4M · ♡ 2kOpen in visualizer NeoQuasar/Kronos-base architecture graphNeoQuasar/Kronos-basetime-series-forecasting · ↓ 1.2M · ♡ 256Open in visualizer

Browse all 204 models with this in the catalog →

Related concepts

Patch embedding Patch merger / pixel shuffle Vision encoder (ViT) Multimodal projector

hfviewer renders the full architecture of 4,000+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →