Multimodal

Cross-attention

Queries come from one sequence, keys/values from another. How a decoder reads an encoder’s output.

First page of Neural Machine Translation by Jointly Learning to Align and TranslateIntroduced inSep 2014Neural Machine Translation by Jointly Learning to Align and TranslateBahdanau et al. · arXiv 1409.0473 ↗
Cross-attention: reading a different sequencequeries come from the current stream · keys and values from the other onedecoderencoder output (audio, image, prompt)

In cross-attention the queries belong to the current stream (say, generated text or noisy image latents) while keys and values come from a different one (encoder output, a text prompt, audio features). It is the bridge in encoder-decoder models like Whisper and the conditioning mechanism in diffusion U-Nets, where image latents cross-attend into the text prompt’s embeddings.

Adoption over time

Share of new models that have included cross-attention over time.

3%202220232024202520263%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

openai/whisper-large-v3-turbo architecture graphopenai/whisper-large-v3-turboautomatic-speech-recognition · ↓ 6.7M · ♡ 3kOpen in visualizer openai/whisper-large-v3 architecture graphopenai/whisper-large-v3automatic-speech-recognition · ↓ 4.7M · ♡ 6kOpen in visualizer stabilityai/stable-diffusion-xl-base-1.0 architecture graphstabilityai/stable-diffusion-xl-base-1.0text-to-image · ↓ 3.1M · ♡ 8kOpen in visualizer facebook/bart-large-mnli architecture graphfacebook/bart-large-mnlizero-shot-classification · ↓ 3.1M · ♡ 2kOpen in visualizer microsoft/Florence-2-base architecture graphmicrosoft/Florence-2-baseimage-text-to-text · ↓ 3.0M · ♡ 400Open in visualizer openai/whisper-small architecture graphopenai/whisper-smallautomatic-speech-recognition · ↓ 2.9M · ♡ 602Open in visualizer facebook/sam3 architecture graphfacebook/sam3mask-generation · ↓ 2.2M · ♡ 3kOpen in visualizer facebook/musicgen-medium architecture graphfacebook/musicgen-mediumtext-to-audio · ↓ 1.9M · ♡ 169Open in visualizer openai/whisper-tiny architecture graphopenai/whisper-tinyautomatic-speech-recognition · ↓ 1.7M · ♡ 446Open in visualizer Helsinki-NLP/opus-mt-nl-en architecture graphHelsinki-NLP/opus-mt-nl-entranslation · ↓ 1.6M · ♡ 10Open in visualizer Lightricks/LTX-2.5 architecture graphLightricks/LTX-2.5image-to-video · ↓ 1.6M · ♡ 5kOpen in visualizer openai/whisper-base architecture graphopenai/whisper-baseautomatic-speech-recognition · ↓ 1.5M · ♡ 289Open in visualizer

View 370 catalog matches for Cross-attention →

Related concepts

Patch embedding Patch merger / pixel shuffle Vision encoder (ViT) Multimodal projector

hfviewer renders the full architecture of 6,700+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →