A transformer over image patches that turns pictures into a sequence of visual embeddings.
The vision tower of a multimodal model is usually a ViT-family encoder (SigLIP and CLIP variants dominate) pretrained on image-text pairs. It converts patch embeddings into contextualized visual features that a projector then aligns with the language model's embedding space. Multimodal capability lives or dies by this component and how it was pretrained.
Open any of these on hfviewer to find this block in the interactive architecture graph.
hfviewer renders the full architecture of 2,300+ Hugging Face models as interactive graphs - hover any block to see what it does, with this model's real numbers.
Browse all model graphs →