TL;DR: A multimodal model handles more than text - it can look at an image, listen to audio, or watch video and reason about it in the same conversation. The trick is representation: every input type is converted into the token-like pieces one transformer already understands, so "describe this chart" and "describe this poem" run through the same machinery.
How it works
Language models work on sequences of tokens, and multimodality is the art of making everything look like one. An image is sliced into a grid of patches, each patch encoded by a vision network into an embedding that lands in the same space as word embeddings; audio is chopped into short time slices and encoded the same way. The transformer then attends across all of it at once - which is why a model can connect "the figure in the upper left" in your text to the actual pixels in the upper left. Practically, a screenshot simply costs some number of tokens out of your context window, just as a paragraph does.
A concrete workflow shows the point. Photograph a whiteboard covered in boxes and arrows, hand it to a multimodal model, and ask for a working prototype: the model reads the sketch, infers the layout, and writes the code - no transcription step in between. The same mechanics let models extract tables from PDFs of scanned invoices, describe screenshots for accessibility, debug a UI from a screen capture, or answer questions about an hour of video. Output is going multimodal too: image and speech generation are increasingly folded into the same systems, though many products still route those through separate specialized models behind one API.
The capability has limits worth engineering around. Vision inherits the usual failure modes of generation - a model can misread a chart axis or "see" text that is not there, hallucination with pixels as the trigger. Fine-grained perception (tiny fonts, dense tables, precise counts) trails human eyes, and image inputs consume context quickly, so resolution and page counts matter. Anything truly critical read from an image deserves verification, the same as any other model claim.
For builders, multimodality is what makes several whole categories work: computer use agents navigate real screens by looking at them, and document pipelines skip brittle OCR by letting the model read pages directly. Frontier model families - Gemini, Claude, and GPT among them - now treat image understanding as table stakes rather than a separate product.
Where it sits in the AI stack
Multimodality lives at the model's front door - encoders translate every medium into the shared sequence the transformer reasons over:
Key tools and implementations
-
Frontier multimodal models
Claude, Gemini, and GPT all accept images natively; several handle audio and video too.
-
Open vision-language models
Downloadable families like Llama's and Qwen's vision variants for self-hosted image understanding.
-
CLIP-style encoders
Models that place images and text in one embedding space, the backbone of cross-modal search.
-
Speech models
Transcription and voice-generation systems increasingly folded into the same multimodal APIs.
Related entries
- Token The chunk of text - roughly three-quarters of a word - that a language model reads and writes one at a time.
- Embedding A list of numbers capturing a piece of content's meaning, so similar things sit close together in vector space.
- Gemini Google's multimodal model family, accessed through the Gemini API, Vertex AI, and Google's own products.
- Computer use An agent capability where the model reads the screen and issues clicks and keystrokes to operate software built for humans.