Multimodal
A multimodal AI is like a person who can see, hear, and read all at once, allowing it to understand and create things using a mix of pictures, sounds, and words instead of just typing.
A multimodal model is an AI architecture designed to ingest or output multiple data types—such as text, images, and audio—by mapping them into a unified vector space, allowing the system to relate different media formats to one another.
A model that processes and/or generates more than one modality (text, image, audio, video) within a shared representation, enabling cross-modal tasks.
evolution
- 2015 · historyShow and Tell
Google researchers introduced a neural network that could automatically generate captions for images, marking a foundational step in vision-language integration.
- 2017 · historyTransformer Architecture
The introduction of the Transformer model provided the unified mathematical framework necessary to process diverse data types as sequences.
- 2021 · historyCLIP
OpenAI released CLIP, which demonstrated that contrastive learning could effectively bridge the gap between visual concepts and natural language.
- 2022 · historyDALL-E 2
The release of DALL-E 2 showcased high-fidelity text-to-image generation, bringing multimodal AI into mainstream public awareness.
- 2023 · historyGPT-4V
The integration of vision capabilities into large language models enabled native multimodal reasoning and interaction.