Multimodal AI

Why This Matters

The real world is not text-only. Customers upload photos of defective products. Doctors review medical scans. Engineers inspect factory floors on camera. Contracts arrive as scanned PDFs. Audio calls need transcription. For years, handling these inputs required separate specialized models stitched together with fragile pipelines. Multimodal AI changes the architecture: a single model can now receive an image, a PDF, a voice clip, or a video frame alongside text and reason across all of them simultaneously. Understanding multimodal AI opens up a far larger category of AI products.

Everyday Analogy

Think of a skilled human expert: a doctor can look at an X-ray, listen to a patient describe symptoms, read the lab report, and integrate all three sources to form a diagnosis. That is multimodal reasoning. Early AI systems could only read the lab report. Modern multimodal AI can do all three, and do it at scale.

What Is Multimodal AI?

A multimodal AI model is one that can process and/or generate more than one type of data modality. The primary modalities in current AI systems are:

• Text: Natural language, the original modality of large language models.
• Images: Photographs, diagrams, charts, screenshots, medical images, documents.
• Audio: Speech, music, ambient sound, phone calls.
• Video: Sequences of frames combining visual and audio information.
• Structured data: Tables, code, time-series (often treated as a text modality via formatting).

Multimodal models fall into two categories:

Understanding models (also called encoder-heavy): accept multi-modal inputs and produce text output. Example: you send an image and ask "What is wrong with this product?" and the model replies in text.

Generation models (decoder-heavy): accept text (or other inputs) and produce non-text outputs such as images, audio, or video. Example: you send "a sunset over the ocean, photorealistic" and the model produces an image.

Some frontier models, like Gemini Ultra and GPT-4o, do both: they can understand images and generate them within the same system.

Vision-Language Models (VLMs)

Vision-language models extend large language models with the ability to process images. The architecture typically has two components:

1. Image encoder: Converts an image into a sequence of vector embeddings. The most common approach uses a Vision Transformer (ViT), which divides the image into patches and encodes each patch as a token. A 512×512 image might be divided into 256 patches of 32×32 pixels each, producing 256 image tokens.

2. Language model backbone: The LLM that generates text. Image tokens are injected into the input sequence alongside text tokens, allowing the language model to attend to both.

A projection layer (a small linear layer or MLP) maps the image encoder's output into the same embedding space as the language model's text tokens.

Leading Vision-Language Models

What VLMs Can Do

Document parsing: Extract text, tables, and structure from scanned PDFs, invoices, contracts, and forms, far more accurately than traditional OCR because the model understands layout and context.

Chart and diagram reading: Interpret bar charts, line graphs, flowcharts, and engineering diagrams, answering quantitative questions about the data.

Image captioning and alt-text generation: Automatically describe images for accessibility, content moderation, or search indexing.

Visual question answering (VQA): Answer natural language questions about image content: "How many defects are visible?" "What brand is the product?"

Multimodal document Q&A: Send a PDF with embedded images and ask questions that require understanding both the text and the figures together.

Screenshot understanding: Interpret UI screenshots, error messages, and dashboards, enabling AI coding assistants that see what the developer sees.

Image Generation: Diffusion Models

The dominant technology for AI image generation is the diffusion model. The core idea (see AI-079 for a technical deep dive into the diffusion process, latent diffusion, and conditioning mechanisms):

1. During training, clean images are progressively corrupted by adding Gaussian noise across many steps until only random noise remains. The model learns to reverse this process, denoising step by step.
2. During inference, the model starts with pure noise and iteratively denoises it, guided by a text prompt, until a coherent image emerges.

This process is conditioned on text using a technique called classifier-free guidance: the model is trained both with and without text conditioning, and at inference time the text-conditioned and unconditioned predictions are combined to steer the image strongly toward the prompt.

Key Diffusion Model Concepts

Latent diffusion: Rather than operating in pixel space (computationally expensive), modern models like Stable Diffusion operate in a compressed latent space. A VAE (Variational Autoencoder) compresses the image to a small latent representation; diffusion happens in that compressed space; the VAE decoder reconstructs the full image. This is why inference is fast enough to run on consumer hardware.

Text encoder: The text prompt is encoded by a language model (CLIP or T5) into embeddings that condition the denoising process via cross-attention.

Sampling steps and schedulers: The number of denoising steps (typically 20–50) and the noise schedule (DDPM, DDIM, DPM++) trade off between image quality and inference speed.

img2img: Instead of starting from pure noise, img2img starts from a real image with some noise added. The result is a new image that preserves the structure of the input while following the text prompt. Used for style transfer, variation generation, and inpainting.

Inpainting: Mask a region of an image and let the model fill it in based on the text prompt and the surrounding context.

Leading Image Generation Models

Audio AI

Speech-to-Text (ASR)

OpenAI Whisper is the dominant open-source automatic speech recognition model. Trained on 680,000 hours of multilingual audio, Whisper transcribes speech to text in 99 languages with word-level timestamps, speaker diarization support, and strong performance on accented speech. It is the backbone of countless transcription pipelines.

Managed STT services include:

• OpenAI Whisper API: pay-per-minute transcription.
• Google Cloud Speech-to-Text, enterprise-grade with real-time streaming.
• AWS Transcribe integrates with the AWS ecosystem.
• AssemblyAI adds speaker labels, topic detection, and sentiment analysis.
• Deepgram offers ultra-low-latency STT for real-time applications.

Text-to-Speech (TTS)

AI-generated voices have crossed the uncanny valley. Leading providers:

• ElevenLabs: Best-in-class voice cloning and synthesis. Clone a voice from 30 seconds of audio. Supports emotional control.
• OpenAI TTS: Fast, natural-sounding voices via API. Six built-in voices (Alloy, Echo, Nova, Shimmer, etc.).
• Google Cloud TTS: WaveNet and Neural2 voices with SSML support for fine-grained prosody control.
• Microsoft Azure Neural TTS: 400+ neural voices across 140 languages.

Music Generation

• Suno: Generates full songs with vocals, lyrics, and instrumentation from a text prompt.
• Udio: High-quality music generation with genre and mood control.
• MusicGen (Meta): Open-source music generation model.

Video Generation

Video generation is the newest and most rapidly evolving frontier of multimodal AI.

Sora (OpenAI): Generates high-quality 1080p videos up to 60 seconds from text prompts. Uses a diffusion transformer (DiT) operating on video patches across space and time. Notable for understanding physics, camera motion, and scene consistency.

Runway Gen-3 Alpha: Professional-grade video generation and editing. Strong temporal consistency. Used in film and advertising production.

Kling (Kuaishou): Chinese video generation model with strong physical realism and motion quality. Competitive with Sora.

Luma Dream Machine: Fast video generation with natural motion and camera control.

Pika: Consumer-friendly video editing and generation. Strong for product demos and social content.

Current Limitations of Video Generation

Video generation remains challenging:

• Duration: Most models generate 5–30 seconds. Long-form coherent video is still out of reach.
• Cost: Generating a few seconds of 1080p video costs orders of magnitude more than a text query.
• Consistency: Characters and objects can change appearance between cuts. Long-term narrative consistency is poor.
• Physics and hands: Models still struggle with hands, finger counts, and realistic physics in many scenarios.
• Latency: Generation takes seconds to minutes, not suitable for real-time applications.

Multimodal Embeddings

CLIP (Contrastive Language-Image Pre-training) from OpenAI is a foundational multimodal model that learns a shared embedding space for images and text. An image and its caption are encoded into vectors that end up close together in embedding space. An unrelated image and text end up far apart.

CLIP enables:

• Zero-shot image classification: "Is this photo a cat or a dog?", with no task-specific training needed.
• Image search by text query: Search a photo library with a natural language query.
• Semantic image similarity: Find visually similar images in an embedding index.
• Conditioning for diffusion models: CLIP's text encoder is used to condition Stable Diffusion on prompts.

More recent models extend CLIP's approach: ImageBind (Meta) aligns six modalities (images, text, audio, depth, thermal, IMU) in a single embedding space, enabling cross-modal retrieval and reasoning.

Engineering Multimodal Pipelines

Sending Images to APIs

Most VLM APIs accept images as base64-encoded strings or URLs. Example with the Anthropic API:

import anthropic, base64

client = anthropic.Anthropic()

with open("invoice.png", "rb") as f:
    image_data = base64.standard_b64encode(f.read()).decode("utf-8")

message = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image",
                "source": {
                    "type": "base64",
                    "media_type": "image/png",
                    "data": image_data,
                },
            },
            {
                "type": "text",
                "text": "Extract all line items from this invoice as JSON."
            }
        ],
    }],
)

Image Token Economics

Images are expensive in token terms. Understanding the economics is critical for production pipelines:

Tips to control image costs:

• Resize images to the minimum resolution needed before sending them.
• Use low-detail mode when fine visual detail is not required.
• Cache responses for identical images using prompt caching where supported.
• For high-volume pipelines, consider pre-processing images with a smaller/cheaper model to filter irrelevant inputs.

File Format Considerations

• Most VLM APIs accept JPEG, PNG, GIF (first frame), and WebP.
• PDFs must be converted to images (page by page) before sending; use pdf2image or equivalent.
• For audio, WAV and MP3 are universally supported; the Whisper API also accepts FLAC, M4A, OGG, and WebM.
• Video APIs typically accept MP4; extract key frames if using a VLM rather than a native video model.

Real-World Multimodal Applications

Document Q&A: A legal platform uploads contracts as PDFs, converts them to images, and uses Claude or GPT-4V to answer questions that require reading both text and signature blocks, stamps, and tables.

Visual product inspection: A manufacturing line captures high-resolution images of parts on a conveyor belt. A VLM classifies defects in real time, routing defective parts for manual review.

Accessibility alt-text: An e-commerce platform uses image captioning to generate descriptive alt-text for every product photo, improving screen reader accessibility and SEO.

Audio transcription pipeline: A call center transcribes every customer call with Whisper, then uses an LLM to extract intent, sentiment, and action items, replacing a manual review team.

Meeting intelligence: Platforms like Otter.ai, Fireflies.ai, and Notion AI capture meeting audio, transcribe it, identify speakers, and generate structured summaries and action items automatically.

Medical imaging assistance: Radiologists use VLMs to pre-annotate X-rays and CT scans, highlighting regions of interest for human review, not replacing the radiologist, but accelerating their workflow.

Challenges and Limitations

Hallucinations on images: VLMs can confidently describe objects, text, or relationships in an image that are not actually present. This is especially dangerous in medical, legal, or safety-critical contexts. Always validate critical image interpretations.

Cost per image: At scale, image tokens can cost 10–100× more than equivalent text tokens. A pipeline processing millions of images daily can incur significant costs.

Latency: Image processing adds latency compared to text-only queries. For real-time applications, model selection and image size optimization are critical.

Content moderation: Image generation models can be used to create harmful, misleading, or infringing content. Most providers have guardrails, but adversarial prompting remains an active challenge.

Copyright and training data: The legal status of AI-generated images derived from copyrighted training data is unsettled in many jurisdictions. Commercial use of generated content carries legal risk in some markets.

Video coherence: Long-form video generation lacks narrative consistency and often suffers from character identity drift, temporal discontinuities, and unnatural motion.

Summary

Multimodal AI is transforming what AI products can do by extending language model reasoning to images, audio, and video. Vision-language models can read documents, answer questions about images, and generate captions at scale. Diffusion models produce photorealistic images from text prompts, with capabilities like inpainting and img2img enabling powerful editing workflows. Audio AI enables transcription and synthesis pipelines that rival human performance. Video generation is rapidly maturing but remains expensive and limited for long-form content. Engineering multimodal pipelines requires understanding image token economics, file format constraints, and the real limitations of VLM hallucination. The teams who master multimodal AI will build the next generation of AI products, ones that interact with the world the way humans do.

Try It Yourself

1. Test cross-modal understanding: give a multimodal chatbot a photo of your whiteboard or a chart and ask "what's wrong or missing here?", then ask the same question in text only. The capability gap you observe is the practical value of shared embedding spaces.
2. Probe the failure modes: show a model an image with small text (a receipt, a dense slide) and ask for exact numbers. Verify against the original; you'll calibrate exactly how far to trust vision for detail work.

Key Takeaways

• Multimodal AI models process and/or generate text, images, audio, and video within a single system.
• Vision-language models use a ViT image encoder and an LLM backbone to reason over images and text together.
• Diffusion models generate images by iteratively denoising random noise guided by a text prompt.
• Audio AI enables STT (Whisper), TTS (ElevenLabs, OpenAI), and music generation (Suno, Udio).
• Video generation (Sora, Runway, Kling) is powerful but expensive and limited to short clips.
• CLIP aligns image and text embeddings in a shared space, enabling zero-shot classification and semantic search.
• Image tokens are expensive, so resize images and use low-detail modes to control costs in production.
• VLMs hallucinate on images; never rely on VLM output without validation in high-stakes applications.