Multimodal LLMs: Images, Audio, and Video

OpenAI vision is a shipping API, not a 2024 demo. So are Gemini and Claude vision. What matured is the endpoint. What did not mature is an unsourced leaderboard. No bake-off was run for this post, so no accuracy column.

Endpoints first

GPT-4o made omni-modal chat famous. The 2026-09 mainline for new OpenAI vision samples is GPT-5.6 (gpt-5.6). Gemini samples use gemini-3.7-flash (model page). Claude samples use claude-opus-5 (Opus 5 overview).

All three take images and return text. The useful difference is what the vendor documents: who accepts video, who is image-only, which file size is a hard reject, whether coordinates survive downscale. Those facts have URLs. A cross-vendor “who is more accurate” table does not.

Pick a general multimodal model when the job is one screenshot, one UI frame, or a short clip of audio that a human will read. Skip ranking vendors by invented percentages.

Images: three readers, one non-generator

OpenAI reads images on Responses or Chat Completions and generates on a different path (gpt-image-2). Inputs: URL, base64, or a Files API ID. Sample from the vision guide; not run here.

from openai import OpenAI

client = OpenAI()
response = client.responses.create(
    model="gpt-5.6",
    input=[
        {
            "role": "user",
            "content": [
                {"type": "input_text", "text": "what's in this image?"},
                {
                    "type": "input_image",
                    "image_url": "https://api.nga.gov/iiif/a2e6da57-3cd1-4235-b20e-95dcaefed6c8/full/!800,800/0/default.jpg",
                },
            ],
        }
    ],
)
print(response.output_text)

Documented constraints: PNG, JPEG, WEBP, non-animated GIF; 512 MB payload; 1,500 images per request. Unsuitable for medical scans. Small text, rotation, panoramas, precise spatial tasks, and counting are listed failure modes. detail: "original" can still resize. Images over the patch budget are rejected, not silently fitted. OCR, tiny UI chrome, and computer-use coordinates need a pre-resize that matches the sizing table, then a map back onto the original file.

Claude accepts JPEG, PNG, GIF, WebP. Direct API: 10 MB per image (5 MB on Bedrock and Google Cloud). 8000×8000 px max. Requests with more than 20 images tighten the per-image edge. Oversized images are downscaled unless the block opts into an error — fatal for bounding boxes. Official limitations: no people identification, hallucinations on low-quality / rotated / sub-200 px images, approximate counts, no synthetic-image detector, no image generation or editing.

import anthropic

client = anthropic.Anthropic()
message = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "url",
                        "url": "https://platform.claude.com/docs/images/vision-example.jpg",
                    },
                },
                {"type": "text", "text": "Describe this image."},
            ],
        }
    ],
)
print(message)

Failure to expect: a retina screenshot is silently resized, then every coordinate is wrong relative to the file on disk. Pre-resize, or set oversized images to error. Claude also wants the image before the text. Re-sending base64 on every turn inflates the payload; reuse file_id from the Files API.

Gemini image understanding lives at image understanding. Samples use gemini-3.7-flash. Inline bytes plus prompt cap around 20 MB; larger files go through the Files API. media_resolution caps tokens per image or frame. Schematic request shape from that page — not executed here:

# Schematic. Auth and URL: Gemini image-understanding docs.
curl -sS "$GEMINI_INTERACTIONS_URL" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.7-flash",
    "input": [
      {"type": "text", "text": "Caption this image."}
    ]
  }'

Pick Claude or GPT-5.6 when the rest of the toolchain is already there. Pick Gemini when the next turn is audio or video. Skip medical advice, CAPTCHAs, exact inventories, and “is this synthetic?”

Audio: transcript versus understanding

A meeting summary is two different jobs. Verbatim text is one product. Meaning is another. Their caps do not match, so they should not share one helper.

Words: OpenAI transcription. Default for new work: gpt-transcribe. Word timestamps: whisper-1. 25 MB per file. Diarization is gpt-4o-transcribe-diarize, explicitly not the recommended general transcriber (speech to text).

Meaning: Gemini audio lists describe, transcribe, translate, diarize, emotion, timestamped spans, and non-speech sound. Published limits: 32 tokens per second of audio (1,920 / minute); 9.5 hours per prompt; 16 kbps downsample; channels folded to mono; 20 MB inline; Files API above that. Realtime is the Live API. The generateContent path is not a realtime transcriber.

9.5 hours and 25 MB are different walls. The Whisper path hits file size first; a lower bitrate packs a longer recording. The Gemini path hits duration and tokens; audio is downsampled to 16 kbps, so ambience survives and spectral detail does not. Verbatim captions want a transcriber. “What was that alarm” wants an understanding model.

# Schematic fields from the Gemini audio page
# model=gemini-3.7-flash, input text "Describe this audio clip" plus an audio part

Claude’s current vision documentation is image → text. Capabilities that page does not claim are not treated as facts.

Pick OpenAI when the artifact is a transcript or caption file. Pick Gemini when the same model must answer questions about the recording. Skip unsourced accuracy as a vendor picker, and skip using an understanding summary as a subtitle file.

Video: only Gemini documents the file path

Gemini video understanding accepts File API uploads (20 GB paid / 2 GB free), Cloud Storage, inline under 100 MB, and public YouTube URLs. Use the Files API when the total request exceeds 20 MB, the clip is long, or the file will be reused. Default tokenization is about 300 tokens per second of video (about 100 at low media resolution). File API processing is documented at 1 FPS.

1 FPS means a one-frame UI flash can disappear. If a product demo hinges on a single highlight, send that still through the image path. Do not assume “the video was uploaded” means “every frame was seen.” The soundtrack is ingested with the pictures, which is the part a ten-frame collage cannot do.

The sample prompt on that page is Please summarize the video in 3 sentences. Model ID in the samples: gemini-3.7-flash.

OpenAI’s vision guide covers image analysis and image generation, not a documented mp4-understanding comparison. Video generation is a different product. Claude vision is still images. No official cross-vendor video accuracy, so none here.

Pick Gemini for product recordings that need picture and soundtrack together. Skip sending keyframes to an image-only model and calling that video understanding.

When the original file should not go in

Multimodal tokens are the expensive part. A 4K screenshot is billed after the documented resize, not at camera resolution. An hour of Gemini audio at 32 tokens per second is about 115k audio tokens before the text prompt. Crop, transcribe, or grab the key frame first when the question does not need layout, color, or the soundtrack.

Skip the original blob when the deliverable is a caption file (use transcription), a single button label (send that still), or a digital PDF table (extract text; scans still need vision).

Keep the original blob when the question depends on layout, arrows, overlapping speakers, or a mismatch between picture and voice. Those facts die in a pure-text reduction. Record the resolution or frame rate the model actually saw, so downscaled coordinates do not land in a ticket as if they were native.

OpenAI’s vision guide lists detail as low / high / original / auto. Not every model accepts original. Precise coordinates need the sizing table for that model ID; auto is not “native pixels.”

Limits

1. File caps and token rates can be checked. Unsourced accuracy cannot.
2. Understanding and generation are different APIs. GPT reads on Responses and paints with gpt-image-2. Claude only reads. Gemini’s understanding stack is not its image-generation models.
3. Downscale and 1 FPS change what the model saw. Quote coordinates and “what happened at 00:12” against the processed media unless the app resized first.
4. Visible failures: OpenAI rejects images over the patch budget; Claude tightens dimensions on many-image requests; Gemini inline payloads over 20 MB should move to the Files API instead of retrying. File API objects expire (48 hours on the files page). A later uri 404 is TTL, not a model regression.

Default: the general model already in the stack for stills. Gemini for long audio and video. Whisper / gpt-transcribe for verbatim text. A dedicated image model when pixels must come back out.