Google gives Gemini Flash models control over which video frames they read
Letting a model choose where to look, and how closely, shows why input selection matters as much as model size.
Google on 1 September announced a feature it calls agentic video understanding, which changes how its Gemini models read video. Rather than pulling frames from a clip at one fixed rate from start to finish, the model now decides which stretches of the footage deserve close inspection and how many frames per second to sample from each. The post was written by Rohan Doshi, a senior product manager at Google DeepMind, and Mario Lučić, a research director there.
The capability ships in Gemini 3.7 Flash, Gemini 3.6 Flash and Gemini 3.5 Flash-Lite. Google reports that the approach reduces token consumption by up to 88 percent and cost by up to 66 percent compared with fixed-rate processing, while lifting accuracy by up to 7 percent on the tasks it measured. The company describes Gemini 3.7 Flash as sitting on the accuracy-to-cost frontier for this kind of work.
The announcement lists four situations where the approach helps: finding a specific moment in a clip to sub-second precision for editing, working through recordings that run for several hours, detecting anomalies, and counting actions or objects. Each of these needs dense sampling in a small window and only a light skim elsewhere, which is exactly what a static frame rate cannot provide.
Developers can use the feature now through the Gemini API in Google AI Studio and through the Gemini Enterprise Agent Platform. Google says it is rolling out to the Gemini app soon, and that YouTube's "Ask YouTube" feature will pick it up over the coming months.
Why it matters
Video is the most token-hungry input a multimodal model handles, and uniform sampling spends most of that budget on frames that add nothing new. Letting the model allocate its own attention is the same shift that made tool use effective for text agents: the model plans what it reads instead of receiving a fixed dump. If the reported savings hold in production, multi-hour video analysis moves from a research demonstration to something a small team can afford to run every day. It also suggests the next round of efficiency gains in multimodal AI will come from smarter input selection as much as from larger models.
Video is the most token-hungry input a multimodal model handles, and uniform sampling spends most of that budget on frames that add nothing new.
- Token consumption reduction (up to)-88%
- Cost reduction (up to)-66%
- Accuracy improvement (up to)+7%
Figures: Figures published by Google DeepMind in its 1 September 2026 announcement; described as maximums.
What you can learn from this
Video becomes tokens, and tokens are the bill. A multimodal model does not watch a video; it receives a sequence of still frames, each encoded into a few hundred image tokens, plus any audio transcript. At one frame per second, a two-hour recording turns into 7,200 frames before the model has answered anything. Because both cost and latency scale roughly with token count, the number of frames you send is the single biggest lever you control, and it is set before the model sees a single pixel.
Adaptive sampling is coarse-to-fine search. The idea is to scan a video sparsely first, identify the segments that look relevant to the question, and then re-sample only those segments at a high frame rate. It works because most video is temporally redundant: consecutive frames in a static scene are nearly identical, so sampling them densely adds cost without adding information. A person scrubbing a timeline does the same thing, dragging quickly until something interesting appears and then stepping through frame by frame. The saving comes from skipping the uneventful parts, so a video with no uneventful parts saves little.
The word "agentic" here means the model plans its own inputs. In a classic pipeline the frames are chosen before the model runs. In an agentic loop the model observes a cheap overview, reasons about what it still needs, requests a closer look at a particular window, and repeats until it can answer. That observe-think-act cycle is the same structure used for tool-calling agents; the "tool" is simply the video decoder, and each call to it has a price. Each iteration of the loop is a decision that can be logged and inspected afterwards.
Read "up to" figures as a best case, not a guarantee. Savings of 88 percent on tokens depend heavily on the task: a needle-in-a-haystack query benefits enormously, while a question that requires every frame, such as counting every person who walks past a camera, benefits far less. Vendors report the top of the range because it is real for some workloads. Your job is to measure the number for yours before building a budget around it.
Temporal grounding is harder than it looks. Returning a timestamp to sub-second precision means the model must keep track of where each sampled frame sits on the timeline, even when the sampling rate changes mid-video. Systems typically attach explicit time metadata to each frame so the model reasons about position rather than inferring it from frame order, which would break as soon as the rate varies. Getting this wrong produces confident answers that point at the wrong second.
We teach this
How to use this in practice
Measure the token cost of one video at three sampling rates. Take a ten-minute clip you own, send it to the Gemini API with a moment-retrieval question, and read the token count from the usage metadata in the response. Repeat with the API's frame-rate option set low, default and high, keeping the question identical so that sampling is the only variable. Record the response time as well, since latency usually tracks token count. Done looks like a three-row table of tokens, cost, latency and whether the returned timestamp was correct.
Build a two-pass coarse-to-fine script. Use
ffmpeg -i input.mp4 -vf fps=0.1 coarse_%03d.jpgto pull one frame every ten seconds, ask a model which frame is closest to the event you care about, then extract five frames per second for the thirty seconds around that point and ask again. Keep the coarse frames small, around 480 pixels wide, so the first pass stays cheap. Done looks like a timestamp within one second of the true moment, and a note of how many frames each pass used.Draw the agentic loop and mark where money is spent. Sketch four boxes: overview scan, segment selection, dense resample, and answer. Next to each box write whether it costs tokens, compute time or nothing. Add a fifth box for the case where the first pass finds nothing relevant, and note what the loop should do then. Done looks like a diagram you could hand to a colleague to explain why the second pass is where the budget goes, and the one to bring to a cost review.
Write a twenty-question evaluation set from your own footage. Pick videos relevant to your work, write questions with known answers and exact timestamps, and store them as a JSON file. Include at least five questions where the correct answer is that the event never occurs, since those catch models that guess. Done looks like a script that runs the set against a model and prints accuracy and total tokens, so future claims of improvement can be checked against your data rather than a vendor's.
Sources
- Introducing agentic video understanding with Gemini — Google DeepMind
Our reporting is an original summary; full coverage is at the links above.
Don't just read about it — build it.
Square 1 teaches the skills behind the headlines, with every line of your work graded by AI. Find your starting point in 3 minutes.
Get your free skill report