Ask a coding agent to inspect a repository and it has excellent primitives: read files, search text, run commands. Ask it what happens in an Instagram reel or a product demo and the useful abstractions disappear. A video is a container, an audio stream, thousands of frames and often a login wall.
Reelay turns that moving target into a small folder with a single report. The report contains metadata, a timestamped transcript and a contact sheet. The original video is deleted when processing finishes unless I explicitly keep it.
reelay "https://www.instagram.com/reel/XXXXXXXXX/"
# → ~/.reelay/instagram-XXXXXXXXX/READ.mdChoose frames by change, not only by time
Sampling every five seconds is predictable and often wrong. A fast cut may be missed completely, while a static talking head produces twelve nearly identical frames.
Reelay asks ffmpeg for scene changes first. Those frames carry the visual transitions most likely to matter. If there are not enough of them, evenly spaced samples fill the remaining slots so that a static scene still has timeline coverage.
The default is twelve frames. They are kept individually for close inspection, but the first thing the agent sees is the contact sheet: one grid, one image request, one view of the whole timeline.
One image changes the context bill
Vision input has overhead per image. Sending nine separate frames means paying that overhead nine times and giving the model nine objects to relate mentally.
On a 59-second reel, I measured three ways of presenting the same visual evidence:
The contact sheet used roughly one ninth of the context of separate images. An optional free vision model can describe the grid and reduce the next agent's input again. That last step is deliberately optional: a text summary loses small labels and visual nuance, so the original frames stay available when the question turns on detail.
Compression is useful only if the evidence remains reachable.
The transcript needs to admit uncertainty
When a platform provides subtitles, Reelay uses them. Otherwise it can send audio to Whisper. The awkward part is not transcription itself; it is deciding how much to trust the result.
Whisper sometimes invents stock phrases such as a closing thank-you over music or silence. Confidence values can still look high. Reelay therefore removes a transcript made entirely of known phantom phrases and marks the clip as having no speech. If the transcript covers very little of the video, the report labels it weak evidence instead of presenting it as complete.
Long audio is processed in chunks. A failed chunk is reported with the missing time range instead of discarding the entire transcript. The report is meant to be useful to an agent, but also honest enough that the agent knows where not to infer.
The small interface is the feature
Anything supported by yt-dlp can use the same command: YouTube, Instagram, TikTok, Vimeo, Loom and many other sites, plus local video files. Cookies can be borrowed from a signed-in browser when a platform requires them.
The result does not force the calling agent to understand codecs, subtitle formats or scene detection. It gets a path to READ.md and can answer the user's actual question.
That is the pattern I keep finding in agent tools: the valuable layer is rarely a new model. It is a stable reduction from a messy medium into evidence the model can already reason over.
Inspect the pipeline or use it on a local video.