Finding footage with one sentence: how close is multimodal video search?
"Find me the footage of someone in a red raincoat passing the north gate yesterday afternoon" — on a conventional system that means replaying channel after channel at speed, and it usually costs hours. Multimodal retrieval exists to solve exactly this.
The technical path has two steps. First, models process the video into structured form, extracting attributes such as people, vehicles, colour, direction and time to build an index. Second, retrieval matches a natural-language query or a reference image against that index. Recent progress has been concentrated in the second step: replacing exact keyword matching with semantic matching, so a phrase like "a person in a red raincoat" can be understood directly.
Two difficulties remain. Accuracy: real footage involves occlusion, backlight and low light at night, and recall and precision are hard to guarantee at the same time, so practice usually pairs retrieval with human verification. Cost: structuring all footage demands substantial compute, so indexing priority positions and priority hours is more workable than covering everything.
For users, the sensible start is to apply retrieval where the time cost is highest — incident review, tracing lost property, gathering evidence in a dispute — prove the value through measurable efficiency gains, and widen the scope from there.