Multimodal Video Understanding for Real-World Surveillance: Tasks, Dataset, Models, and Challenges

The presentation introduces multimodal video understanding tailored to real surveillance scenarios, outlining five major sections: an overview of video understanding tasks, background of a new surveillance video dataset, annotation guidelines, statistics of the dataset, and multimodal learning tasks.

Task Introduction includes video subtitle generation, video moment localization, and multimodal anomaly detection, each illustrated with examples and potential applications such as generating subtitles for movies, retrieving specific moments in long videos, and detecting abnormal events by fusing visual and textual cues.

Dataset Background describes existing surveillance datasets (e.g., UCF‑Crime) and their limitations, then introduces the newly constructed UCA dataset, which provides over 23,000 fine‑grained semantic annotations for 110.7 hours of video, with millisecond‑level timestamps and average description length of ~20 words.

Annotation Guidelines emphasize fine‑grained labeling, precise time recording, balanced description length, and clear action description, with a quality‑control process that reviews every 100 records.

Dataset Statistics show that most annotated clips are 5–10 seconds long, average duration ~16.8 seconds, and most descriptions contain 10–30 words.

Multimodal Learning Tasks enabled by the UCA dataset include video moment localization, video caption generation, and dense video captioning. Baseline methods for moment localization (CTRL, SCDM, 2D‑Tan, Moment Diffusion) are evaluated with IoU‑based recall, all achieving below 10 % recall on the surveillance domain.

The small‑model framework consists of a video encoder, a text encoder, a multimodal fusion module, and a prediction head, supporting the three downstream tasks.

For video captioning , classic models such as S2VT and recent Transformer‑based SwinBERT are compared; SwinBERT achieves the highest CIDEr score but still lags behind performance on general video datasets.

Large multimodal models (e.g., GPT‑4, Google’s multimodal LLMs) are discussed, and a pipeline that leverages a large language model for multimodal fusion (visual encoding, Q‑Former, text generation, and optional audio modality) is proposed.

Fine‑tuning experiments on the UCA dataset use VideoChat2 as the base model with a three‑stage LoRA strategy (freeze visual encoder, then jointly train visual and connector, finally adapt the language module). After three stages, CIDEr improves from ~10 % to ~25 % on the test set.

The presentation concludes with identified challenges—poor video quality, small object size, and sudden abnormal events—and future directions such as constructing VQA instruction data and integrating an anomaly‑frame detection module to build a comprehensive multimodal surveillance model.

A short Q&A addresses practical concerns about low frame‑rate streaming, frame extraction rates (8–16 fps), and strategies for real‑time multi‑camera processing, emphasizing lightweight models, data down‑sampling, and hierarchical filtering.