We asked professional video editors a simple question: where exactly do AI-generated product videos fall apart? This dataset holds 544 timestamped notes on fifteen videos made by Google Veo 3.1, Adobe Firefly Video, and Grok Imagine.
We gave all three models the same five prompts, with a focus on product shots. Professional video editors from the Contra network watched every clip and marked in detail what they saw as the positives and issues in these videos.
The professional editors were asked to label on five important video quality dimensions: Brand & Text Consistency, Material & Texture Realism, Camera Shot Adherence & Quality, Multi-Shot Cuts & Continuity, and Product Consistency. They also rated how much each issue matters, from High down to Low. What we collected is a close-up view of where the current frontier video models still struggle on real professional work.
Each annotation includes a start and end timestamp, a free-text comment, a dimension label, and a severity rating. This structure makes the data directly usable in several training contexts:
- Multimodal reasoning mid-training: pair video segments with expert critique to teach models to reason about temporal consistency, physical plausibility, and visual quality in motion.
- RLHF and reward modeling: use severity ratings as preference signals to train reward models that score generated video quality the way a professional would.
- Fine-tuning video evaluation models: train automated critics or judges that can flag specific artifact types (text hallucination, texture instability, bad cuts) at the frame level.
- Benchmarking and evals: the cross-model, same-prompt structure makes it a natural held-out test set for comparing new video models against a human expert baseline.
The dataset was collected in June 2026. It is released under the CC BY 4.0 license on Hugging Face.