Video-Index: A Curated Meta-Benchmark for Video Understanding

Video benchmarks are meant to show whether a model understands video. We attacked 115 of them with five kinds of shortcuts, and 35 break before the attacker sees a single frame. From the questions no shortcut solves, we select the 840 hardest to form Video-Index.

视频 benchmark 本该检验模型能不能看懂视频。我们用五类捷径攻击了 115 个视频 benchmark，其中 35 个，攻击者一帧画面都没看就攻破了。我们再从所有捷径都解不出的题里选出最难的 840 道，组成 Video-Index。

By Enxin Song作者 Enxin Song

GitHub Hugging Face arXiv

A mosaic of 115 benchmark tiles colored by capability group, and below it a chain of eleven screening stages from 505,518 items to the 840 of Video-Index. — **Figure 1.图 1** **Video-Index.Video-Index。**

The question问题

Each benchmark was built against the models of its year. Language priors are stronger now and one frame carries more of the answer, so a question that forced a 2023 model to watch the video may need no video in 2026. We asked every benchmark whether it can still tell if a model has the capability it claims to test.

每个 benchmark 都是针对它发布那一年的模型设计的。如今模型的语言先验更强，单看一帧能得到的答案线索也更多，所以一道在 2023 年非看视频不可的题，到 2026 年可能根本用不着视频。我们对每个 benchmark 都问同一个问题：它还能分辨出模型是否具备它声称考察的能力吗？

The attack pyramid攻击金字塔

An attacker is any strategy that raises the score without the claimed capability. We arrange attackers in five levels of growing access to an item, the attack pyramid.

攻击者指任何不具备 benchmark 声称考察的能力、却能提高分数的策略。我们按攻击者能拿到的题目信息从少到多，把它们分成五层，即攻击金字塔。

Level 1第 1 层Option选项

The answer options alone.

只有答案选项。

Level 2第 2 层Text文本

The question and options.

题干与选项。

Level 3第 3 层Pool题库

The other evaluated items.

评测中的其他题目。

Level 4第 4 层Frame帧

One frame or captions.

一帧画面或视频描述。

Level 5第 5 层Order顺序

The video, shuffled or a contiguous tenth.

打乱帧序的视频，或其中连续的十分之一。

After level 5第 5 层之后Unbroken未攻破

The attack pyramid.攻击金字塔。

A benchmark breaks at the first level where an attacker comes within 5 points of a reference model watching the video, Gemini 3.5 Flash at 32 frames, both counted above chance. A benchmark that no level breaks survives.

参照模型是看 32 帧视频的 Gemini 3.5 Flash。攻击者和参照模型都按超出随机水平的部分计分，攻击者第一次追到参照模型 5 个百分点以内的那一层，就是 benchmark 被攻破的层。五层都没被攻破的 benchmark 算作幸存。

Video-Index

Screening every item逐题筛查

We screen all 381,614 multiple-choice items of 112 audited benchmarks, dropping near-duplicates and every item that a small model answers from the options, the text, one frame, or 32 frames. The attacks remove 76% of the distinct questions.

我们筛查了审计过的 112 个 benchmark 中的全部 381,614 道选择题，去掉近似重复的题，也去掉小模型只凭选项、题干、一帧画面或 32 帧就能答对的题。这些攻击筛掉了去重后 76% 的问题。

Video-Index holds the 840 hardest verified questions from 76 sources, 210 per capability group and one per video. Selection ranks items by the attacker margins and ignores reference-model accuracy.

Video-Index 收录 840 道经过核验的最难题目，来自 76 个来源：每个能力组 210 道，每段视频只取一道。选题按攻击者的 margin 排序，不看参照模型的准确率。

Results结果

With the same fixed input, Claude Opus 5 outscores every open model by more than 37 points, and tools raise it by another 19.8. GPT-6-Astra leads at 79.3%. Human volunteers reach 55.0%, below the three leading agents and above every open model.

输入相同且固定时，Claude Opus 5 比所有开源模型都高出 37 个百分点以上，用上工具后又提高 19.8 个百分点。GPT-6-Astra 以 79.3% 排第一。人类志愿者的准确率是 55.0%，低于前三名 agent，高于所有开源模型。

Model模型	Video看视频	Blind盲读	Gain增益	Percep.感知	Temp.时序	Spatial空间	Reason.推理
GPT-6-Astra	79.3	18.7	60.6	75.4	83.7	80.4	77.1
Claude Fable 5.1	71.5	20.7	50.8	68.4	77.7	71.1	68.7
Claude Opus 5	70.6	24.9	45.7	68.4	77.2	69.1	67.6
Claude Opus 5	56.8	21.3	35.5	62.4	56.2	46.7	61.9
Human volunteers人类志愿者	55.0	17.5	37.5	56.0	68.0	56.0	40.0
Gemini 3.1 Pro	54.7	23.9	30.8	58.2	59.1	60.6	40.9
Gemini 3.5 Flash-Lite	22.5	17.5	5.0	20.0	30.0	10.0	30.0
Molmo2-8B	19.0	4.8	14.2	20.0	16.2	17.1	22.9
InternVL3.5-8B	15.2	4.4	10.8	13.3	20.0	8.1	19.5
LLaVA-OneVision-2-8B	14.2	1.8	12.4	11.4	21.0	11.9	12.4
InternVideo2.5-8B	12.4	7.5	4.9	12.9	11.0	8.1	17.6
VideoLLaMA3-7B	12.4	5.2	7.1	11.9	18.6	8.6	10.5
Video-XL-2	12.4	7.3	5.1	12.4	15.7	9.0	12.4
Cambrian-S-7B	11.3	5.3	6.0	12.4	10.0	3.8	19.0
Qwen3-VL-8B	9.5	1.4	8.1	9.0	11.4	6.7	11.0
VideoChat3-4B	9.4	3.8	5.7	9.5	12.4	6.2	9.5

Table 1.表 1 Agents lead open models on Video-Index.Video-Index 上 agent 领先开源模型。

How agents gather evidenceagent 如何找证据

GPT-6-Astra, Claude Fable 5.1, and Claude Opus 5 tile frames into grids on nearly every answer and then diverge. GPT-6-Astra seeks timestamps more often, Claude Fable 5.1 and Claude Opus 5 favor cropping and zooming, and Claude Fable 5.1 extracts far more images without reaching GPT-6-Astra’s accuracy. All three score lower on videos past ten minutes.

GPT-6-Astra、Claude Fable 5.1 和 Claude Opus 5 几乎每道题都会把帧拼成网格，之后的做法各不相同。GPT-6-Astra 更常跳到指定时间，Claude Fable 5.1 和 Claude Opus 5 更常裁剪和放大；Claude Fable 5.1 抽取的图像多得多，准确率却不如 GPT-6-Astra。三个 agent 在十分钟以上的视频上分数都更低。

Top left, accuracy against median time per item with quartile bars for five agents. Top right, the share of answers using each operation: sampling at intervals, seeking to times, tiling frames, cropping and zooming. Bottom, median images per item by video duration for each agent on shared log axes. — **Figure 12.图 12** **Agents differ in accuracy, runtime, and inspection.agent 在准确率、耗时和查看方式上各不相同。**

What the audit finds审计发现了什么

Newer benchmarks break earlier越新的 benchmark 越早被攻破

Among releases through 2024, 19% break before any visual input; among releases in 2025 and 2026, 34% do, and the rise comes from the text and pool levels. Recent benchmarks use language models to write or refine questions and answers, which can leave the answer recoverable from the text alone.

2024 年及以前发布的 benchmark 中，19% 在攻击者看到任何画面之前就被攻破；2025 和 2026 年发布的 benchmark 中，这一比例是 34%，多出来的部分来自文本层和题库层。近期的 benchmark 常用语言模型来写或润色问题和答案，答案因此可能单凭文字就能推出来。

Left, a funnel of benchmarks reaching each level: 115 at option, 98 at text, 85 at pool, 80 at frame, 53 at order, 38 survive. Right, stacked bars of breaking levels by release year, each as long as the year's releases: 6 for 2023 and earlier, 20 for 2024, 54 for 2025 and 35 for 2026, of which 17, 20, 39 and 26 percent break before video. — **Figure 6.图 6** **Newer benchmarks break earlier.越新的 benchmark 越早被攻破。**

Each claim breaks differently每类能力各有软肋

Reasoning and knowledge benchmarks break on the text or on captions, perception benchmarks most often at the frame level, and spatial and physical benchmarks on the options and under temporal perturbation. Temporal benchmarks split, with 6 of 18 breaking on the options and 10 surviving every level.

推理与知识类 benchmark 在文本或视频描述上失守，感知类最常在帧层失守，空间与物理类在选项层和时序扰动下失守。时序类两极分化：18 个里有 6 个在选项层就被攻破，10 个五层全部幸存。

Stacked horizontal bars of each capability group's benchmarks by breaking level, each as long as the group: temporal 18, spatial and physical 20, perception 45, reasoning and knowledge 32. — **Figure 7.图 7** **Each capability group breaks at its own levels.每个能力组在各自的层被攻破。**

Frames, pixels, and long video帧数、像素与长视频

Given more visual input, 11 benchmarks gain more from pixels and 3 from frames; on the rest the two gains differ by less than the tolerance. Resolution recovers small details in slides, endoscopy, and synthetic artifacts, and denser sampling catches brief events.

增加视觉输入时，11 个 benchmark 从更高的分辨率中得益更多，3 个从更多的帧中得益更多，其余 benchmark 的两种增益相差不到 5 个百分点。更高的分辨率能看清幻灯片、内窥镜画面和合成瑕疵里的小细节，更密的采样能抓到一闪而过的事件。

A density curve of pixel gain minus frame gain with named dots for benchmarks; three sit on the frames side and eleven on the pixels side. — **Figure 8.图 8** **Eleven benchmarks gain more from pixels and three from frames.11 个 benchmark 从像素中得益更多，3 个从帧数中得益更多。**

Long videos want more frames. Among the 30 benchmarks whose videos average at least five minutes, 18 gain at least 5 points from larger frame budgets, and 9 still gain at the reference’s 600-frame limit.

长视频需要更多帧。在视频平均时长至少五分钟的 30 个 benchmark 中，18 个加大帧预算后至少提高 5 个百分点，9 个到了参照模型 600 帧的上限时仍在提高。

Accuracy of 30 long-video benchmarks at frame budgets of 32, 128, 512 and 1024, with group means and quartiles: blue for the 9 where the reference still gains at 600 frames, red for the other 21. — **Figure 9.图 9** **Most long videos gain from larger budgets.多数长视频 benchmark 在更大的帧预算下得分更高。**

Errors and duplicates错误与重复

Errors concentrate in few capabilities. The two most frequent categories of a benchmark hold a median 71% of its attributed errors, and fine-grained action leads on 27 benchmarks, so covering the capability gaps takes several sources.

错误集中在少数几种能力上。每个 benchmark 最常见的两类错误合计占其归因错误的比例，中位数是 71%；细粒度动作在 27 个 benchmark 上排第一。因此要覆盖这些能力缺口，需要组合多个来源。

A heat map with 18 fine capability categories as rows and 76 benchmarks as columns grouped by first breaking level, each column summing to 100 percent. — **Figure 10.图 10** **Errors concentrate in few capabilities.错误集中在少数几种能力上。**

Table 12.表 12 Cross-benchmark near-duplicate shares (%).跨 benchmark 的近似重复占比（%）。

Which benchmarks to use for a capability按能力选 benchmark

38 of the 115 benchmarks survive every level. The strongest benchmarks for a capability are the survivors of its group that stay furthest below the breaking threshold.

115 个 benchmark 中有 38 个五层全部幸存。对某项能力来说，最可靠的 benchmark 是该能力组的幸存者中离攻破门槛最远的那些。

Tables 34–37.表 34–37 The ten strongest benchmarks of each capability group.各能力组最强的十个 benchmark。

Report cards报告卡

Citation引用

To cite this post:

引用本文：

@misc{song2026videoindex,
  title        = {Video-Index: A Curated Meta-Benchmark for Video Understanding},
  author       = {Song, Enxin},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://enxinsong.com/blog/video-index/}
}

← All posts← 全部文章 Back to top ↑回到顶部 ↑