Video Models Start to Solve Chess, Maze, Sudoku, Mental Rotation, and Raven' Matrices

Code

• github.com/hokindeng/vmevalkit
  OfficialIn paper★ 83

Abstract

We show that video generation models could reason now. Testing on tasks such as chess, maze, Sudoku, mental rotation, and Raven's Matrices, leading models such as Sora-2 achieve sixty percent success rates. We establish a robust experimental paradigm centered on the "Task Pair" design. We build a code framework, with 39 models available already, that supports this paradigm and allows for easy scaling - users can add models and tasks efficiently. We show our automated evaluation strongly correlates with human judgment, and therefore this paradigm is highly scalable. We see an opportunity, given the availability of our paradigm, to do reinforcement learning for improving reasoning in video models. You could checkout all of our raw https://grow-ai-like-a-child.com/video-reason/results and our https://github.com/hokindeng/VMEvalKitVMEvalKit codebase.

Reproductions