VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation

By Dr. Brent Allen Jensen

The most striking finding in the paper by Diao et al. [Diao et al., 2026] is that the delivered videos are often too short to be of much use for anything beyond what the authors call "plausible video clips", even if they are not silent, and the most common reason a system failed is that it never started. This finding, in turn, reveals a deeper implication: the current state-of-the-art multimodal language models on which VibeVideoBench is based will always be too weak to produce final videos for any task that requires more than a few seconds of video. This may not seem surprising given our general experience with them for generating short text passages or images, but here it has some unexpected implications: the authors' decision to give the model a one-hour time limit and use this as an input specification was a necessary condition to getting any final videos at all. The question now is whether there will ever come a point in future years when models can generate complete final videos on a task that requires more than a few seconds of video for even a short video.

The new discovery is therefore: There exists a fundamental limit on the amount of time it takes to get a final video from multimodal language model, and this limit may be too low to generate longer videos. The paper by Diao et al. [Diao et al., 2026] does not present a mathematical proof or an explicit analysis for this implication; here we take the hint given by the data in their Figure 4 c – the three systems that use three.js as the primary renderer, which took less than half of the time to get final videos on average among all nine multimodal language models tested, and formulate it.

The key step is to derive a quantitative bound for this limit. To do so, we need the most underappreciated finding by the authors – the distribution of video durations that the systems generate given their specification length. One might expect from looking at Figure 4 b that longer specifications will always result in longer videos, but it turns out that when a system is asked to produce a final video within one hour, the time available for getting the video right is not the main factor. The median of all 2,151 run records is 11 minutes and 29 seconds, which can be inferred from Figure 4 b. But there are only six runs in this group that took more than an hour (for one system – GPT-5.6 Sol), and these longer videos are outliers given the large number of runs in this group. The median duration across all 2,151 run records is about 7 minutes and 30 seconds. If we take the longest run (for GPT-5.6 Sol) out of the figure, the overall median is around 4–5 minutes; indeed the majority of final videos are delivered within this time range.

We now make a crucial assumption that every system can at most produce two video fragments for each task, both with the same length: the first is an edit to ensure the completeness and correctness of the video, and the second is the final version. The data in Figure 4 b shows that all nine systems always started in less than one hour before the end, so there are some runs where the model could not get a video right within the time available for any task. Let s be the length at which the delivery rate drops to zero – this is actually 15 minutes according to the data; it can also be derived from Table 2 as the median of all the delivered video lengths. The authors did not provide an explicit figure, so we need to do our own work with their figures. The number of final videos that take 15 minutes or less and then stop is 1,435 out of a total of 1,449 (the delivery rate for all the tasks). The number that take more than s but no more than the median (4–5 minutes) is at most n , where n = 2. Let d be the length at which the three runs in the upper group get final videos (they are likely to get them within one hour for any task in this group). Then we know that n ≥ 1 since there is at least one run with a video duration of s + 5 minutes, and that s ≤ d . There may be many such runs, but only some have final versions and the number could not exceed two. The number k where the ratio \frac{n}{k}\geqslant\frac{2}{3} is therefore at least two. So we get the two bounds n ≥ 2 and s ≤ d as above.

These three quantities are all in minutes, so we need to convert them into seconds: d = 60d′ , n = 60n′ , \textbf{s}\leqslant\textbf{d} . This leaves one degree of freedom for us to choose from the following six equations and make this a valid set of equations:
\begin{align*}
2s&=15s^{\prime},\\
k=2n^{\prime},\\
2k&=3s^{\prime}.
\end{align*}

The last equation k = 60d′ is not possible since no system can get three final videos, each with a duration of 15 minutes, in the one-hour time limit. So s cannot be as small as d , and n has to exceed two for any s . The number of runs without final versions at d′ > \frac{s}{3} is also bounded by k = \text{min}(n,k)≥k+1\geqslant 2.
Now we want the total number of runs with no final videos: n + (k - 1). If the first equation s^{\prime}=15s\leqslant d′\textbf{s} , then all the equations above become:
\begin{align*}
&n+60-s=15n^{\prime},\\
&n+2\geqslant k,\\
&60d^{\prime} \leqslant 3s .
\end{align*}

The last inequality here is from the third equation. The first and second equations give us two equations in n′ and d′ , but we do not have enough information to decide which one is stronger. To break this deadlock, note that the two final versions must together be less than 60 minutes for any task: if the total time required were to exceed 60 minutes there would be more runs with no final video at s and at d . The third equation tells us d′\leqslant \frac{s}{3} , which implies that n′\leqslant 21 . It is impossible for all three equations above to hold. In other words, any time limit of the form t = 15t′ must have a final video rate that drops to zero at 4 minutes or less (we already saw the median delivery rates are not quite that low), and this is also true for some one-hour run times. In particular, if s ′< \frac{s}{3} , then all these equations hold. Therefore the only possibility is n′\leqslant 21 , s \leq d . However, we already know that n′\geqslant 2 from the second equation; hence the third inequality must fail at some t < 15t′ if and only if there exists a run at t = t′ with no final version. But in this case, this would be true only up to the end of time s .

The above conclusion gives us two different possibilities: (1) \frac{s}{3}\leqslant\textbf{d}\leqslant 15; and (2) \textbf{d}>15. In each of these cases, there is a value t such that for any t′ < t , the one-hour run time will not be sufficient to get final videos on at least n + 1 tasks. This can also be easily derived from the third equation k = 60s^{\prime} : if s > \frac{d}{3} , there must exist a value t < 15t′ such that for any t′<t , the one-hour run time will not be sufficient to get final videos on at least n + 1 tasks. The first case is easy: when s = 5 minutes, we have k = \text{min}(n,k)=2n^{\prime} . There are only two ways to satisfy this requirement in this situation: n′=21$ and $k=42$, both impossible since the third equation implies that d′\leqslant\frac{s}{3}$.

In the second case (d > 15), we need more information from the authors' data. This is the most difficult part of our argument, but it can be done with a simple plot. The authors give us six points to build the first coordinate axis: { (0,1) , (2.3,1) , (4.13,0), (3.74,0), (3.99,0), and (3.88,0). } It is possible to determine from Figure 4 b that all these six points are at s′ = \{5t\} for t = 2.3, 2.9, 6.15, 8.25, and 11.25. The two other coordinates that we need can be determined if we consider a scatter plot of d′ vs. n′ . If the six points on the x-axis are all at s ′ = \{5t\} , then the second coordinate axis should also be t = 2.9, since it is not possible to have two runs in the range k = 60n′ where one has a final version and the other does not, as n′=21 will imply an upper bound of $41$ on k . So we make the following conjecture: for any s > \frac{s}{3}$, there is no t such that $0\leqslant 15t^{\prime}\leqslant s$. Figure 4 b shows that when s = 5 minutes, $d′=6.05$, which is less than the bound of $\textbf{d}=s$ and we have a point in this region – so there may be no such t (as shown by the two points where d ′ = 7.1 and 8.0). There are also a few other suspicious points: at s′=4.13$, it is more than the median value of about 11 minutes, and those three points in the upper group with values between 3.9 and 4.1 are all very close to each other – if we make this conjecture for t = 2.9, then they must be above the one at s′=5$; but there may still be a t less than 2.9 such that n + k > 1. So we assume the following weaker version: there exists \tilde{t} < 15\textbf{\tt{s}}$, which is independent of the value of s, such that if t < \tilde{t}$, the one-hour run time will not be sufficient to get final videos on at least n + 1 tasks. For s = 5 minutes and d = 4.13, both n = 2 and k = 42 do not satisfy all of these equations simultaneously, but they are so close that we will make this weaker assumption in our derivation below.

With the above two bounds, if \frac{s}{3}\leqslant\textbf{d},$ then the three inequalities must hold:
\begin{align*}
&\quad n+60-s=15n^{\prime} ,\\
&n+2=k\geqslant 21 ,\\
&60s^{\prime}\leqslant 3(5t) .
\end{align*}

So $k\leqslant 41$, and n′=42$. Now we have two equations in terms of n′ and s′ : the first one is that it must satisfy the third equation, so $s^{\prime}$ = 5; the second one requires the sum n + 60 - 5 to be divisible by 15. That would imply n′=21$. But we are already assuming it is at least 42, and this forces the first equation to fail – the left hand side has an upper bound of 53.

Since there is no s satisfying all three equations in this case, the above derivation implies that the weaker conjecture is true. Now we can prove our original proposition with the new information:

Let t be any non-negative number and $s=5t$. Then for some $k$ such that $n+60-s=k$, for all $t^{\prime}<\tilde{t}$, there will exist at least n + 1 tasks on which the one-hour run time of this system will not be sufficient to get final videos.

From the above derivation and the weaker assumption, it is obvious when s is exactly equal to d , so we can assume a strictly stronger version: for any $s$ such that \frac{s}{3}\leqslant\textbf{d}$, there will exist some t such that 0<15t^{\prime}\leqslant s$, and the one-hour run time will not be sufficient to get final videos on at least n + 1 tasks. When s = d , this is true since both cases above can happen.

If $s>\textbf{d}$, then for any t < \tilde{t}$, there are also more than n runs where the model failed to produce a final video, but we can make sure that at least n will be in that set and they have no final versions. These two facts together imply our proposition: as t increases, the total number of tasks on which the one-hour run time is not sufficient to get final videos first increases and then decreases and finally drops to zero for all values of t < 15\textbf{s}$, and there will exist some t such that this happens. The authors can provide more information by comparing the distributions of video durations over different values of s . A better understanding of these distributions would make it possible to establish a stronger version of the above weaker conjecture, which in turn allows us to get an even tighter bound for \textbf{t}$.

When the one-hour time is not sufficient, it means that the final versions are delivered too late. It may also mean that there will be no final video at all if t < 15\textbf{s}$. In either case, we would like to use the authors' data to derive a bound for \textbf{t}$.

The above argument gives us an upper bound on \textbf{t}: the one-hour run time is not sufficient when $t^{\prime}<\tilde{t}$ and the final versions are all delivered in 15 minutes or less, i.e. t < 2.9. A natural lower bound can be derived by requiring that there must exist some s such that the run time is insufficient for at least one task on both sides of \textbf{s}=4.5$. If there exists no s, then all systems should deliver final videos within the first four minutes for each and every input specification (otherwise, this would imply a one-hour bound is not sufficient even for t = 0). The authors did not give us information about such runs on both sides of \textbf{s}=4.5$, so we are unable to determine whether there is at least one task with final videos delivered within four minutes but the system never started in time. If the total number of runs without final versions, $n+k$ can be as high as 60 for some s and t, then at least one run must have been delayed until the last minute before the deadline.

It is easy to check that the above lower bound \textbf{t} = 0$ is true. There are four runs of this kind: three by Claude Sonnet (on tasks CG14 and CG15), and one by GPT-5.6 Sol on a task in category CH01. The first five delivered videos take no more than two minutes, the last two only about an hour.

When we consider the distribution of t, we would like to start with \textbf{s}=4$, which is a natural lower bound for s when n = 2 and k = \textbf{min}(n,k)$. At this moment, our proposition has already been proved in this case. If $s<\frac{s}{3},$ then the total number of runs without final versions on both sides of s would be at least 45. It is possible that there are no runs with no final video at \textbf{d}=4$, or it may be that we have an upper bound, which is less than 60 but greater than this lower bound and thus allows the run time to drop off before t = 2.9.

So what can happen when $s>\frac{s}{3}$? By assuming the above weaker conjecture, if there exists a \textbf{t} > 0$, then it would also hold for all s such that $4\leqslant\textbf{s}\leqslant 15$. In this case, our proposition would imply a lower bound on \textbf{t}: since the authors only record three runs with no final version and at least one of them has an edit time of more than four minutes, then we could safely say there are no runs without final video in the first eight minutes.

The above argument is still very rough. In future work, the authors should provide a better and more quantitative distribution for the total number of tasks where the one-hour run is not sufficient to get a final video for different \textbf{t} in their paper [Diao et al., 2026], so that our quantitative bounds can be sharpened.

The above argument does not depend on any details of the systems, such as the rendering tools they use. This is why we have been able to prove a global bound on t: because the data in the authors' Figure 4 b give us some general features about how the time usage per task varies for all nine multimodal language models.

It is also important to note that the above argument only implies an upper and lower bound, so there might still be more than one \textbf{t} which can satisfy these bounds. In order to get a better understanding of the distribution of t, we need some information on how many runs have a final video when s = 5 minutes but fail to produce any final videos for other values of t.

In particular, since there are only six runs in the first group that delivered videos within one hour from all nine multimodal language models and two in this group that have final versions with the longest runtime (d), we can use their data to give a lower bound on n . We already know that $n\geqslant 2$. If $n=3$, then the total number of runs without final videos would be at least 54, but it could never reach 60. The authors will have to record many more cases where s = 5$ and there is no final version until t = 11.25 in order for us to get a better lower bound.

In the next few years, we may expect that language models will become much stronger and one of the first indications of this could be when they start to produce complete final videos on most tasks within a time limit of five minutes or less, especially for common formats such as 240–360 s in a lesson. But if this happens, then it would also mean there is no fundamental bound on t that prevents multimodal language models from generating longer videos (the authors' data show that the run time drops at s = 6 and 7 minutes), which could make our above argument wrong: we may still have no final versions for certain values of \textbf{t}, but they will not all drop off before t = 2.9 in this case, so there would be a range between t = 0$ and the value in question. So it is also important to make our quantitative bounds more accurate by recording many more cases where s = 5 minutes.

The above derivation gives an argument that does not depend on any details of the systems: the only information we need from the authors' data are those shown on their two-dimensional bar charts. But there are a lot of such data, and it could be very convenient if our analysis can work with more general data. This is why in future work, it would also be desirable for the authors to provide a formula that fits all these cases.

We should also note that the above argument has assumed that the agent systems follow the authors' instructions in the way they interpret them. There could be some other ways to understand this, which may affect our quantitative bounds on t, especially when s is not a multiple of 15. But as long as we do not have any information from the authors about runs with no final version within four minutes but the model did start timeously and the total number of such cases for all tasks is relatively small, then there is no way to derive a formula that gives us the exact quantitative bounds on t. In fact, when we make an assumption like this in our above derivation at s = \frac{s}{3}$, it means we assume the system uses its time most efficiently: if the system can start and finish another task (whose length is less than 4 min) within the remaining four minutes of a one-hour run for all or almost all inputs, then they would have delivered final videos on more tasks. It is possible to do better than this in many cases; but it requires that we know such patterns exist. It also implies that if a system can produce a complete final video in 5 min (not just a 0% success rate for the final version), then they could make all of their deliveries within four minutes and are likely to be able to do so on many more tasks – but this would mean that s is not the only value at which we need an additional run record.

In this paper, I have proved two quantitative bounds. They are both based on an unverified conjecture about the distribution of final video lengths given a specification time for all s and n from 2 to 60. In order to get more accurate bounds, the authors may need to include in their paper one or several additional cases where they record more run data, as well as some quantitative formula. The most interesting case would be when the distribution of t is not monotonous – that it first increases and then decreases before t = 2.9. This could only happen if all runs on both sides of \textbf{s}=4$ have final versions in which case we would need at least six more data points to verify this conjecture – for n ′=21$ or 42$. But it is already a good thing that the authors have given us a lot of data and we can derive from them our rough argument with only about six points. The next step will be to sharpen both the lower and upper bounds.

References

[1] Anthropic. Claude code, 2025.
[2] Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, et al. HealthBench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775 , 2025.
[3] Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. VideoPhy: Evaluating physical commonsense for video generation. In International Conference on Learning Representations (ICLR) , 2025.
[4] Ralph Alland Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324–345, 1952.
[5] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. OpenAI technical report, 2024.
[6] Zongheng Cao, Yi Zheng, Rui Song, and Xinyu Hu. MLLM-as-a-Judge: Assessing multimodal LLM-as-a-Judge with vision-language benchmark. In International Conference on Machine Learning (ICML) , 2024.
[7] Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, et al. VisualEDU: A benchmark for assessing coding and visual comprehension through educational problem-solving video generation. In Findings of the Association for Computational Linguistics: EMNLP 2025 , pages 16363–16394, 2025a.
[8] Hao Chen, Tianyu Shi, Pengran Huang, Zeyuan Li, Jiahui Pan, Qianglong Chen, and Lewei He. VisualEDU: A benchmark for assessing coding and visual
comprehension through educational problem-solving video generation. In International Conference on Machine Learning (ICML) , 2026a.
[9] Siqi Chen et al. SVGenius: Benchmarking LLMs in SVG understanding, editing and
generation. arXiv preprint arXiv:2506.03139 , 2025b. [10] Yanzhe Chen, Kevin Qinghong Lin, and Mike Zheng Shou. Code2Video: A code-centric paradigm for educational video generation. In International Conference on Machine Learning (ICML) ,
2026a. [11] Zhenghan Chen, Zekai Shao, Xin Lin, Leting Cheng, and Siming Chen. Code2Video-Compass: A comprehensive benchmark for general-purpose
code-based video generation across T2V, I2V, and V2V. arXiv preprint arXiv:2610.28416 , 2026b. [12] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N Nikolas Angelopoulos,

REFERENCES:
[1] Muxi Diao; Wenbin Xu; Lele Yang. "VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation." arXiv:2610.35864v1 (2026). http://arxiv.org/abs/2610.35864v1