What is the plug-and-play video depth prediction framework?

(1) We propose the first plug-and-play learning-based video depth framework NVDS, which can be used for any monocular image depth prediction model to remove temporal jitter and enhance inter-frame consistency.

(2) The Video Depth in the Wild (VDW) dataset we proposed is currently the largest and most scene-rich natural scene video depth dataset.

As shown in the figure below, compared with previous video depth prediction methods, the proposed method NVDS has significantly improved spatial accuracy, temporal smoothness, and reasoning efficiency. At the same time, the Video Depth in the Wild (VDW) dataset proposed in this paper is currently the largest and most diverse natural scene video depth dataset.

1. Task Background and Motivation

Video depth prediction has an important impact on many downstream tasks (such as video bokeh rendering, 3D video synthesis, video special effects generation, etc.). An ideal video depth prediction model needs to solve two problems: (1) spatial accuracy of depth; (2) temporal consistency between frames. In recent years, monocular image depth prediction algorithms have significantly improved spatial accuracy, but how to remove jitter and improve temporal consistency between frames remains a difficult problem.

Mainstream video depth prediction methods rely on test-time training. During inference, they use geometric constraints and camera parameters to force a monocular image depth prediction model to overfit the temporal relationship of the current specific test video. This has two obvious disadvantages: (1) poor robustness. Camera parameters are often difficult to be accurate and reliable in many videos, resulting in obvious artifacts and completely wrong prediction results for methods such as CVD and Robust-CVD; (2) low efficiency. Taking CVD as an example, it takes more than forty minutes to process a 244-frame video on four Tesla M40 GPUs.

Therefore, a natural idea is whether we can establish a learning-based video depth prediction method that can directly learn the ability and prior of temporal consistency on the dataset and directly predict better results without the need for test-time training. As with all deep learning algorithms, the design and implementation of such a learning-based video depth method requires addressing two core issues: (1) reasonable model design that can model inter-frame dependencies and improve the consistency of prediction results; (2) sufficient training data to train and stimulate the best performance of the model. Unfortunately, the performance of previous learning-based video depth methods is still inferior to that of test-time training methods, and the effectiveness of the result design still needs further research and exploration. Due to the high cost of annotation, existing video depth datasets are still relatively limited in data volume and scene richness.

2. Methods and Contributions

To address the two core challenges mentioned above, we made two contributions:

(1) We propose the first plug-and-play learning-based video depth framework NVDS. NVDS consists of a depth predictor and a stabilization network. The stabilization network can be directly applied to any monocular image depth prediction model to remove temporal jitter and maintain inter-frame consistency. All previous learning-based video depth prediction models are stand-alone models, whose spatial performance cannot benefit from the monocular image model of sota, and vice versa, they cannot smooth and stabilize a large number of existing monocular image models. The NVDS method breaks the barriers between monocular image depth prediction and monocular video depth prediction. On the one hand, it can benefit from various high-precision single-image models, and on the other hand, it can smooth and stabilize any single-image model, achieving mutual promotion and a win-win situation. For the stabilization network, we use cross-attention to model the inter-frame relationship between the key frame and the reference frame. At the same time, we design a bidirectional prediction mechanism to expand the temporal receptive field and further improve consistency.

(2) We proposed the Video Depth in the Wild (VDW) dataset, which is currently the largest and most scene-rich natural scene video depth dataset. Due to the huge annotation cost, most of the existing video depth datasets are closed scenes. The few natural scene video depth datasets are far from sufficient in size and richness. For example, Sintel only contains 23 animation videos. Our VDW dataset is collected from a variety of data sources such as movies, animations, documentaries, and online videos. It contains 14,203 videos of more than 200 hours and a total of 2.23 million frames. We also designed mechanisms such as sky segmentation voting, as well as strict data screening and annotation processes to ensure the accuracy of our data. The figure below contains some examples of datasets, from online videos, documentaries, animations, and movies.

3. Experimental Overview: Methods

In terms of experiments, we have achieved SOTA spatial accuracy and temporal consistency on the VDW dataset, as well as the public Sintel and NYUDV2 datasets. VDW and Sintel are natural scene datasets. For closed scene data such as NYUDV2, we can achieve SOTA performance by training with the unified NYUDV2 training set instead of our VDW dataset; and pre-training with our VDW dataset and then fine-tuning on the closed scene NYUDV2 can further improve the performance of the model.

At the same time, to prove the effectiveness of our plug-and-play, we used three different depth predictors for experiments, and our NVDS achieved significant improvements.

We also demonstrated the effectiveness of bidirectional inference through ablation. Unidirectional (forward or backward) prediction can already achieve satisfactory consistency, while bidirectional inference can further expand the temporal receptive field and improve consistency.

Some qualitative results are shown in the figure below. Our NVDS method has achieved significant improvements. Each group of examples has an RGB frame on the left and a video time domain slice on the right. Fewer stripes in the slice indicate better consistency and stability. For more visualization results, please refer to our paper, supplementary materials, and results video.

4. Experimental Overview: Dataset

For the VDW dataset, we compared existing video depth datasets. Our VDW dataset is currently the largest and most scene-rich natural scene video depth dataset.

We also explored the effect of using different data to train the model. Since our VDW dataset has the best volume and scene richness, for the same model, training with the VDW dataset achieved the best performance.

For the statistical experiments of the dataset, we plotted the word cloud of the object category of the dataset, as well as the statistical graph of the semantic category, etc. For more statistical results and examples of the dataset, please see the paper and supplementary materials.

5. Open source code and datasets

Our code is open source:

https://github.com/RaymondWang987/NVDS

The dataset is building the official website of VDW and drafting the corresponding open source agreement. It will be released as soon as it is ready. Our dataset is large, so it will take some time to build the website and transfer the data. We will split the data and upload it gradually. The VDW dataset can be used for academic and research purposes, but not for commercial purposes.