Annoucement

- The CroMo paper is on arxiv! (2022/3/23)
- The CroMo dataset has been released! (2022/3/28)
- The CroMo paper has been accepted at CVPR 2022! (2022/3/28)

About

We introduce a new large-scale dataset for depth estimation from multiple modalities. CroMo consists of synchronized stereo polarisation, indirect ToF, structured light depth and IMU measurements, captured at video rates. It contains large frame counts captured simultaneously by hardware synchronised sensors and will serve as a challenging benchmark for the depth estimation community.

Learning-based depth estimation has witnessed recent progress in multiple directions; from self-supervision using monocular video to supervised methods offering highest accuracy. Complementary to supervision, further boosts to performance and robustness are gained by combining information from multiple signals. In this paper we systematically investigate key trade-offs associated with sensor and modality design choices as well as related model training strategies. Our study leads us to a new method, capable of connecting modality-specific advantages from polarisation, Time-of-Flight and structured-light inputs. We propose a novel pipeline capable of estimating depth from monocular polarisation for which we evaluate various training signals. The inversion of differentiable analytic models thereby connects scene geometry with polarisation and ToF signals and enables self-supervised and cross-modal learning.

In the absence of existing multimodal datasets, we examine our approach with a custom-made multi-modal camera rig and collect CroMo ; the first dataset to consist of synchronized stereo polarisation, indirect ToF and structured-light depth, captured at video rates. Extensive experiments on challenging video scenes confirm both qualitative and quantitative pipeline advantages where we are able to outperform competitive monocular depth estimation methods.

If you have any questions about CroMo, please contact yannick.verdie@huawei.com for further help.

Video

Citation

@misc{verdié2022cromo,
      title={CroMo: Cross-Modal Learning for Monocular Depth Estimation}, 
      author={Yannick Verdié and Jifei Song and Barnabé Mas and Benjamin Busam and Aleš Leonardis and Steven McDonagh},
      year={2022},
      eprint={2203.12485},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}