[ECCV 2024] Sparse Beats Dense: Rethinking Supervision in Radar-Camera Depth Completion

Huadong Li^*, Minhao Jing^*, Jing Wang, Shichao Dong, Jiajun Liang, Haoqiang Fan, Renhe Ji^‡

MEGVII Technology

^*Equal contribution ^†Lead this project ^‡Corresponding author

News

• [🔥🔥🔥 2024.07.18] Code is released.

Overview

It is widely believed that sparse supervision is worse than dense supervision in the field of depth completion, but the underlying reasons for this are rarely discussed. To this end, we revisit the task of radar-camera depth completion and present a new method with sparse LiDAR supervision to outperform previous dense LiDAR supervision methods in both accuracy and speed.

Specifically, when trained by sparse LiDAR supervision, depth completion models usually output depth maps containing significant stripe-like artifacts. We find that such a phenomenon is caused by the implicitly learned positional distribution pattern from sparse LiDAR supervision, termed as LiDAR Distribution Leakage (LDL) in this paper. Based on such understanding, we present a novel Disruption-Compensation radar-camera depth completion framework to address this issue. The Disruption part aims to deliberately disrupt the learning of LiDAR distribution from sparse supervision, while the Compensation part aims to leverage 3D spatial and 2D semantic information to compensate for the information loss of previous disruptions.

Preparation

• Environment

  • Detailed environment settings should be found with requirements.txt

pip3 install -r requirements.txt

• Dataset
  • Download the segmentaion masks files at Google Drive
  • Please visit the Official Nuscenes Website to download the dataset.

-  data
  - nuscenes_radar_5sweeps_infos_test.pkl
  - nuscenes_radar_5sweeps_infos_train.pkl
  - nuscenes_radar_5sweeps_infos_val.pkl
  - nuscenes
     - samples
     - seg_mask.tar

cd data/nuceneses
tar xvf seg_mask.tar

• Pretrained weights

  • Please Download our pretrained weights on Google Drive.

mkdir checkpoints & cd checkpoints
mv model.ckpt ./chekpoints/model.ckpt

Training

OMP_NUM_THREADS=4 torchrun --nproc_per_node ${GPU Nums} train_DDP.py

Eval

python3 eval.py -m ./checkpoints/model.ckpt

BibTeX

@misc{li2023sparsebeatsdenserethinking,
      title={Sparse Beats Dense: Rethinking Supervision in Radar-Camera Depth Completion}, 
      author={Huadong Li and Minhao Jing and Jiajun Liang and Haoqiang Fan and Renhe Ji},
      year={2023},
      eprint={2312.00844},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2312.00844}, 
}

Contact

If you have any questions, feel free to open an issue or contact us at lihuadong@megvii.com or jirenhe@megvii.com.