CamP: Camera Preconditioning
for Neural Radiance Fields

Neural Radiance Fields (NeRF) can be optimized to obtain high-fidelity 3D scene reconstructions of objects and large-scale scenes. However, NeRFs require accurate camera parameters as input --- inaccurate camera parameters result in blurry renderings. Extrinsic and intrinsic camera parameters are usually estimated using Structure-from-Motion (SfM) methods as a pre-processing step to NeRF, but these techniques rarely yield perfect estimates. Thus, prior works have proposed jointly optimizing camera parameters alongside a NeRF, but these methods are prone to local minima in challenging settings.

In this work, we analyze how different camera parameterizations affect this joint optimization problem, and observe that standard parameterizations exhibit large differences in magnitude with respect to small perturbations, which can lead to an ill-conditioned optimization problem. We propose using a proxy problem to compute a whitening transform that eliminates the correlation between camera parameters and normalizes their effects, and we propose to use this transform as a preconditioner for the camera parameters during joint optimization.

Our preconditioned camera optimization significantly improves reconstruction quality on scenes from the Mip-NeRF 360 dataset: we reduce error rates (RMSE) by 67% compared to state-of-the-art NeRF approaches that do not optimize for cameras like Zip-NeRF, and by 29% relative to state-of-the-art joint optimization approaches using the camera parameterization of SCNeRF. Our approach is easy to implement, does not significantly increase runtime, can be applied to a wide variety of camera parameterizations, and can straightforwardly be incorporated into other NeRF-like models.

We demonstrate the impact of our method on the SE3+Focal camera parameterization. We apply a small perturbation to each parameter and visualize how preconditioning unifies the scale of the parameters (right) whereas the original parameterization exhibits large differences (left).

Modern cellphones are able to estimate pose information using visual-inertial odometry, which is often used for Augmented Reality (AR) applications. While the poses are sufficiently good for AR effects, it can be challenging to recover high quality NeRFs from those sequences without resorting to running expensive, offline SfM pipelines.

Here we show reconstructions computed from casually captured scenes using the open source NeRF Capture app on an iPhone 13 Pro, which exports camera poses estimated by ARKit. We show qualitative comparisons that highlight the benefits of camera optimization.