2/6/2025 | 11:00 AM - 12:20 PM | Regency B

Multiplexed pixels: Light field camera with overlapping images for 3D reconstruction from few views

Author(s)

Vi Tran | Robinhood Markets, Inc
Shamus Li | Cornell University
Ruiming Cao | University of California, Berkeley
Sara Fridovich-Keil | Stanford University
Laura Waller | University of California, Berkeley

Abstract

Recent work in novel view synthesis has shown impressive capabilities to recover photorealistic 3D models from calibrated photographs. However, these methods typically rely on many camera poses to produce high-fidelity models, or operate with fewer training views and stronger regularization. In contrast, light field cameras replace the standard single lens with an array of microlenses, thereby encoding richer information that allows for limited-angle 3D reconstruction from a single measurement. However, this 3D information comes at the cost of decreased spatial resolution, as each microlens records an image over a small subset of pixels on the full camera sensor. We propose an imaging method that borrows aspects of light field imaging and novel view synthesis, enabling 3D reconstruction of a scene using a light field camera with multiplexed pixels, such that images formed by different microlenses may overlap on some pixels. Our method affords higher spatial resolution than traditional light-field cameras, because multiplexing allows each microlens a larger subset of the sensor. We demonstrate our method in simulation using Plenoxels as the underlying volume representation, training on 10 exposures or a single exposure, and find favorable reconstruction quality compared to optimizing with either a single-lens camera or a light field camera.