High-Quality Multi-View Image Extraction from a Light Field Camera Considering Its Physical Pixel Arrangement

Shu FUJITA  Keita TAKAHASHI  Toshiaki FUJII  

Publication
IEICE TRANSACTIONS on Information and Systems   Vol.E102-D   No.4   pp.702-714
Publication Date: 2019/04/01
Online ISSN: 1745-1361
DOI: 10.1587/transinf.2018AWI0004
Type of Manuscript: INVITED PAPER (Special Section on Award-winning Papers)
Category: 
Keyword: 
light field camera,  super resolution image synthesis,  multi-veiw image,  

Full Text: PDF(11.6MB)>>
Buy this Article




Summary: 
We propose a method for extracting multi-view images from a light field (plenoptic) camera that accurately handles the physical pixel arrangement of this camera. We use a Lytro Illum camera to obtain 4D light field data (a set of multi-viewpoint images) through a micro-lens array. The light field data are multiplexed on a single image sensor, and thus, the data is first demultiplexed into a set of multi-viewpoint (sub-aperture) images. However, the demultiplexing process usually includes interpolation of the original data such as demosaicing for a color filter array and pixel resampling for the hexagonal pixel arrangement of the original sub-aperture images. If this interpolation is performed, some information is added or lost to/from the original data. In contrast, we preserve the original data as faithfully as possible, and use them directly for the super resolution reconstruction, where the super-resolved image and the corresponding depth map are alternatively refined. We experimentally demonstrate the effectiveness of our method in resolution enhancement through comparisons with Light Field Toolbox and Lytro Desktop Application. Moreover, we also mention another type of light field cameras, a Raytrix camera, and describe how it can be handled to extract high-quality multi-view images.