The neural radiance field (NeRF) constructs an implicit representation function to substitute the traditional 3D representation, such as point cloud, mesh, and voxels, leading to consistent and efficient image rendering at desired observing spatial position. However, NeRF requires dense sampling in 3D space to build the continuous representation function. The huge amount of sampling points occupies intensive computing resources, which hinders NeRF from being integrated into the lightweight system. In this paper, we present a learning-based sampling strategy, which conducts dense sampling in regions with rich texture and sparse sampling in other regions, extremely reducing the computation resources and accelerating the learning speed. Furthermore, to alleviate the additional computation overhead caused by the proposed sampling strategy, we present a distributed structure to conduct the sampling decision individually. The distributed design releases the computation burden on the devices, which enables the deployment of the proposed strategy to the practical systems.
We review the recent deep learning reconstruction algorithms for spectral snapshot compressive imaging (SCI), which used a single shot measurement to capture the three-dimensional (3D, x, y, λ) spectral image. Recent years, deep learning has been the dominant algorithm to conduct reconstruction due to high speed and accuracy. Various frameworks such as end-to-end neural networks, deep unfolding, plug-and-play networks have been developed. Furthermore, the untrained neural networks have also been used. In this paper, we review diverse deep learning methods for spectral SCI. In addition to the aforementioned frameworks, different backbones and network structures including the most recent Transformers are reviewed. Simulation and real data results are presented to compare these methods.
Due to limited spatial bandwidth, one has to compromise between large field of view and high spatial resolution in both photography and microscopy. This dilemma largely hampers revealing fine details and global structures of the target scene simultaneously. Recently, a mainstream method is formed by utilizing multiple sensors for synchronous acquisition across different sub-FOVs with high resolution and stitching the patches according to the spatial position of the cameras. Various inpainting algorithms have been proposed to eliminate the intensity discontinuities, but conventional optimization methods are prone to misalignment, seaming artifacts or long processing time, and thus unable to achieve dynamic gap elimination. By taking advantage of generative adversarial networks (GANs) on image generation and padding, we propose a conditional GAN-based deep neural network for seamless gap inpainting. Specifically, a short series of displaced images are acquired to characterize the system configuration, under which we generate patch pairs with and without gap for deep network training. After supervised learning, we can achieve seamless inpainting in gap regions. To validate the proposed approach, we apply our approach on real data captured by large-scale imaging systems and demonstrate that the missing information at gaps can be retrieved successfully. We believe the proposed method holds potential for all-round observation in various fields including urban surveillance and systems biology.
We propose a novel joint compressive imaging system, which combines the merit of Single Pixel Camera (SPC) and Coded Aperture Snapshot Spectral Imaging (CASSI) system. This enables us to capture multi- or hyperspectral information with a single pixel detector. The desired 3D image cube is reconstructed by a concatenation of deep-unfolding-based algorithm and plug-and-play algorithm with deep-learning-based denoiser. We demonstrate the feasibility of the proposed system in both simulation and experiments. With advanced algorithms, the joint compressive imaging system is able to output comparable hyperspectral images with existing SD-CASSI system. Moreover, by adapting ultra-broad-spectrum photodiodes, the proposed system can be easily expanded to Near- and Mid-infrared band and thus being a low-cost approach to IR spectroscopy.
As a novel asynchronous imaging sensor, event camera features low power consumption, low temporal latency and high dynamic range, but abundant noise. In real applications, it is essential to suppress the noise in the output event sequences before successive analysis. However, the event camera is of address-event-representation (AER), and requires developing new denoising techniques rather than conventional frame-based image denoising methods. In this paper, we propose two learning-based methods for the denoising of event-based sensor measurements, i.e., convolutional denoising auto-encoder (ConvDAE) and sequence-fragment recurrent neural network (SeqRNN). The former converts the event sequence into 2D images before denoising, which is compatible with existing deep denoisers and high-level vision tasks. The latter, utilizes recurrent neural network’s advantages in dealing with time series to realize online denoising while keeping the event’s original AER representation. Experiments based on real data demonstrate the effectiveness and flexibility of the proposed methods.
The conventional high-level sensing techniques require high-fidelity images to extract visual features, which consume high software complexity or high hardware complexity. We present the single-pixel sensing (SPS) technique that performs high-level sensing directly from a small amount of coupled single-pixel measurements, without the conventional image acquisition and reconstruction process. The technique consists of three steps, including binarized light modulation, single-pixel coupled detection, and end-to-end deep-learning based decoding. The binarized modulation patterns are optimized with the decoding network by a two-step training strategy, leading to the least required measurements and optimal sensing accuracy. The effectiveness of SPS is experimentally demonstrated on the classification task of handwritten MNIST dataset, and 96% classification accuracy at ∼1kHz is achieved. The reported SPS technique is a novel framework for efficient machine intelligence, with low hardware and software complexity. Further, it maintains strong encryption.
Spectroscopy is an important tool having already been applied in various research fields, but still limited in observation of dynamic scenes. In this paper, we propose a video rate spectroscopy via Fourier-spectral-multiplexing (FSM-VRS) which exploits both spectral and spatial sparsity. Under the computational imaging scheme, the hyperspectral datacube is first modulated by several broadband bases, and then mapped into different regions in the Fourier domain. The encoded image compressed both in spectral and spatial are finally collected by a monochrome detector. Correspondingly, the reconstruction is essentially a Fourier domain extraction and spectral dimensional back projection with low computational load. The encoding and decoding process of the FSM-VRS is simple thus can be implemented in a low cost manner. The temporary resolution of the FSM-VRS is only limited by the camera frame rate. We demonstrate the high performance of our method by quantitative evaluation on simulation data and build a prototype system experimentally for further validation.
Multimodal microscopy offers high flexibilities for biomedical observation and diagnosis. Conventional multimodal approaches either use multiple cameras or a single camera spatially multiplexing different modes. The former needs expertise demanding alignment and the latter suffers from limited spatial resolution. Here, we report an alignment-free full-resolution simultaneous fluorescence and quantitative phase imaging approach using single-pixel detectors. By combining reference-free interferometry with single-pixel detection, we encode the phase and fluorescence of the sample in two detection arms at the same time. Then we employ structured illumination and the correlated measurements between the sample and the illuminations for reconstruction. The recovered fluorescence and phase images are inherently aligned thanks to single-pixel detection. To validate the proposed method, we built a proof-of-concept setup for first imaging the phase of etched glass with the depth of a few hundred nanometers and then imaging the fluorescence and phase of the quantum dot drop. This method holds great potential for multispectral fluorescence microscopy with additional single-pixel detectors or a spectrometer. Besides, this cost-efficient multimodal system might find broad applications in biomedical science and neuroscience.
Existing multispectral imagers mostly use 2D array sensors to separately measure 2D data slices in a 3D spatialspectral data cube. They suffer from low photon efficiency, limited spectral range, and high cost. To address these issues, we propose to conduct multispectral imaging using a photodiode, to take full advantage of its high sensitivity, wide spectral range, low cost, and small size. Specifically, utilizing the photodiode’s fast response, a scene’s 3D spatial-spectral information is sinusoidally multiplexed into a dense 1D measurement sequence, and then demultiplexed computationally under the single-pixel imaging scheme. A proof-of-concept setup is built to capture multispectral data of 256 pixels × 256 pixels × 10 wavelength bands ranging from 450 nm to 650 nm. The imaging scheme holds great potentials for various biological applications such as fluorescence microscopy and endoscopy.
Fourier ptychographic microscopy (FPM) is a recently developed technique stitching low-resolution images in Fourier domain to realize wide-field high-resolution imaging. However, the time-consuming process of image acquisition greatly narrows its applications in dynamic imaging. We report a wavelength multiplexing strategy to speed up the acquisition process of FPM several folds. A proof-of-concept system is built to verify its feasibility. Distinguished from many current multiplexing methods in Fourier domain, we explore the potential of high-speed FPM in spectral domain. Compatible with most existing FPM methods, our strategy provides an approach to high-speed gigapixel microscopy. Several experimental results are also presented to validate the strategy.
Conventional multispectral imaging methods detect photons of a 3D hyperspectral data cube separately either in the spatial or spectral dimension using array detectors, and are thus photon inefficient and spectrum range limited. Besides, they are usually bulky and highly expensive. To address these issues, this paper presents single-pixel multispectral imaging techniques, which are of high sensitivity, wide spectrum range, low cost and light weight. Two mechanisms are proposed, and experimental validation are also reported.
Optical coherence tomography (OCT) is an important interferometric diagnostic technique, which provides cross-sectional views of biological tissues’ subsurface microstructures. However, the imaging quality of high-speed OCT is limited by the large speckle noise. To address this problem, we propose a multiframe algorithmic method to denoise OCT volume. Mathematically, we build an optimization model which forces the temporally registered frames to be low-rank and the gradient in each frame to be sparse, under the constraints from logarithmic image formation and nonuniform noise variance. In addition, a convex optimization algorithm based on the augmented Lagrangian method is derived to solve the above model. The results reveal that our approach outperforms the other methods in terms of both speckle noise suppression and crucial detail preservation.
The hyper-spectrum data exhibits the structure, materials, and semantic meaning of a nature scene and its fast acquisition is of great importance due to its potential for parse these properties of dynamic scenes. Targeting for high speed hyperspectrum imaging of a nature scene, this paper proposes to capture the coded hyper-spectrum reflectance of a nature scene using low cost hardware and reconstruct the latent data using a corresponding decoding algorithm. Except for a wide spectrum light source, the imaging system includes mainly a commercially available projector color wheel and a high speed camera, which work at their own constant periods and are self-synchronized by our algorithm. The introduced light source and color wheel cost less than 50 dollars and makes the proposed approach widely available. The results on the data captured by our prototype system show that, the proposed approach can reconstruct the high precision hyper-spectrum data at real time.
Non-uniform image blur caused by camera shake or lens aberration is a common degradation in practical capture. Different from the uniform blur, non-uniform one is hard to deal with for its extremely high computation complexity as the blur model computation cannot be accelerated by Fast Fourier Transform (FFT). We propose to compute the most computational consuming operation, i.e. blur model calculation, by an optical computing system to realize fast and accurate non-uniform image deblur. A prototype system composed by a projector-camera system as well as a high dimensional motion platform (for motion blur) or original camera lens (for optics aberrations) is implemented. Our method is applied on a series of experiments, either on synthetic or real captured images, to verify its effectiveness and efficient.
Capturing four dimensional light field data sequentially using a coded aperture camera is an effective approach but
suffers from low signal noise ratio. Although multiplexing can help raise the acquisition quality, noise is still a big issue
especially for fast acquisition. To address this problem, this paper proposes a noise robust light field reconstruction
method. Firstly, scene dependent noise model is studied and incorporated into the light field reconstruction framework.
Then, we derive an optimization algorithm for the final reconstruction. We build a prototype by hacking an off-the-shelf
camera for data capturing and prove the concept. The effectiveness of this method is validated with experiments on the
real captured data.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.