2.1

The feasibility of implementing neural network

As is introduced in Section 1, the computational costs of double phase retrieval method is quite expensive for real-time imaging. We build a neural network with nonlinear structures to approximate operator g : b → x so that x can be estimated directly from b.

In the following we discuss the existence of the operator g. Based on the inverse function theorem, the operator f: x → b must be injective or g cannot be existed. But in phase retrieval problem, f is not a single valued operator usually. When x ∈ ℝⁿ, f(x) = f(–x), while x ∈ ℂⁿ, f(x) = f(cx), |c|= 1. As a result, x is defined up to a global phase factor. In this paper, we concern with the map f : ℝⁿ / {±1} → ℝ^m when x is real, and f : ℂⁿ / 𝕋 → ℝ^m when x is complex (where 𝕋 is the complex unit circle). When training the network, it is necessary to sift the whole dataset so that every b⁽ⁱ⁾, i = 1,…, m is not equal. Then Theorem 2.2 and Theorem 3.3¹⁶ guarantee the injective of f.

In the multiple-scattering media imaging, media which may be frost glass or painted wall. Under this condition, rows of transmission matrix A in traditional analysis are often assumed to be the Gaussian or sub-Gaussian which generic vectors. Consequently, if m is large enough, f is injective, the existence of g can also be guaranteed. Furthermore, g can also be approximated by neural network g_Θ :

Theorem 2.1. If the inverse operator g of f : ℝⁿ / {±} → ℝ^m: (or f : ℂⁿ/𝕋 → ℝ^m) in phase retrieval problem exists, then g can be approximated by neural network g_Θ (where Θ is the parameters of the neural network) with any desired degree of accuracy.

2.2

Neural network training

When the training set is in ℝⁿ / {±1} or ℂⁿ / 𝕋, we can train a neural network to approximate this inverse operator g. In this paper, a neural network called TCNN is built to train g_Θ as an approximation of g. The mathematical model for neural network training is:

where g_Θ is the neural network with hierarchical structures so that nonlinear information can be mined from the dataset. The encoding and decoding diagram of TCNN is shown in Figure 1, and all the blocks are shown in Figure 2.

Figure 1.

Encoding and decoding part.

Figure 2.

The details of blocks in TCNN.

·Encoding part

In the encoding part, the multiple flow structure is utilized in g_Θ which can learn the features of input in different scales.

The input of g_Θ is decimated by the downsample Block which is constructed by one convolution layer with batch normalization, Relu activation function, and one maxpooling layer, which can be modeled as below:

where * is the convolution, Ꞷ^l is the convolutional kernel, l indicates the kernel is in the l-th path. The size of those kernels is 1×l×16. The maxpooling layer here decreases the dimension of the input. Specifically, b⁽ⁱ⁾, i = 1,…,k are downsampled by ×2, ×4, ×8 respectively. Then, the four different tensors are successively passed through five residue blocks, each block contains two convolution layers with batch normalization and a shortcut from the input to the output. The shortcut can accelerate the convergence of TCNN. After processed by the five residue blocks, the high-order features in different scales can be learned.

·Decoding part

In the decoding part, those features created above will generate four different tensors keeping the same width and length with the b⁽ⁱ⁾, i = 1,…,k. So there will be 3, 2, 1 upsampling blocks for each tensor respectively. In each upsample block, there will be one convolutional layer with batch normalization and one deconvolutional layer. The deconvolution in this paper adopts the way of upsampling¹¹ for super-resolution. This method can alleviate the effect of zero padding by traditional deconvolution besides fully utilizing the information in the network.

·Transforming block

After being handled by those upsample blocks, tensors will be fused and passed through the transformation block instituted by one convolutional layer and transformation layer. The transformation layer is actually a full connection layer which acts as a linear transformation between the transform domain and object domain. In the test, to alleviate the influence of over-fit, the units in the transformation layer are randomly neglected.

Because of large quantities of the dataset, mini-batch gradient descent method is implemented to update Θ in (2), which is actually a non-convex optimization, it is hard to guarantee the algorithm converges to the global optimum. But in practice, some efficient gradient based methods outperform when training neural network. In this paper, we utilized ADAM method¹⁴, which can accelerate the decreasing of the loss function besides it can also adaptively calibrate the first moment, second moment and the learning rate. The skeleton of parameter updating is shown in Algorithm 1.

During the training procedure, the learning rate μ decays with a geometrical rate so that the evaluation cannot vibrate violently in the final stage of iteration. In the next section, the results of numerical tests will be given to demonstrate the efficiency of the TCNN.

3.1

Datasets of the numerical tests

The experiment setup is made by Rice University¹⁵. Specifically, a laser beam that is spatially filtered and collimated (λ = 632.8nm) illuminates a spatial light modulator (SLM) from Holoeye. This reflective type display (LC2012), which is equipped with 1024×768 resolution and 36μm square pixels, modulates the phase of the laser beam. Then the lens L (f = 150mm) focus the laser beam on the scattering medium. The experiment uses a microscope objective (Newport, X10, NA: 0.25) to image the SLM calibration pattern to the sensor (Point Grey Grasshopper 2 with pixel size 6.45μm). As the phase only SLM is 8 bit, it can modulate the wavefront by an element of . In the test, the phase modulation is restricted to {0,π}, which means the value of the x is {–1,1}. For the amplitude-only SLM, the source pixel is set to be either completely 0 or completely 1. Then the value of x is {0,1}. By constantly modulate the SLM, values of SLM and their speckle patterns can be recorded to estimate transmission matrix.

The dataset can be downloaded from https://rice.app.box.com/v/TransmissionMatrices. Two types of data are utilized in the experiment. For amplitude-only SLM, the size of the image is 16×16. For the phase-only SLM, the size of the image is 40×40. Experiments are implemented by a desktop with GPU NVIDIA 1080 and CUDA 9.0.

3.2

Experimental results

Some hyper parameters of in this test can be found in Table 1. TCNN is built on the framework of tensorflow. The learning rate μ is 10^–3 at initial and decays after each epoch with the factor 0.85. The total number of the epoch in the training procedure is 100. To preprocess the data in this experiment, we firstly sift the data so that every image and corresponding speckle pattern is unique. Then we train TCNN by two different sizes of data.

Table 1.

Some hyper parameters in TCNN.

We compare TCNN with GS¹⁷, WF¹⁸, PhaseLift¹⁹, Prvamp¹⁵. TCNN can directly recover images from those speckle patterns without estimating the transmission matrix. For other methods, we utilize the transmission matrix A estimated by Prvamp to apply phase retrieval algorithm. For fair, those algorithms are all accelerated by the GPU. Several examples of recovered images are shown in Figure 3.

Figure 3.

The reconstruction for amplitude-only SLM with image size (a) 16×16 and (b) 40×40.

From Figure 3, we can observe that pictures recovered by TCNN are competitive with state-of-arts. In fact, it is quite challenging for all these methods recovering high quality images from speckle patterns, since multiple scattering media deteriorates the light besides the phase information get lost by CCD. Moreover, the noise and system error also exist in the practical experiment. But Table 2 shows the time cost by TCNN is far less than other methods. It can save the time 10 to 100 folds. The advantage becomes clearer when 64×64 which demonstrates the possibility of TCNN to realize real time imaging. Moreover, TCNN only performs an end-to-end recovery but other methods must estimate the transformation matrix in advance then getting the evaluation by phase retrieval. Thus, the experiments fully illustrate the power of TCNN in recovering images from the intensity of the speckle pattern.

Table 2.

Time cost per image by different methods (“—” means TCNN don’t need iterations).

During the training step, the curves of training error and validation error of amplitude only SLM 16 ×16 are depicted in Figure 4. Considering the large quantities of the training set, training error is the Mean Square Error (MSE) of one of the inputs in training set selected randomly, as a result, there are some oscillations in training error curve. The validation error is the mean MSE of the validation set. From Figure 4, we can find the solution quickly converges to the local optimum after 20 epochs, there is little fluctuation for both errors. Besides the validation error gets close to the training error which fully demonstrates network relieves from over-fit.

Figure 4.

The training and validation errors.

The core of TCNN is to utilize two sections g_Θ1 and g_Θ2 to approximate the inverse function g which can simulate the imaging procedure in multiple scattering media, namely,

For g_Θ1, features of intensity of speckle patterns b are learned from different scales which can be seen from Figure 5. We can see that the details from different scales of speckle pattern are learned (the features in Figure 5 are resized to keep the same shape). Then, deconvolution procedures decode those features into a new element g_Θ1(b) in the transform domain so that it can be approximated to x by transforming layer g_Θ2.

Figure 5.

The features learned in TCNN.