2.1.

Ethics Statement

All subject recruitment and imaging took place at the Eye Care Centre of Vancouver General Hospital. The project protocol was approved by the Research Ethics Boards at the University of British Columbia and Vancouver General Hospital, and the experiment was performed in accordance with the tenets of the Declaration of Helsinki. Written informed consent was obtained by all subjects.

2.2.

Speckle Variance Optical Coherence Tomography Imaging

Speckle variance OCT images of the foveal region in 12 eyes from 6 healthy volunteers aged $36.8\pm 7.1$ years were acquired using a graphics processing unit-accelerated OCT-A clinical prototype.²⁴ In total, 80 images were acquired. Briefly, the OCT system uses a 1060-nm swept source (Axsun Inc.) with 100-kHz A-scan rate and a full-width half-maximum bandwidth of 61.5 nm, which corresponds to a coherence length of $\sim 6\mu \mathrm{m}$ in tissue. For the speckle variance calculation, three repeat acquisitions were obtained at each B-scan location. The scan area was sampled in a $300\times 300(\times 3)$ grid with a $\sim 1\times 1\mathrm{mm}$ field of view in 3.15 s. Images were acquired either directly superiorly, nasally, inferiorly, or temporally from the foveal avascular zone. Processing of the OCT intensity image data and en face visualization of the retinal microvasculature was performed in real time using our open source code.^25,26

2.3.

Manual Segmentation

For comparison, two raters segmented OCT-A images using a Wacom Intuos 4 tablet and GNU image manipulation program. For the cross-validation and training, Rater A segmented all 80 OCT-A images. For the repeatability analysis, 10 images were used and segmented by both rater A and rater B. Rater A segmented each image twice for intrarater agreement, while Rater B segmented each image once for interrater agreement.

2.4.

Deep Neural Network Architecture

The automated segmentation of the blood vessels in the OCT-A images was performed by classifying each pixel into either the vessel or the nonvessel class using deep convolutional neural networks. Convolutional and max pooling layers are used as hierarchical feature extractors, which map raw pixel intensities into a feature vector, which is then classified using fully connected layers.

The convolutional layers in our algorithm are made of a sequence of square filters, which perform a two-dimensional convolution with the input image. To calculate the output of each map, convolutional responses are summed and passed through a nonlinear activation function. The nonlinear activation function used in this paper is a rectifying linear unit.

Max pooling layers generate their output by taking the maximum activation over nonoverlapping square regions. These layers do not have adjustable parameters and their size is fixed. By taking the maximum value of the activation function, the most prominent features are selected from the input image.

After six stages of varied convolutional and max pooling layers, a dropout layer was inserted, which can prevent network over-fitting and provide a way of combining an exponentially increasing number of different neural networks in an efficient manner.²⁷ Then, two fully connected layers are used to classify the feature vector generated by the previous layers. The final fully connected layer contains two neurons where one neuron represents the vessel and other the nonvessel class. The network architecture used in this paper is very similar to the network architecture first used in Ref. 20. An overview is presented in Table 1 and graphically in Fig. 1.

Table 1

Network layers architecture.

Fig. 1

Graphical representation of the network structure. Input image is a $61\times 61$ patch cut at each image point. Two output neurons represent the probabilities of blood vessels and background at the central pixel in the input image.

2.5.

Network Training Methods

To train our network, original OCT-A images and the corresponding manual segmentations were used as inputs. Each training example consists of a $61\times 61\text{pixel}$ square window around the training pixel. Missing pixels in windows at the image border were set to zero. To have a balanced training set, an equal number of vessel and nonvessel pixels were extracted from each image. If the number of vessel pixels was larger than the number of nonvessel pixels in an image, then all nonvessel pixels were selected for the training set and an equal number of vessel pixels were randomly selected from the pool of vessel pixels. Similarly, if the number of nonvessel pixels was larger than the number of vessel pixels in an image, then all vessel pixels were selected for the training set and an equal number of nonvessel pixels were randomly selected from the pool of nonvessel pixels.

2.6.

Network Segmentation Methods

The trained network was then used to segment the original OCT-A images. First, a square window of the same size used for the training purposes was extracted around each pixel of the test images. A forward pass using all test image pixels was performed using the trained network, and each pixel was assigned a grayscale value, with higher values representing higher confidence of the pixel being a vessel pixel. These pixel values were aggregated into the output grayscale images, and median filtering with a small $3\times 3$ window was performed in order to decrease the noise level in the image.

2.7.

Cross-Validation Methods

Three-fold cross-validation on all images manually segmented by Rater A was performed. All 80 original images were randomly divided into three sets. Images from two of the sets were used to train the network, and images from the remaining set were used to test the network. This procedure was repeated three times with a different test set each time. Each set of the cross-validation was evaluated on a separate computer in order to decrease the total training and testing time. Each computer had a recent generation NVIDIA graphics card, which decreased computation time. The Caffe deep learning toolkit²⁸ was used to efficiently use the processing power of the graphics card for computation of convolutional neural network parameters. Using parallel processing, all three sets were used to train the proposed neural network in approximately 30 h. Segmentation of a single image using the trained network took $\sim 2\min$.

3.1.

Performance Evaluation

The segmentation performance was evaluated by pixel-wise comparison of the manually segmented images and the thresholded binary output of the neural network using varying thresholds. The number of true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN) were calculated using pixel-wise comparison between a reference manual segmentation and a target, which was either another manual segmentation, or the output of our automated method. In our context, a pixel is considered as TP if it is marked as a blood vessel in both the reference manual segmentation and in the target. A pixel is considered as FN if it is marked as blood vessel in the manual segmentation but missed by the target. A pixel is considered as FP if it is marked as vessel by our method but it is not marked as blood vessel in the target. A pixel is considered as TN if it is not marked as blood vessel in both the manual segmentation and in the target. Using the TP, FP, FN, and TN numbers we can calculate the accuracy: $(\mathrm{TP}+\mathrm{TN})/(\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN})$, sensitivity: $\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})$, specificity: $\mathrm{TN}/(\mathrm{TN}+\mathrm{FP})$ and positive predictive value (PPV): $\mathrm{TP}/(\mathrm{TP}+\mathrm{FP})$ of the segmentation.

Using the PPV and sensitivity we can calculate the $F_1$ measure using Eq. (1).

All of these measures can be calculated on individual images but can also be calculated for the whole dataset. In Fig. 2, the dotted blue line shows the accuracy for all pixels in the dataset against the threshold value used to binarize the output of the network. The accuracy of blood vessel detection increases from the threshold value at 0, peaks at 0.83 with threshold value of 0.78, and then begins to decline. It is important to note that similar results are obtained in a wide range of thresholds, which indicates that the performance is not sensitive to the threshold chosen.

Fig. 2

Accuracy of the segmentation using the DNN. The blue dotted line is the accuracy for all pixels in the dataset and the red line is the accuracy using only the images used for assessing the intrarater and interrater accuracies. The black and cyan lines are the corresponding, intrarater and interrater accuracies, respectively. As the accuracy for several thresholds are above that of the intrarater and interrater accuracies, we can say that the performance is not sensitive to the chosen threshold.

In Fig. 3, the accuracy for each image was calculated and averaged over all images. One standard deviation below the mean values is marked with a green dotted line and one standard deviation above the mean values is marked with a blue dotted line. Qualitatively, the deviation of accuracies is reasonably small for different thresholds, with the maximum mean accuracy of $0.83\pm 0.02$ at the threshold value of 0.76, signifying that the performance of the method is consistent over the whole dataset. The accuracy of the deeper capillary network [inner nuclear layer (INL) to outer plexiform layer] is 0.8247 while the accuracy of the superficial capillary networks (inner limiting membrane to INL) is 0.8389; the lower accuracy of the deeper layers is likely due to projection artifact from the superficial vascular layers.

Fig. 3

Mean accuracy of the segmentation using the DNN. The red line is the mean accuracy of the segmentation for all possible threshold values, and the blue dotted line and green dashed line are one standard deviation above and below the mean accuracy, respectively. The small deviation of accuracies signifies a consistent performance of the whole dataset.

Using the sensitivity and specificity measurements over the range of thresholds we can plot the receiver operator characteristic (ROC) for our method, as shown in Fig. 4 with blue dots. The sensitivity and specificity were calculated using all pixels from the dataset.

Fig. 4

ROC curves of the segmentation using the DNN. The blue dotted line is the ROC curve for all pixels in the dataset and the red line is the ROC curve for the images used for assessing the intrarater and interrater accuracies. The black cross and cyan dots are the corresponding, intrarater and interrater points, respectively.

In Fig. 5, the $F_1$ measure was calculated for the machine output using all pixels from the dataset and shown with blue dots.

Fig. 5

$F_1$ measure of the segmentation using the DNN. The blue dotted line is the $F_1$ measure for all pixels in the dataset and the red line is the $F_1$ measure of the images used for assessing the intrarater and interrater accuracies. The straight black and dotted cyan lines are the corresponding, intrarater and interrater $F_1$ measures, respectively. The results from the automated DNN method are better than the manual segmentation results for a large range of thresholds, again showing the performance is not sensitive to the threshold chosen.

3.2.

Intrarater and Interrater Agreement

As described in Sec. 2.3 among the 80 images segmented by Rater A, 10 images were additionally segmented a second time by Rater A, and also by Rater B for assessing the intra- and interrater agreement. For convenience, we used the accuracy measures discussed above and the original segmentation by Rater A (Rater A1) as the ground-truth in order to assess its agreement with (1) the repeat segmentation of Rater A (Rater A2), (2) Rater B, and (3) the network. The machine segmentation accuracy results of (3) were obtained as part of the threefold validation in Sec. 3.1. The results are shown in Fig. 2 in dotted cyan, solid black, and solid red lines, respectively. The intra- and interrater accuracies for the manual raters are plotted as lines because they are independent of the threshold used for the machine based segmentation. From the Fig. 2, the intrarater, interrater, and machine-rater accuracies are comparable, suggesting that the automated segmentation is comparable to that of a human rater. As it was expected, the accuracy of the repeated segmentation is better than the accuracy of the second rater but the difference is small.

In Fig. 4, the ROC curve of the automated segmentation is compared with Rater A1 (solid red line). In the same figure, the cyan star represents the sensitivity and specificity pair for Rater A2 compared with Rater A1 and the black cross represents the sensitivity and specificity pair for Rater B compared with Rater A1. The ROC curve was created by plotting the sensitivity against the false-positive rate (1-specificity) at various thresholds to depict relative trade-offs between true positives and false positives. A completely random result would be represented by a diagonal line. As seen in Fig. 4, the results from the automated DNN method are better than the manual segmentation results and well above the random result.

In Fig. 5, we can see the $F_1$ measure curve for the machine output marked with solid red curve, and the $F_1$ measures for Rater A2 (dotted straight cyan line) and for Rater B (solid straight black line). The $F_1$ measure depicts the trade-off between precision and recall with each variable weighted equally. As such a higher $F_1$-measure has a better balance between precision and recall. As seen in Fig. 5, there is a wide range of thresholds in which the balance between precision and recall is higher than the manual raters.

3.3.

Capillary Density

Capillary density (CD) is a clinical measure of quantifying retinal capillaries present in the OCT-A images. After segmentation of the vessels, CD can be calculated as the number of pixels in the segmented areas. Using the same 10 images from Sec. 2.3, we obtained the CD values from the segmentations by Rater A1, Rater A2, Rater B, and the network, and calculated the mean capillary density in order to evaluate the intrarater, interrater, and machine-to-rater repeatability of the CD measures. The result is presented in Table 2.

Table 2

Mean capillary density comparison.

A paired-samples $t$-test was conducted to compare the capillary density of manual and automated segmentations. There was no significant difference in the scores for either of the manual raters or the machine.