Purpose

The objective of this study is to review the accuracy of an augmented reality navigational guidance system designed to facilitate improved visualization, guidance, and accuracy during percutaneous needle-based procedures including biopsies and ablations.

Approach

Using the HoloLens 2, the system registers and projects 3D CT-based models of segmented anatomy along with live ultrasound, fused with electromagnetically tracked instruments including ultrasound probes and needles, giving the operator comprehensive stereoscopic visualization for intraoperative planning and navigation during procedures.

Tracked needles were guided to targets implanted in a cadaveric model using the system. Image fusion registration error, the multimodality error measured as the post-registration distance between a corresponding point measured in the stereoscopic CT and tracked ultrasound coordinate systems, and target registration error, the Euclidean distance between needle tip and target after needle placement, were measured as registration and targeting accuracy metrics. A t-distribution was used for statistical analysis.

Results

Three operators performed 36 total needle passes, 18 to measure image fusion registration error and 18 to measure target registration error on four targets. The average depth of each needle pass was 8.4 cm from skin to target center. Mean IFRE was 4.4 mm (H0: μ=5 mm, P<0.05). Mean TRE was 2.3 mm (H0: μ=5 mm, P<0.00001).

Conclusions

The study demonstrated high registration and targeting accuracy of this AR navigational guidance system in percutaneous, needle-based procedures. This suggests the ability to facilitate improved clinical performance in percutaneous procedures such as ablations and biopsies.

Purpose

Virtual reality (VR) technology has emerged as a promising tool for physicians, offering the ability to assess anatomical data in 3D with visuospatial interaction qualities. The last decade has witnessed a remarkable increase in the number of studies focusing on the application of VR to assess patient-specific image data. This systematic review aims to provide an up-to-date overview of the latest research on VR in the field of surgical planning.

Approach

A comprehensive literature search was conducted based on the preferred reporting items for systematic reviews and meta-analyses covering the period from April 1, 2021 to May 10, 2023. It includes research articles reporting on preoperative surgical planning using patient-specific medical images in virtual reality using head-mounted displays. The review summarizes the current state of research in this field, identifying key findings, technologies, study designs, methods, and potential directions for future research.

Results

The selected studies show a positive impact on surgical decision-making and anatomy understanding compared to other visualization modalities. A substantial number of studies are reporting anecdotal evidence and case-specific outcomes. Notably, surgical planning using VR led to more frequent changes in surgical plans compared to planning with other visualization methods when surgeons reassessed their initial plans. VR demonstrated benefits in reducing planning time and improving spatial localization of pathologies.

Conclusions

Results show that the application of VR for surgical planning is still in an experimental stage but is gradually advancing toward clinical use. The diverse study designs, methodologies, and varying reporting hinder a comprehensive analysis. Some findings lack statistical evidence and rely on subjective assumptions. To strengthen evaluation, future research should focus on refining study designs, improving technical reporting, defining visual and technical proficiency requirements, and enhancing VR software usability and design. Addressing these areas could pave the way for an effective implementation of VR in clinical settings.

Significance

Conventional ultrasound-guided vascular access procedures are challenging due to the need for anatomical understanding, precise needle manipulation, and hand–eye coordination. Recently, augmented reality (AR)-based guidance has emerged as an aid to improve procedural efficiency and potential outcomes. However, its application in pediatric vascular access has not been comprehensively evaluated.

Aim

We developed an AR ultrasound application, HoloUS, using the Microsoft HoloLens 2 to display live ultrasound images directly in the proceduralist’s field of view. We presented our evaluation of the effect of using the Microsoft HoloLens 2 for point-of-care ultrasound (POCUS)-guided vascular access in 30 pediatric patients.

Approach

A custom software module was developed on a tablet capable of capturing the moving ultrasound image from any ultrasound machine’s screen. The captured image was compressed and sent to the HoloLens 2 via a hotspot without needing Internet access. On the HoloLens 2, we developed a custom software module to receive, decompress, and display the live ultrasound image. Hand gesture and voice command features were implemented for the user to reposition, resize, and change the gain and the contrast of the image. We evaluated 30 (15 successful control and 12 successful interventional) cases completed in a single-center, prospective, randomized study.

Results

The mean overall rendering latency and the rendering frame rate of the HoloUS application were 139.30 ms (σ=32.02 ms) and 30 frames per second, respectively. The average procedure completion time was 17.3% shorter using AR guidance. The numbers of puncture attempts and needle redirections were similar between the two groups, and the number of head adjustments was minimal in the interventional group.

Conclusion

We presented our evaluation of the results from the first study using the Microsoft HoloLens 2 that investigates AR-based POCUS-guided vascular access in pediatric patients. Our evaluation confirmed clinical feasibility and potential improvement in procedural efficiency.

Purpose

Visualization of medical images on a virtual reality (VR) head-mounted display (HMD) requires binocular fusion of a stereoscopic pair of graphical views. However, current image quality assessment on VR HMDs for medical applications has been primarily limited to time-consuming monocular optical bench measurement on a single eyepiece.

Approach

As an alternative to optical bench measurement to quantify the image quality on VR HMDs, we developed a WebXR test platform to perform contrast perceptual experiments that can be used for binocular image quality assessment. We obtained monocular and binocular contrast sensitivity responses (CSRs) from participants on a Meta Quest 2 VR HMD using varied interpupillary distance (IPD) configurations.

Results

The perceptual result shows that contrast perception on VR HMDs is primarily affected by optical aberration of the VR HMD. As a result, monocular CSR degrades at a high spatial frequency greater than 4 cycles per degree when gazing at the periphery of the display field of view, especially for mismatched IPD settings consistent with optical bench measurements. On the contrary, binocular contrast perception is dominated by the monocular view with superior image quality measured by the contrast.

Conclusions

We developed a test platform to investigate monocular and binocular contrast perception by performing perceptual experiments. The test method can be used to evaluate monocular and/or binocular image quality on VR HMDs for potential medical applications without extensive optical bench measurements.

Purpose

Virtual reality (VR) and augmented reality (AR) have led to significant advancements in cardiac preoperative planning, shaping the world in profound ways. A noticeable gap exists in the availability of a comprehensive multi-user, multi-device mixed reality application that can be used in a multidisciplinary team meeting.

Approach

A multi-user, multi-device mixed reality application was developed, supporting AR and VR implementations. Technical validation involved a standardized testing protocol and comparison of AR and VR measurements regarding absolute error and time. Preclinical validation engaged experts in interventional cardiology, evaluating the clinical applicability prior to clinical validation. Clinical validation included patient-specific measurements for five patients in VR compared with standard computed tomography (CT) for preoperative planning. Questionnaires were used at all stages for subjective evaluation.

Results

Technical validation, including 106 size measurements, demonstrated an absolute median error of 0.69 mm (0.25 to 1.18 mm) compared with ground truth. The time to complete the entire task was 892±407 s on average, with VR measurements being faster than AR (804±483 versus 957±257 s, P=0.045). On clinical validation of five preoperative patients, there was no statistically significant difference between paired CT and VR measurements (0.58 [95% CI, −1.58 to 2.74], P=0.586). Questionnaires showcased unanimous agreement on the user-friendly nature, effectiveness, and clinical value.

Conclusions

The mixed reality application, validated through technical, preclinical, and clinical assessments, demonstrates precision and user-friendliness. Further research of our application is needed to validate the generalizability and impact on patient outcomes.

Purpose

eXtended Reality (XR) technology, including virtual reality (VR), augmented reality (AR), and mixed reality (MR), is a growing field in healthcare. Each modality offers unique benefits and drawbacks for medical education, simulation, and clinical care. We review current studies to understand how XR technology uses medical imaging to enhance surgical diagnostics, planning, and performance. We also highlight current limitations and future directions.

Approach

We reviewed the literature on immersive XR technologies for surgical planning and intraoperative augmentation, excluding studies on telemedicine and 2D video–based training. We cited publications highlighting XR’s advantages and limitations in these categories.

Results

A review of 556 papers on XR for medical imaging in surgery yielded 155 relevant papers reviewed utilizing the aid of chatGPT. XR technology may improve procedural times, reduce errors, and enhance surgical workflows. It aids in preoperative planning, surgical navigation, and real-time data integration, improving surgeon ergonomics and enabling remote collaboration. However, adoption faces challenges such as high costs, infrastructure needs, and regulatory hurdles. Despite these, XR shows significant potential in advancing surgical care.

Conclusions

Immersive technologies in healthcare enhance visualization and understanding of medical conditions, promising better patient outcomes and innovative treatments but face adoption challenges such as cost, technological constraints, and regulatory hurdles. Addressing these requires strategic collaborations and improvements in image quality, hardware, integration, and training.

Purpose

Deformable image registration establishes non-linear spatial correspondences between fixed and moving images. Deep learning–based deformable registration methods have been widely studied in recent years due to their speed advantage over traditional algorithms as well as their better accuracy. Most existing deep learning–based methods require neural networks to encode location information in their feature maps and predict displacement or deformation fields through convolutional or fully connected layers from these high-dimensional feature maps. We present vector field attention (VFA), a novel framework that enhances the efficiency of the existing network design by enabling direct retrieval of location correspondences.

Approach

VFA uses neural networks to extract multi-resolution feature maps from the fixed and moving images and then retrieves pixel-level correspondences based on feature similarity. The retrieval is achieved with a novel attention module without the need for learnable parameters. VFA is trained end-to-end in either a supervised or unsupervised manner.

Results

We evaluated VFA for intra- and inter-modality registration and unsupervised and semi-supervised registration using public datasets as well as the Learn2Reg challenge. VFA demonstrated comparable or superior registration accuracy compared with several state-of-the-art methods.

Conclusions

VFA offers a novel approach to deformable image registration by directly retrieving spatial correspondences from feature maps, leading to improved performance in registration tasks. It holds potential for broader applications.

Purpose

Pancreatic ductal adenocarcinoma is forecast to become the second most significant cause of cancer mortality as the number of patients with cancer in the main duct of the pancreas grows, and measurement of the pancreatic duct diameter from medical images has been identified as relevant for its early diagnosis.

Approach

We propose an automated pancreatic duct centerline tracing method from computed tomography (CT) images that is based on deep reinforcement learning, which employs an artificial agent to interact with the environment and calculates rewards by combining the distances from the target and the centerline. A deep neural network is implemented to forecast step-wise values for each potential action. With the help of this mechanism, the agent can probe along the pancreatic duct centerline using the best possible navigational path. To enhance the tracing accuracy, we employ landmark-based registration, which enables the generation of a probability map of the pancreatic duct. Subsequently, we utilize a gradient-based method on the registered data to extract a probability map specifically indicating the centerline of the pancreatic duct.

Results

Three datasets with a total of 115 CT images were used to evaluate the proposed method. Using image hold-out from the first two datasets, the method performance was 2.0, 4.0, and 2.1 mm measured in terms of the mean detection error, Hausdorff distance (HD), and root mean squared error (RMSE), respectively. Using the first two datasets for training and the third one for testing, the method accuracy was 2.2, 4.9, and 2.6 mm measured in terms of the mean detection error, HD, and RMSE, respectively.

Conclusions

We present an algorithm for automated pancreatic duct centerline tracing using deep reinforcement learning. We observe that validation on an external dataset confirms the potential for practical utilization of the presented method.

Purpose

Self-supervised pre-training can reduce the amount of labeled training data needed by pre-learning fundamental visual characteristics of the medical imaging data. We investigate several self-supervised training strategies for chest computed tomography exams and their effects on downstream applications.

Approach

We benchmark five well-known self-supervision strategies (masked image region prediction, next slice prediction, rotation prediction, flip prediction, and denoising) on 15 M chest computed tomography (CT) slices collected from four sites of the Mayo Clinic enterprise, United States. These models were evaluated for two downstream tasks on public datasets: pulmonary embolism (PE) detection (classification) and lung nodule segmentation. Image embeddings generated by these models were also evaluated for prediction of patient age, race, and gender to study inherent biases in models’ understanding of chest CT exams.

Results

The use of pre-training weights especially masked region prediction-based weights, improved performance, and reduced computational effort needed for downstream tasks compared with task-specific state-of-the-art (SOTA) models. Performance improvement for PE detection was observed for training dataset sizes as large as ∼380 K with a maximum gain of 5% over SOTA. The segmentation model initialized with pre-training weights learned twice as fast as the randomly initialized model. While gender and age predictors built using self-supervised training weights showed no performance improvement over randomly initialized predictors, the race predictor experienced a 10% performance boost when using self-supervised training weights.

Conclusion

We released self-supervised models and weights under an open-source academic license. These models can then be fine-tuned with limited task-specific annotated data for a variety of downstream imaging tasks, thus accelerating research in biomedical imaging informatics.

Purpose

Eye morphology varies significantly across the population, especially for the orbit and optic nerve. These variations limit the feasibility and robustness of generalizing population-wise features of eye organs to an unbiased spatial reference.

Approach

To tackle these limitations, we propose a process for creating high-resolution unbiased eye atlases. First, to restore spatial details from scans with a low through-plane resolution compared with a high in-plane resolution, we apply a deep learning-based super-resolution algorithm. Then, we generate an initial unbiased reference with an iterative metric-based registration using a small portion of subject scans. We register the remaining scans to this template and refine the template using an unsupervised deep probabilistic approach that generates a more expansive deformation field to enhance the organ boundary alignment. We demonstrate this framework using magnetic resonance images across four different tissue contrasts, generating four atlases in separate spatial alignments.

Results

When refining the template with sufficient subjects, we find a significant improvement using the Wilcoxon signed-rank test in the average Dice score across four labeled regions compared with a standard registration framework consisting of rigid, affine, and deformable transformations. These results highlight the effective alignment of eye organs and boundaries using our proposed process.

Conclusions

By combining super-resolution preprocessing and deep probabilistic models, we address the challenge of generating an eye atlas to serve as a standardized reference across a largely variable population.

Purpose

The BRCA1-associated protein 1 (BAP1) gene is of great interest because somatic (BAP1) mutations are the most common alteration associated with pleural mesothelioma (PM). Further, germline mutation of the BAP1 gene has been linked to the development of PM. This study aimed to explore the potential of radiomics on computed tomography scans to identify somatic BAP1 gene mutations and assess the feasibility of radiomics in future research in identifying germline mutations.

Approach

A cohort of 149 patients with PM and known somatic BAP1 mutation status was collected, and a previously published deep learning model was used to first automatically segment the tumor, followed by radiologist modifications. Image preprocessing was performed, and texture features were extracted from the segmented tumor regions. The top features were selected and used to train 18 separate machine learning models using leave-one-out cross-validation (LOOCV). The performance of the models in distinguishing between BAP1-mutated (BAP1+) and BAP1 wild-type (BAP1−) tumors was evaluated using the receiver operating characteristic area under the curve (ROC AUC).

Results

A decision tree classifier achieved the highest overall AUC value of 0.69 (95% confidence interval: 0.60 and 0.77). The features selected most frequently through the LOOCV were all second-order (gray-level co-occurrence or gray-level size zone matrices) and were extracted from images with an applied transformation.

Conclusions

This proof-of-concept work demonstrated the potential of radiomics to differentiate among BAP1+/− in patients with PM. Future work will extend these methods to the assessment of germline BAP1 mutation status through image analysis for improved patient prognostication.

Purpose

Various brain atlases are available to parcellate and analyze brain connections. Most traditional machine learning and deep learning studies analyzing Attention Deficit Hyperactivity Disorder (ADHD) have used either one or two brain atlases for their analysis. However, there is a lack of comprehensive research evaluating the impact of different brain atlases and associated factors such as connectivity measures and dimension reduction techniques on ADHD diagnosis.

Approach

This paper proposes an efficient and robust multimodality model that investigates various brain atlases utilizing different parcellation strategies and scales. Thirty combinations of six brain atlases and five distinct machine learning classifiers with optimized hyperparameters are implemented to identify the most promising brain atlas for ADHD diagnosis. These outcomes are validated using the statistical Friedman test. To enhance comprehensiveness, the impact of three different connectivity measures, each representing unique facets of brain connectivity, is also analyzed. Considering the extensive complexity of brain interconnections, the effect of various dimension reduction techniques on classification performance and execution time is also analyzed. The final model is integrated with phenotypic data to create an efficient multimodal ADHD classification model.

Results

Experimental results on the ADHD-200 dataset demonstrate a significant variation in classification performance introduced by each factor. The proposed model outperforms many state-of-the-art ADHD approaches and achieves an accuracy of 77.59%, an area under the curve (AUC) score of 77.25% and an F1-score of 75.43%.

Conclusions

The proposed model offers clear guidance for researchers, helping to standardize atlas selection and associated factors and improve the consistency and accuracy of ADHD studies for more reliable clinical applications.

Purpose

Squamous cell carcinoma (SCC) accounts for 90% of head and neck cancer. The majority of cases can be diagnosed and even treated with endoscopic examination and surgery. Deep learning models have been adopted for various medical endoscopy exams. However, few reports have been on deep learning algorithms for segmenting head and neck SCC.

Approach

Head and neck SCC pre-treatment endoscopic images during 2016–2020 were collected from the Kaohsiung Veterans General Hospital Department of Otolaryngology-Head and Neck Surgery. We present a new modification of the neural architecture search-U-Net-based model called SCC-Net for segmenting our enrolled endoscopic photos. The modification included a new technique called “Learnable Discrete Wavelet Pooling” to design a new formulation that combines the outputs of different layers using a channel attention module and assigns weights based on their importance in the information flow. We also incorporated the cross-stage-partial design from CSPnet. The performance was compared with other eight state-of-the-art image segmentation models.

Results

We collected a total of 556 pathologically confirmed SCC photos. The new SCC-Net algorithm achieves a high mean intersection over union (mIOU) of 87.2%, accuracy of 97.17%, and recall of 97.15%. When comparing the performance of our proposed model with eight different state-of-the-art image segmentation artificial neural network models, our model performed best in mIOU, Dice similarity coefficient, accuracy, and recall.

Conclusions

Our proposed SCC-Net architecture was able to successfully segment lesions from white light endoscopic images with promising accuracy, with a single model performing well in all upper aerodigestive tracts.

Purpose

The critical time between stroke onset and treatment was targeted for reduction by integrating physiological imaging into the angiography suite, potentially improving clinical outcomes. The evaluation was conducted to compare C-Arm cone beam CT perfusion (CBCTP) with multi-detector CT perfusion (MDCTP) in patients with acute ischemic stroke (AIS).

Approach

Thirty-nine patients with anterior circulation AIS underwent both MDCTP and CBCTP. Imaging results were compared using an in-house algorithm for CBCTP map generation and RAPID for post-processing. Blinded neuroradiologists assessed images for quality, diagnostic utility, and treatment decision support, with non-inferiority analysis (two one-sided tests for equivalence) and inter-reviewer consistency (Cohen’s kappa).

Results

The mean time from MDCTP to angiography suite arrival was 50±34 min, and that from arrival to the first CBCTP image was 21±8 min. Stroke diagnosis accuracies were 96% [93%, 97%] with MDCTP and 91% [90%, 93%] with CBCTP. Cohen’s kappa between observers was 0.86 for MDCTP and 0.90 for CBCTP, showing excellent inter-reader consistency. CBCTP’s scores for diagnostic utility, mismatch pattern detection, and treatment decisions were noninferior to MDCTP scores (alpha = 0.05) within 20% of the range. MDCTP was slightly superior for image quality and artifact score (1.8 versus 2.3, p<0.01).

Conclusions

In this small paper, CBCTP was noninferior to MDCTP, potentially saving nearly an hour per patient if they went directly to the angiography suite upon hospital arrival.

Purpose

We investigate terahertz (THz) polarimetry imaging of seven human breast cancer surgical specimens. The goal is to enhance image contrast between adjacent tissue types of cancer, healthy collagen, and fat in excised breast tumors. Based on the biological perception of random growth of cancer and invasion of surrounding healthy tissues in the breast, we hypothesize that cancerous cells interact with the THz electric field in a different manner compared with healthy cells. This difference can be best captured using multiple polarizations instead of single polarization.

Approach

Time domain pulsed signals are experimentally collected from each pixel of the specimen in horizontal–horizontal, vertical–horizontal, vertical–vertical, and horizontal–vertical polarizations. The time domain pulses are transformed to the frequency domain to obtain the power spectra and 16 Mueller matrix images. The whole-slide pathology imaging was used to interpret and label all images.

Results

The results of the cross and co-polarization power spectrum images demonstrated a strong dependency on the tissue orientation with respect to the emitted and detected electric fields. At the 130-deg rotation angle of the scanned samples, the detector showed the strongest reflected signal in cross-polarization. Furthermore, the Mueller matrix images consistently demonstrated patterns in fresh and block tissues confirming the differentiation between tissue types in breast tumor specimens.

Conclusions

THz polarimetry imaging shows a potential for improving image contrast in excised tumor tissues compared with single polarization imaging. Cross-polarization signals demonstrated smaller amplitudes compared with co-polarized signals. However, averaging the signal during measurements has tremendously improved the image. Furthermore, in post-processing, averaging the frequency domain images and the Mueller matrix elements with respect to frequency has led to better image contrast. Some patterns in the Mueller matrix images were difficult to interpret leading to the necessity of more investigation of the Mueller matrix and its physiological interpretation of breast tumor tissues.

Purpose

Micro-computed tomography (CT) analysis of soft tissues alongside bone remains challenging due to significant differences in X-ray absorption, preventing spatial inspection of bone remodeling including the cellular intricacies of mineralized tissues in developmental biology and pathology. The goal was to develop a protocol for contrast-enhanced micro-CT imaging that effectively visualizes soft tissues and cells in conjunction with bone while minimizing bone attenuation by decalcification.

Approach

Murine femur samples were decalcified in ethylenediaminetetraacetic acid and treated with three different contrast agents: (i) iodine in ethanol, (ii) phosphotungstic acid in water, and (iii) Lugol’s iodine. Micro-CT scans were performed in the laboratory setup SkyScan 1172 and at the synchrotron radiation for medical physics beamline in synchrotron radiation facility Elettra. Soft and hard tissue contrast-to-noise ratio (CNR) and contrast efficiency after decalcification were measured.

Results

In laboratory micro-CT, Lugol’s iodine demonstrated a threefold higher CNR in the bone marrow, representing the soft tissue portion, compared with the bone. Contrast efficiencies, measured in synchrotron micro-CT, were consistent with these findings. Higher resolutions and the specificity of Lugol’s iodine to cellular structures enabled detailed visualization of bone-forming cells in the epiphyseal plate.

Conclusions

The combination of decalcification and the utilization of the contrast agent Lugol’s iodine facilitated an enhanced soft tissue visualization in conjunction with bone.

Purpose

We aimed to compare the efficacy of the double contrast enhancement (CE)-boost technique with that of conventional methods to improve vascular contrast attenuation in lower-extremity computed tomography (CT) angiography.

Approach

This retrospective study enrolled 45 patients (mean age, 70 years; range, 26 to 90 years; 30 males). To generate the CE-boost image, the degree of CE was determined by subtracting the post-contrast CT images from the pre-contrast CT images. The double CE–boost technique involves the application of this CE process twice. Both objective assessments (CT attenuation, noise level, signal-to-noise ratio [SNR], contrast-to-noise ratio [CNR], and image sharpness) and subjective quality evaluations were conducted on three types of images (conventional, CE-boost, and double CE–boost images).

Results

Double CE–boost images demonstrated significantly reduced noise in Hounsfield units (HUs) compared with conventional and CE-boost images (p<0.001). CT attenuation values (HUs) were substantially higher in all different locations of the lower extremity with double CE–boost images (834.49±140.73), as opposed to conventional (399.63±62.01) and CE-boost images (572.66±93.61). The SNR and CNR were notably improved in the double CE–boost image compared with both conventional and CE-boost images. Image sharpness analysis of the popliteal artery (p=0.828), anterior tibial artery (p=0.671), and dorsalis pedis artery (p=0.281) revealed consistency across conventional, CE-boost, and double CE–boost images. Subjective image analysis indicated superior ratings for the double CE–boost compared with other types.

Conclusions

The implementation of the double CE–boost technique improves image quality by decreasing image noise, increasing CT attenuation, and improving SNR, CNR, and subjective assessment compared with CE-boost and conventional imaging.

Purpose

Cells are building blocks for human physiology; consequently, understanding the way cells communicate, co-locate, and interrelate is essential to furthering our understanding of how the body functions in both health and disease. Hematoxylin and eosin (H&E) is the standard stain used in histological analysis of tissues in both clinical and research settings. Although H&E is ubiquitous and reveals tissue microanatomy, the classification and mapping of cell subtypes often require the use of specialized stains. The recent CoNIC Challenge focused on artificial intelligence classification of six types of cells on colon H&E but was unable to classify epithelial subtypes (progenitor, enteroendocrine, goblet), lymphocyte subtypes (B, helper T, cytotoxic T), and connective subtypes (fibroblasts). We propose to use inter-modality learning to label previously un-labelable cell types on H&E.

Approach

We took advantage of the cell classification information inherent in multiplexed immunofluorescence (MxIF) histology to create cell-level annotations for 14 subclasses. Then, we performed style transfer on the MxIF to synthesize realistic virtual H&E. We assessed the efficacy of a supervised learning scheme using the virtual H&E and 14 subclass labels. We evaluated our model on virtual H&E and real H&E.

Results

On virtual H&E, we were able to classify helper T cells and epithelial progenitors with positive predictive values of 0.34±0.15 (prevalence 0.03±0.01) and 0.47±0.1 (prevalence 0.07±0.02), respectively, when using ground truth centroid information. On real H&E, we needed to compute bounded metrics instead of direct metrics because our fine-grained virtual H&E predicted classes had to be matched to the closest available parent classes in the coarser labels from the real H&E dataset. For the real H&E, we could classify bounded metrics for the helper T cells and epithelial progenitors with upper bound positive predictive values of 0.43±0.03 (parent class prevalence 0.21) and 0.94±0.02 (parent class prevalence 0.49) when using ground truth centroid information.

Conclusions

This is the first work to provide cell type classification for helper T and epithelial progenitor nuclei on H&E.

Purpose

The rapid development of highly multiplexed microscopy has enabled the study of cells embedded within their native tissue. The rich spatial data provided by these techniques have yielded exciting insights into the spatial features of human disease. However, computational methods for analyzing these high-content images are still emerging; there is a need for more robust and generalizable tools for evaluating the cellular constituents and stroma captured by high-plex imaging. To address this need, we have adapted spectral angle mapping—an algorithm developed for hyperspectral image analysis—to compress the channel dimension of high-plex immunofluorescence (IF) images.

Approach

Here, we present pseudo-spectral angle mapping (pSAM), a robust and flexible method for determining the most likely class of each pixel in a high-plex image. The class maps calculated through pSAM yield pixel classifications which can be combined with instance segmentation algorithms to classify individual cells.

Results

In a dataset of colon biopsies imaged with a 13-plex staining panel, 16 pSAM class maps were computed to generate pixel classifications. Instance segmentations of cells with Cellpose2.0 (F1-score of 0.83±0.13) were combined with these class maps to provide cell class predictions for 13 cell classes. In addition, in a separate unseen dataset of kidney biopsies imaged with a 44-plex staining panel, pSAM plus Cellpose2.0 (F1-score of 0.86±0.11) detected a diverse set of 38 classes of structural and immune cells.

Conclusions

In summary, pSAM is a powerful and generalizable tool for evaluating high-plex IF image data and classifying cells in these high-dimensional images.