KEYWORDS: Image fusion, Feature fusion, Design, Data fusion, Lung cancer, Data modeling, Education and training, Image classification, Genomics, Cancer
Cancer prognosis and survival outcome predictions are crucial for therapeutic response estimation and for stratifying patients into various treatment groups. Medical domains concerned with cancer prognosis are abundant with multiple modalities, including pathological image data and non-image data such as genomic information. To date, multimodal learning has shown potential to enhance clinical prediction model performance by extracting and aggregating information from different modalities of the same subject. This approach could outperform single modality learning, thus improving computer-aided diagnosis and prognosis in numerous medical applications. In this work, we propose a cross-modality attention-based multimodal fusion pipeline designed to integrate modality-specific knowledge for patient survival prediction in non-small cell lung cancer (NSCLC). Instead of merely concatenating or summing up the features from different modalities, our method gauges the importance of each modality for feature fusion with cross-modality relationship when infusing the multimodal features. Compared with single modality, which achieved c-index of 0.5772 and 0.5885 using solely tissue image data or RNA-seq data, respectively, the proposed fusion approach achieved c-index 0.6587 in our experiment, showcasing the capability of assimilating modality-specific knowledge from varied modalities.
Multiplex brightfield imaging offers the significant advantage ofsimultaneously analyzing multiple biomarkers on a single slide, as opposed to single biomarker labeling on multiple consecutive slides. This potentially enables investigating interactions between biomarkers to be analyzed to gain a better understanding of the tumor microenvironment as well as improved predictive and prognostic abilities.
Antibody development is crucial for immunohistochemistry (IHC) applications. To improve the efficiency of primary antibody screening processes, we developed a computer aided detection scheme to automatically identify the non-negative tissue slides which indicate reactive antibodies. A dataset with 564 digital IHC whole slide images were used for algorithm training and testing, each of which was labeled by pathologist as a negative (i.e., no staining) or non-negative (i.e., pure background or partial staining) slide. To avoid unnecessary computations, color deconvolution was first applied to low resolution whole slide images and histogram based image features were extracted from each unmixed single stain image. Then, different classifiers were built using the low resolution image features computed from the training dataset through ten-fold cross validation. The trained model was tested over the testing dataset. Results indicated that linear supported vector machine (LSVM) method yielded the highest area under ROC curve. To further improve the accuracy, our scheme utilized the LSVM classifier score to identify the slides for which additional analysis was needed. The additional analysis was performed through dividing the original whole slide image into non-overlapping tiles and extracting high resolution image features from each tile. The tile-based features are then used to form a bag-of-words (BoW) representation of the corresponding whole slide image, based on which a second classifier was built to perform the predictions. The results showed that the proposed scheme can effectively perform negative versus non-negative classification with high accuracy and thus reduce pathologists’ manual reviewing time for antibody screening.
Performance of image analysis algorithms in digital pathology whole slide images (WSI) is usually hampered by the stain variations cross images. To overcome such difficulties, many stain normalization methods have been proposed where normalization is applied to all the stains in the image. However, for immunohistochemistry (IHC) images, there exist situations where not all the stains in the images are desired or feasible to be normalized, especially when the stain variations relate to certain biological indications. In contrast, the counter stain, usually hematoxylin (HTX), is always desired to be consistent cross images for robust nuclei detection. In this work, we present a framework to normalize the HTX stain in an IHC WSI through alignment to a template IHC WSI. For this purpose, we use the Hue-Saturation- Density (HSD) model and align the chromatic components distribution of the image to the template. Then we shift and scale density component to match the template. In order to retain the non-HTX stain, we differentiate the pixels which have pure HTX stain from those which are mixture of HTX and non-HTX stains, and different normalization strategy is applied accordingly. In the results, for a wide range of HTX stain variations, we show qualitatively the performance of the method. We also show algorithm performance dependence on the stain concentration can be much reduced by the proposed method.
For histochemical staining, to highlight multiple biomarkers within a sample, multiple stains with different light spectral absorption characteristics are deployed (i.e. multiplexing). To reconstruct the single stain contrast from a multiplexed sample, the conventional color deconvolution method assumes that light extinction follows Lambert-Beer’s law during imaging process and the optical density (OD) measured from the image is linearly related to the stain amount. However, this assumption does not hold well for commonly used diaminobenzidine (DAB) stain due to its precipitate-forming reaction during sample processing. Besides absorption, scattering also contributes to the light extinction process which causes the non-linear relation between the OD value and the stain amount. Therefore, using the conventional method may not have sufficient accuracy for quantified stain analysis, especially when DAB presents at high concentration levels. In this paper, our study shows that DAB presents different chromatic properties at different concentration levels. Therefore, we propose a new color deconvolution method to address the issue by employing a set of reference colors vectors, each of which characterizes a DAB concentration level. Then, the reference color vector that best represents the true DAB concentration level in the mixture is automatically selected for color deconvolution. Both visual and quantified assessments are provided to show that the method enables detection for a broader dynamic range of DAB concentration and therefore should be preferred by the user for bright field image analysis.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.