EXAONEPath : a Multi-Instance Learning (MIL) Framework and Patch-by-Patch Foundation Model for Processing Gigapixel-Scale Histopathology Images

LG AI Research is releasing EXAONEPath, pre-trained model specialized for histopathological images, as open source. histopathological images are usually quite large, around 20k x 20k in size, and the range of color tones in the image is different from general-purpose images. Objects covered in histopathological images are also unique, such as nuclei and cytoplasm. Therefore, the need to develop a specialized model that can efficiently process and analyze histopathological images has arisen, and so we’ve released EXAONEPath,pre-trained model specialized for histopathological images. 

EXAONEPath achieves both high performance and affordability. Using six benchmarks, we found that the accuracy of histopathology image analysis was comparable to that of global competing models. Notable is the superior affordability compared to competing models. With a small amount of training data and 1/10th the size of competing models, we found that our model was significantly more affordable than competing models because it required less infrastructure to train the data.

EXAONE 3.0 and EXAONEPath, which were released last week, are the realization of LG AI Research’s vision of “Expert AI.” In particular, we expect EXAONEPath to be the beginning of innovation in the bio field. If AI can understand and analyze histopathology images, it will be able to predict gene mutations and determine the appropriate treatment methods and types of drugs without the need for genetic testing. This will save both time and money by shortening the existing genetic testing period, which can be up to two weeks. We look forward to the moment when bio professionals experience the innovation that EXAONEPath brings to the table, and eventually to our own lives. In addition, Expert AI demonstrated by EXAONE 3.0 and EXAONEPath is in line with the “ABC” direction that LG Group is mapping. In the process of securing future growth engines and technological competitiveness in AI, Bio, and Cleantech, LG AI Research will continue to research and innovate.

This post will introduce EXAONEPath, pre-trained model specialized for histopathological images in detail, along with histopathology imaging research trends. 

Histopathology Imaging Research Trends 

While histopathology images may be unfamiliar to the average person, they are essential data in clinical medicine. Histopathology image analysis can be used to characterize cells and tissues, confirm diagnoses, and develop plans for medication and treatment. Through histopathology images, clinical pathologists are essentially able to observe the morphology and structure of cells and surrounding tissues. Therefore, a lot of work is put into preparing slide specimens so that these features are visually represented.

First, the affected areas and normal tissues are excised for diagnosis. They are then formalin-fixed to remove moisture, preserve the tissue structure, and prevent decay. They are then paraffin-embedded to prevent further tissue deformation. This process is known as the formalin fixed-paraffine embedded (FFPE) technique. The prepared specimens are then sliced to a thickness of about 4um and made into slides for microscopy. These specimens are then stained to make the cell nuclei more visible. Hematoxylin and eosin staining (H&E staining) is a common staining method, which makes it easier to identify the distance between cells and the size of the cells themselves. The specimen slide is then photographed under an optical microscope to obtain a whole slide image (WSI), or in other words, a histopathologic image for diagnosis.

General-purpose images are often resized to between 256 px and 1024 px for training AI models. This is usually acceptable as it does not overly change the overall meaning. However, WSIs are too large to be accepted. For example, an image taken with an objective lens at 20x magnification (x20) usually has a length per pixel (mpp, micro-meter per pixel) of 0.5. If a specimen has 2 cm x 2 cm tissue on it, the image size obtained is at least 40,000 px x 40,000 px. Considering that the maximum pixel size of a modern TV is 8,000 px on a side, this is at least 5 times larger (25 times in area). If an image of this size is drastically reduced to between 256 px and 1024 px, the shape of the cells becomes indistinguishable.

Image 1. MIL Framework

 

Therefore, a framework called multi-instance learning (MIL)^[1] is commonly used in the field of histopathology image processing (Image 1). In MIL, the image is divided into patches (also called tiles), each of which is converted into a latent vector by a pre-trained image encoder, which is then integrated into a phenotype vector for the entire image (slide). By doing this, the morphological characteristics of the cells are preserved while allowing complete information to be stored. The MIL framework is an ML method used when a problem is given in the form of a data bag-label rather than the original data-label form. For example, a gigapixel-sized histopathology image can be viewed as a bag of patches and treated with the MIL framework.

 

Emerging Need for Models Specialized for Histopathology Images 

Until recently (around 2023), most pre-trained image encoders in the field of histopathology image processing used CNNs trained from general-purpose images. The ResNet series, which was trained on ImageNet 1k, was mainly used, and the versatility shown by the pre-trained ResNet played a role in this processing. However, the main reason for use was that there was no way to supervise pathological images. There was no way to organize the labels that could be identified through pathological images. Even if labels were assigned, it is realistically difficult to produce a dataset with a fantastic class balance like ImageNet.

However, the characteristics of histopathology images and general-purpose images are quite different. In general-purpose images, there are various objects (people/objects) with a wide range of colors and sizes. However, in pathological images, the objects consist only of cells and cytoplasm, making them relatively simple. Further, as histopathology images contain a lot of information about the size of cells and the distribution of surrounding tissues, the scale of the patch needs to be constant to be learned. There was doubt surrounding the quality of the phenotype vectors of pre-trained CNNs that did not consider these points, and the performance limitations were clear. Thus, we found it was necessary to secure a pre-trained encoder that learned pathological images in order to become a leader in pathological image processing.

 

EXAONEPath, a Pre-trained Model Specialized for Histopathology Images

Recent advances in self-supervised learning methods have provided the technology to obtain histopathology image pre-trained encoders. Self-supervised learning in vision has grown dramatically in recent years, centered on the teacher-student model. Self-supervised learning methods, represented by algorithms such as MoCo^[2] and DINO^[3,4], learn the image vectors output by the teacher network in a format that the student network learns. Using DINO as an example, the teacher network is fed a low-intensity modification of the original image, and the student network is fed a high-intensity modification of the original image. However, these two images must be semantically the same. The phenotypes output by the two networks should be similar, even if the input images have different modification intensities. Learning in this way allows the model to embed knowledge about the objects, shapes, etc. within the image. This is evidenced by superior performance when training subsequent tasks on labeled data.

Histopathology image pre-training models can also be learned using the DINO method. However, we need to segment only the tissue part of the WSI and create patches with a size that can be input into the teacher-student network. As previously mentioned, it is necessary to keep the scale of each patch constant. Thus, the meta-information of the WSI was read and the image was resized so that all patches had the same mpp. We first trained our pre-trained model with data from 35000 public WSIs. This is about 280 million images per patch.

However, when histopathology image patches were trained using DINO, we ran into a problem. Because H&E staining is done by humans using staining materials, the staining varies from image to image (Image 2, Top).

 

Image 2. Original H&E Staining Image (Top), Image After Stain Normalization (Bottom), Specimen Image (Bottom Left)

 

As a result, the patches learned through DINO were found to cluster on each slide along the color scale (Image 3, Left). From a machine-learning perspective, DINO does not learn the shape of the cells in a patch and the distribution of surrounding tissues, but simply collects data based on color.

Therefore, the patches output by the trained model are dependent on the color of the slide, which can adversely affect the quality of the tuning. To solve this problem, we utilized a normalization technique called stain normalization^[5]. We took an image with good staining quality as a specimen (Image 2, Bottom Left) and transformed the color of the patches to be the same as this specimen (Image 2, Bottom). By doing so, we were able to weaken the variation in color, allowing the model to learn cell shape, size, and surrounding tissue distribution rather than only learning color. This can be seen in Image 3 on the right, where the patches are evenly scattered across the slides rather than clustered together.

Image 3. Phenotypic Vector Distribution Before (Left) and After (Right) Stain Normalization Vector Plot Results for 1000 Random Patches from 10 Slides

We named this pre-trained histopathology image encoder EXAONEPath. EXAONEPath could be compared to competing models on six patch image benchmark datasets. The competing models (prov-GigaPath^[6], Rudolf V^[7], Phikon^[8], CTransPath^[9], and Lunit^[10]) learned as many as 160,000 pathology images, and had as few as 22 million and as many as 1.1 billion parameters. For reference, EXAONEPath used 86 million parameters. As a result, EXAONEPath, trained with relatively little learning data and a small model, showed competitive performance (Image 4, 5). This showcases EXAONEPath’s affordability and efficiency, in that it requires fewer resources for training. Further performance improvements can be expected when additional histopathology images are acquired. More specifically, micro satellite instability (MSI) in colon, rectal, and stomach cancer is a measure of how a mutation in the MMR protein is not repaired by a genetic defect in the MMR protein, resulting in a change in the number of repeats and making it different in length from normal cells. MSI is often categorized as MSS (stable) or MSI-L/H (instable low/high) depending on the number. Prediction performance for our model for MSI in those specific cancers was found to be the highest compared to competing models (based on linear probing).

 

Image 4. Performance Compared to Number of Parameters and Number of Learning Slides  

Image 5. Benchmark Performance Details for Each Model

 

LG AI Research will continue to update EXAONEPath through comprehensive follow-up research, including the addition of learning data and the advancement of learning methods. We are continuing with research agreements between both Korean and overseas medical institutions, as well as conducting empirical benchmarking to build real-world use cases for EXAONEPath.

While we have obtained patch-level encoder models through this study, our ultimate goal for the MIL framework is slide-level inference. However, to date, benchmark datasets at the slide level are scarce, and empirical benchmarks are difficult to obtain due to privacy and ethical concerns. If they do exist, they are small in scale. The lack of datasets makes it difficult to train an aggregation network to combine the patch-level phenotype vectors output by EXAONEPath at the slide level.

To solve the real-world problem of data scarcity, LG AI Research is conducting research to train even integrated networks through self-supervised and unsupervised learning. Here, a large amount of patient data and lab data related to histopathology images are being used together. Our research direction is in line with the direction of translational medicine, which has been actively pursued in the medical field in recent times. LG AI Research will continue to work on medical AI research so that we can help improve human life by discovering patient conditions and treatment methods through various patient data.

 

Try EXAONEPath on Hugging Face →

View more details : EXAONEPath 1.0 Patch-level Foundation Model for Pathology →