[NeurIPS 2022] HLA-Peptide Binding Prediction Model for the Personalized Treatment of Cancer

Introduction

Traditional anticancer drugs include cytotoxic anticancer (first-generation drug) and targeted anticancer drugs (second-generation drug). Where the former works by destroying cancer cells directly with toxic substances, the latter inhibits cancer progression by inhibiting genes that promote cancer growth or abnormal tumor suppressor genes. The disadvantage of the first-generation anticancer drugs is that they negatively affect normal cells, and that of the second-generation anticancer drugs is that they may cause side effects if a mutant gene is produced.

In response, immunotherapeutic drugs have emerged as the third generation of anticancer drugs. These cancer immunotherapy drugs, which are receiving the most attention among the numerous cancer treatments currently being developed, act by stimulating the immune system. By injecting synthetic immune proteins into the body, immune cells selectively attack only cancer cells, as opposed to conventional anticancer drugs that target cancer cells directly.

Figure 1. Personalized Cancer Vaccine Treatment Process^[1]

Numerous immune components are targeted by immunotherapy drugs, among which human leukocyte antigen (HLA) plays a crucial role in adaptive immune responses for cancer immunology. HLA is the human version of Major Histocompatibility Complex (MHC) that plays a crucial role in the human adaptive immune system. Cells in our body continuously release protein fragments, which, when combined with HLA class I molecules, are expressed on the cell surface as 8- to 10-amino acid peptides. By expressing a particular peptide-HLA complexes on the cell surface, cytotoxic T cells identify the cell as cancerous and release cytotoxin to destroy it. A neoepitope is an epitope that arises as a result of genetic mutations in cancer cells; it can be used to develop personalized cancer vaccines. Material Intelligence (MI) Lab at the LG AI Research concentrates on HLA class-I molecules and neoepitope in order to develop an AI model for a personalized cancer vaccine. In addition, an HLA-peptide binding prediction model was created and a thesis titled "Allele-conditional attention mechanism for HLA-peptide complex binding affinity prediction" was published. A presentation was also given at the NeurIPS 2022 Workshop on Machine Learning and Structural Biology (MLSB).

Current Issues & Motivation for Research

With the recent release of an open dataset known as the Immune Epitope Database (IEDB)^[2], the performance of HLA-peptide binding prediction has significantly improved. After NetMHCPAN^[3] first employed a neural network to predict HLA-peptide binding, a number of models employing recurrent neural networks and convolutional neural networks were presented. In addition, many methods utilizing self-supervised pretrained models, such as Evolutionary Scale Modeling (ESM)^[4] and Bidirectional Encoder Representations from Transformer (BERT)^[5] based on the attention mechanism, have been announced recently. Using an attention mechanism, they express at the residue level the interaction between a protein (HLA) and a peptide. Nonetheless, there is a dearth of research on how to utilize the attention mechanism most effectively in situations where data is relatively scarce.

To circumvent this, LG AI Research proposed a conditional attention mechanism that focuses on the HLA structure using peptide information, and confirmed that its performance was superior to the more complex concatenated or cross attention methods. The source code and dataset were both integrated to overcome the limitations of existing models whose objective performance was difficult to compare due to fragmented datasets and source codes. In addition, for a fairer comparison of performance, we set the measurement using the feature expression method and pooling method of multiple amino acids.

Figure 2. Structure of Attention-Based Model^[9]

Research Contents of LG AI Research

Figure 3. Structure of GRU/CNN-Based Model^[9]

The objective of this study is to predict the relationship between [HLA] and [Peptide] string sequences. Importantly, the interaction between two sequences of strings must be represented by an attention mechanism. In the existing methods utilizing the attention mechanism, after connecting peptides and HLA sequences with [SEP] token, as in Bert attention in Figure 2 (a), self-attention is applied to the entire sequence, or cross-attention is applied for the two sequences after self-attention has been applied for the peptide and HLA, respectively, as depicted in Figure 2 (b). Despite the fact that HLA sequences are typically lengthy, the structural diversity of HLA sequences is not wide. In contrast, the length of peptide sequences is short, but the structural diversity between each peptide sequence is relatively high, and the diversity of the information contained is greater.

The structure proposed in this paper makes use of HLA sequences as a context for peptide-protein interactions. Initially, a representation of the peptide sequence is obtained through self-attention. Using the obtained peptide representation as the attention query (Q_pep) and the HLA sequence as the attention key (K_HLA) and value (V_HLA), the attention calculation is carried out to obtain a representation as long as the peptide sequence. Following a pooling calculation on this representation, an MLP operation is applied to predict the binding affinity score of peptide and HLA. Throughout this procedure, the model naturally learns to identify the binding site where the peptide binds to the HLA sequence when the peptide is likely to bind on HLA sequence.

Learned Embedding (LE), Amino Acid Index (AA), and ESM Pretrained Model (ESM) were the three primary methods considered for expressing input embedding. In the case of LE, trainable random initialized embedding was employed for each amino acid character, whereas in the case of AA, the value of each amino acid's physical/biological properties^[6] and its value in the BLOSUM matrix^[7] were expressed. When each HLA sequence was input into ESM, the output embedding of the ESM model was utilized.

The [CLS] token, mean pooling, and learned weighting pooling methods were considered. [CLS] token is a method for appending [CLS] token to the end of the sequence and substituting the final representation as the input to MLP. Mean pooling is a method for calculating the average value of the sequence length of the final representation, whereas learned weighting is a method for calculating the mean value after multiplying each sequence position by a trainable weight.

Table 1. Performance by Model According to Input Embedding (Pooling Method: Learnable weight)^[9]

Table 2. Performance by Model According to Pooling Method(Input embedding: peptide=AA, HLA=ESM)^[9]

As demonstrated in Tables 1 and 2, when compared to other methods such as input embedding and pooling, the conditional attention method proposed in this paper demonstrates superior overall performance. This is due to the fact that despite the limited data, conditional attention was able to perform accurate attention calculations due to the differences in length and diversity of peptides and HLAs. In addition, as increasingly complex HLA input embeddings (LE → AA → ESM) are employed, the conditional attention model's performance gradually improves. This can be attributed to the model's structure, which is suspected to make effective use of the pretrained model's enriched data.

Figure 4. Attention Weight Distribution for Amino Acids in Peptide Sequence
(Left: distribution of positive binding instances; middle: distribution of negative binding instances; right: difference between positive and negative distributions)^[9]

Figure 5. Attention Weight Distribution for Amino Acids in HLA Sequence
(Left: distribution of positive binding instances; middle: distribution of negative binding instances; right: difference between positive and negative distributions)^[9]

In addition, the distribution of attention weights for every position/amino acid of peptide and HLA was visualized. Binding affinity-predicting positions/amino acids are depicted in the plots to the right of Figures 4 and 5. In particular, as shown in Figure 5, it is possible to infer the binding site in the HLA-peptide binding process by identifying the crucial HLA position for predicting binding affinity.

Figure 6. Attention Highlight in 3D Space Between Peptide and HLA; green-hued areas that have been highlighted^[9]

As depicted in Figure 6, the highlighted positions on the attention weight distribution can be seen in the actual 3D structure. Considering that the highlighted positions in peptide and HLA are physically close, it can be inferred that the conditional attention model is also learning structural information through sequence information while learning to predict the peptide-HLA binding.

Future Plans of LG AI Research

In this study, binding affinity was predicted using only HLA and peptide sequence information; however, future research may consider a method employing 3D structural information of HLA and peptide. There have been studies examining the 3D structure of protein language models^[8], but there lies a problem where docking data for peptide and HLA complexes are lacking. An approach employing a pretrained model based on 3D structural data would also be an intriguing method for addressing this issue.

In addition to peptide-HLA binding affinity, the MI Lab of LG AI Research developed an Immunogenicity prediction model. In vitro experiments have validated candidate peptides predicted by the developed model, and future preclinical and clinical trials are further planned. It is anticipated that as preclinical and clinical trials progress, research conducted by LG AI Research will have a greater impact on the development of personalized cancer vaccines.

▶Allele-conditional attention mechanism for HLA-peptide complex binding affinity prediction (Link)