[CLEF 2022] Context Aware Named Entity Recognition and Relation Extraction

LG AI Research is conducting research on information extraction technology in the chemical sector jointly with a world-leading publisher in science, Elsevier. Information extraction is a technology for automatically extracting structuralized information from raw tests. Having constructed a database using various chemical journals and patents, Elsevier provides not only original texts, but also information on chemical reactions, materials, etc. in a structuralized format using the web-based service called Reaxys^[1]. This is very helpful in reducing the time it takes for chemists to search through documents one at a time to find the necessary information for their research. Information extraction technology plays a crucial role in configuring the Reaxys service. As countless chemical journals and patents are appearing in various formats every year, it must be possible to automatically extract necessary information to build a database and provide the information sought out by chemists in an arranged form.

Elsevier is a partner company that held the Cheminformatics Elsevier Melbourne Universities lab (ChEMU)^[2] Challenge with the goal of extracting chemical reaction information from chemical patents at the 2022 Conference and Labs of the Evaluation Forum (CLEF). LG AI Research, which took part in this event through the introduction of Elsevier, participated in two of the five tasks from the Challenge, and achieved the highest performance among the four participating teams. We will introduce the two tasks that the LG AI Research Language Lab participated in and the technologies for extracting chemical reaction information from chemical patents through the applied technologies.

Task

ChEMU 2022 unveiled a total of five tasks and the two tasks that LG AI Research participated in are 1a and 1b. As shown in Figure 1, each task extracted the necessary information from a snippet that describe chemical reactions from the chemical patent.

Figure 1. Examples of Tasks 1a and 1b at ChEMU 2022

Task 1a (Named Entity Recognition)

- Extraction of entities of chemical materials, numerical information, etc. explaining chemical reactions
- Composed of a total of 10 entities such as EXAMPLE_LABEL, REACTION_PRODUCT, and STARTING_MATERIAL

Task 1b (Event Extraction) (Named Entity Recognition)

- Extraction of trigger words that explain chemical reactions and extraction of relations between the extracted trigger words and entity
- Composed of the two trigger words of WORKUP and REACTION_STEP and two relations of ARG1 and ARG M

Each task is evaluated using F1 Score for Exact Match and Relaxed Match. In the case of Task 1a, it evaluates whether the predicted entity type and text position match the answer. To get scores, span must be completely matched in the case of Exact Match, and for Relaxed Match, if the text overlaps with the span even a bit, then scores are given. Task 1b evaluates whether the trigger word and relation are both matching, and in the case of the trigger word, evaluations are made similar to that of Task 1a.

The F1 Score measured in the Challenge was disclosed in two categories: Public Score and Private Score. The Public Score is the score pre-disclosed before the submission deadline, which is the evaluation result of 30% of all the data. Private Score is the score disclosed after the submission deadline, which is the result of evaluating 70% of the total data.

Contributions

In this section, we will introduce five contributions to performance enhancement in the tasks that we participated in.

1. Domain-specific language model

As the given data for the task was limited, a pre-trained language model using large quantities of data was needed. To develop domain-specific language model trained on chemical texts, we chose encoder-based architecture, which is still showing strengths in tasks such as Named Entity Recognition (NER) and Relation Extraction (RE). If there was enough time, the model would have been trained from scratch, but since this was not the case, Bio-LinkBert^[3] showing high performance in the biomedical sector that is similar to that of the chemical sector was post-trained. 

Data used for learning were chemical patents (23GB), journals (22GB), and Pubmed (20GB) consisted of texts in the biomedical sector used for Bio-LinkBert learning. Experiments were conducted with a combination of various data, and the model that learned using chemical journals and Pubmed^[4] was found to have the highest performance. When evaluating Tasks 1a and 1b of ChEMU 2022, higher performance was achieved in Task 1a compared to other disclosed models as shown in Table 1.

Table 1. Comparison of performance of ChEMU 2022 Tasks 1a and 1b when using self-developed domain-specific language model (Ours) and different language models in the disclosed biomedical field

2. NER (Named Entity Recognition) & RE (Relation Extraction) model

We used the aforementioned domain-specific language model to develop the NER and RE models in forms suitable for Tasks 1a and 1b. The NER model is used for extracting the entity of Task 1a and the trigger word of Task 1b, and the RE model is used to extract the relation between the entity of Task 1a and the trigger word. 

There is also the method of developing NER and RE as a single model to learn by multi-tasking, but considering test results that learning models for each task have higher performance in [5] as shown in Table 2, two models were separately produced to operate with a pipeline method.

Table 2. Performance difference when learning using a single model that learns the NER and RE model as multi-tasks or two models operating as a pipeline in the ACE05 dataset that extracts information from the news

The NER model has two main approaches: the sequence tagging approach^[7] and the span-based approach^[5][6]. The sequence tagging approach is a method of classifying tokens as individual BIO tags. On the other hand, the span-based approach is a method that classifies all spans that can be entities as a specific entity type or no-entity. Here, span normally refers to all token sequences under a specific length.

Figure 2. Examples of span-based approach and sequence tagging approach

We experimented with both approaches as shown in Table 3 and finally selected the sequence tagging approach that performed better than span-based approach. It's because ChEMU data has entities including many tokens, and therefore, the span-based approach that classifies all spans is not appropriate. Furthermore, when developing the model with the sequence tagging approach, the classification layer was set not only as the dense layer, but also as the Conditional Random Field (CRF); however, the results did not show big differences in terms of performance, and thus, the dense layer with relatively fast speed was used.

Table 3. Performance comparison in NER of ChEMU 2020 according to NER architecture 
(The NER task of ChEMU 2020 was the same as 2022, but learning and evaluation was conducted with less data. The model must be registered in the ChEMU web page for evaluation in ChEMU 2022, but since it was not appropriate to register and evaluate all models, it was evaluated with data from 2020.)

The RE model creates pairs between all entities and trigger words extracted from NER to categorize as a specific relation type or as no-relation. For this, the input data of models are composed as shown in Figure 3. A special token showing each type and front/back is added to the front and back of the entity as well as the trigger word. The configured data is then entered in the language model, and each special token representation was added in front of the entity and trigger word to be used in categorizing the relation of each pair. 

Figure 3. Example of input data composition of model to categorize the relation between the pairs of entity and trigger word

3. Pre-processing methods

Tasks 1a and 1b data were created by extracting the snippet part explaining the chemical reaction after converting the chemical patent to text using OCR. There may be differences in performance according to the method of pre-processing the given data to create the input data of the model.

Figure 4 is an example of the data, and the color of each text shows the text of each line seen in the OCR results. In general, OCR divides the line when the paragraph, section, etc. are divided to create data.

Figure 4. Snippet example of chemical patent provided by ChEMU 2022

The most important point when making the given data into the input data of the model is to create data including as many tokens as possible within a scope that does not exceed the maximum input length of 512 tokens. This is because when the model predicts the entity or relation in Tasks 1a and 1b, it not only requires the information of surrounding texts, but also needs information of faraway texts at times.

The simplest method for satisfying this condition is to create data that includes as much texts as possible within a scope that does not exceed 512 tokens, which is the max. input length of the model. However, upon analyzing the data, it was evident that for some entities, text information in different lines had to be referred to for extraction, and some relations were spread across several lines. Thus, as shown in Figure 5, input data of the NER model was created in a method that includes as many sentences as possible without exceeding 512 tokens for all snippets. In the case of the RE model, aside from texts between the entity and trigger word, data was created to include as many surrounding texts as possible.

Figure 5. Example of NER input data created after pre-treating in snippet units

Although this method is very simple, it can greatly improve performance as shown in Table 4. When composing the input data of the model with snippet units instead of line units, the NER model was improved by 1 point or more for both Public and Private EM f1 score. In addition, task 1b displayed slightly improved performance.

Table 4. Comparison of performance of task 1a and task 1b during model learning with input data obtained by applying the pre-processing method in line or snippet units

4. Post-processing methods

The post-processing method that compensates the prediction results of NER and RE models learned based on data using rules was applied. Through analysis of model prediction data, rules were developed for compensation in the case of (a) mis-classifying entity types, (b) mis-classifying trigger word types, and (c) wrongfully predicting span of the entity or trigger word. Each rule referred to the definition of the entity, trigger word, and relation type. For example, if the model predicted that there would be an entity type or relation that cannot be connected if a specific trigger word type follows the definition, this was fixed properly. The model in the early stage of research did not have good performance and therefore, performance improved significantly when compensating with the configured rules. The latest model, on the other hand, had good basic performance, thus exhibiting relatively less improvement to performance when compensating with the same rules.

5. Cross validation & Ensemble

In order to improve the performance of the NER and RE models, 10-fold cross validation was applied to train 10 models. An ensemble that determines the prediction results of 10 models through soft-voting was then applied to improve performance compared to a single model as shown in the evaluation results.

Evaluation Results

Table 5 shows the final evaluation results of ChEMU 2022 Task 1a. When looking at the performance of a single model, the Public and Private Exact F1 score was +1.3 and +1.7 higher than the model with the highest performance, respectively. These are the three effects of the domain-specific language model, NER & RE model, and pre-processing method among the five contributions explained above. In addition, it is evident that the performance improved when post-processing and ensemble were both applied, and therefore, Public and Private Exact F1 score improved by +0.8 and +1, respectively.

Table 5. Task 1a performance in ChEMU 2022

Table 6 shows the evaluation results of Task 1b. Similar to Task 1a, the performance of the single model achieved higher performance than all the other models that applied both ensemble and post-processing. As Task 1b refers to the trigger word extraction performance and relation extraction performance, they are the evaluation results of the two models of NER and RE. It was found that the NER model performance was better in the Tasks 1a and 1b evaluation results, but it was unknown whether the performance of the RE model was improved. Therefore, the NER model with lower performance (similar performance as Hokkaido University) was used intentionally to extract the entity and trigger word to utilize the RE model to extract relation. The Public and Private Exact F1 performance improved by +2 and +1.3 points, respectively, compared to the model with the highest performance as shown in Ours (single, worst ner) in Table 6. Thus, it was found that the RE model architecture also had impact on improving the performance.

Table 6. Task 1b performance in ChEMU 2022

Conclusion

In this blog, we introduced ChEMU 2022’s Named Entity Recognition, Event Traction task, and the technologies applied by LG AI Research in the corresponding task. ChEMU had other tasks such as Anaphora Resolution and Chemical Reaction Reference Resolution, and moreover, there exists various tasks such as text classification and sentence similarity within the Information Extraction. The LG AI Research Language Lab focuses not only on the research introduced in this blog, but also on research of large-scale language model that serves as the model architecture and basis in various information extraction tasks. Our goal is to automatically extract information from various chemical documents using this to improve the service quality as shown in Reaxys and to enhance research efficiency of chemical researchers.