KorQuAD 2.0 1st rank: Generative Model for Long Context Question Answering

1. Introduction

LG AI Research’s EXAONE Lab is responsible for creating large-scale language models to be used for collaboration with LG Group affiliates and partners and focuses on language model training techniques to ensure high performance We challenged the leaderboard to showcase the excellent performance of our in-house developed EXAONE and recently achieved 1st rank of KorQuAD2.0^[2] following the 1st rank of KorQuAD1.0^[1] in June.

KorQuAD(The Korean Question Answering Dataset) is a Korean artificial intelligence (AI) machine reading comprehension assessment, and KorQuAD 2.0 is a Korean machine reading comprehension (MRC) dataset consisting of 100,000+ pairs of questions and answers, including 20,000+ pairs from KorQuAD 1.0. This article will discuss the techniques used in KorQuAD2.0 and their significance.

2. KorQuAD2.0

KorQuad2.0 is a multi-paragraph machine reading problem, which is challenging in terms of both inference speed and accuracy due to the increased length of documents to be processed compared to single-paragraph machine reading like KorQuAD 1.0. Given a question in natural language and a Wikipedia web page in HTML code as input, it must answer in a variety of ways, including short answers, very long paragraph answers, answers in tables, and answers in lists.

Table 1. KorQuAD 2.0 Dataset document and question count distribution^[2]

Table 2. KorQuAD 2.0 answer type distribution^[2]

KorQuAD2.0’s data can be largely divided into three groups.

1. Short_text (1-20 characters): The first group consists of short answers in pure text, without HTML tags.

2. Middle_text (20-120 characters): The second group consists of short answers, but they contain HTML tags, which make the answers longer than the Short_text group.

3. Long_text (120+ characters): The third group consists of long answers. Since there are different lengths of passages and answers, it is necessary to design a model that takes length into account.

In other words, to solve KorQuAD2.0 problems well, the model needs to be able to utilize the characteristics of HTML documents well and be able to give long and short answers while considering tags such as Table and List.

3. Related Works

Existing studies have mainly used the span prediction methodology with an encoder model structure to solve the extractive QA problem of extracting correct answers from contexts such as KorQuAD.

Image 1. Encoder base span prediction using Bert^[3]

Span prediction is a methodology that categorizes where an answer starts and ends from a given context and infers the entire answer in that area. It has the advantage of being fast because it only needs to infer the start and end, but also has the disadvantage of not being able to guess the correct answer if there is no answer in the given context.

Since then, attempts to solve the question-answering problem in a generative way have emerged, with models with strong encoder-decoder structures such as Bart^[4] and T5^[5]. Unlike the span prediction method, they generate new answers by combining the input context and the model’s parametric knowledge, so even answers that are not in the input context can be generated using the information trained during the pre-training process.

Image 2. T5 generative problem solutions^[5]

In addition, unlike the Bert model, which was only available in large sizes, the Google T5 model has been released in larger sizes such as 3B and 11B, and the instruction-tuned model, FLAN-T5^[6], has also been released to the public and is constantly being updated. In addition, models are continuously used in new methodologies such as DPR^[7], FID^[8], and RAG^[9].

4. Contributions

1. EXAONE LM 1.0

Recently, large-scale language models have been gaining attention for their powerful performance, but running large-scale models requires infrastructure investment, including large amounts of computing resources. Researchers are tackling this issue in a variety of areas (model compression, distillation, pruning, etc.), but it is still a technically challenging issue. In accordance with this, LG AI Research strived to create an effective model that can be operated with less equipment for various real-world industrial applications and developed the EXAONE LM 1.0 model with an encoder-decoder structure.

The encoder-decoder structure is excellent at understanding documents and generating sequences based on them, as the decoder generates sequences based on the representation of the encoded document. It also has the advantage of being able to use the encoder part separately from a model with an already learned encoder-decoder structure^[10].

The EXAONE LM 1.0 model was trained using expert data held by LG AI Research and has shown good performance with less equipment in supervised fine-tuning tasks such as summarization, question answering, and classification tasks. Furthermore, it is trained in two languages, Korean and English, so it has strengths in both languages.

LG AI Research wanted to verify the performance of EXAONE LM1.0 through a public leaderboard, and the model was used as a backbone for korQuAD1.0 and then korQuAD2.0, achieving the 1st rank. LG AI Research plans to continue to develop the EXAONE LM 1.0 model and utilize it as a backbone model for applications that use external knowledge for LLM, such as RAG.

1. Post-Training(QA, HTML-T5)

The post-training method is a technique often used in domain adaptation^[11], etc. to achieve higher performance by learning with additional pre-train objectives on a corpus that was not trained through the pre-training process.

Since KorQuAD 2.0 is given a body as an HTML structure, important contextual information in the HTML can be lost due to the structure of the LM, which pre-trains using plain text without HTML tags, mainly through preprocessing. Based on this, we collected HTML documents and question-answering datasets to shift to the HTML domain and question-answering task domain, and performed post-training using the span corruption objective. Here, we used the long-span denoising method proposed in HTML-T5^[12] for the span length instead of the general value.

Image 3. Long Span Denoising method^[12]

In the HTML-T5 paper, it was argued that using the general span length value of 3 for HTML documents could cause meaningful information to be lost due to the small breakdown of HTML tags, so we increased the length to 8 to improve the model’s ability to fit meaningful HTML tags.

1. Data preprocessing and augmentation

As the body of KorQuAD 2.0 is given as an HTML structure, if it is used without processing, the number of tokens will increase rapidly and become inefficient in terms of resources because it contains unimportant information. This is why data preprocessing is necessary to process it into a form that the model can utilize.

For important tags such as Table and List, we replaced them with special tokens to distinguish them from words in general sentences and processed them to avoid using unnecessary head and tail information.

Spacing and paragraph marks, such as <p> <a>, were preserved so that the model can better infer the range of correct answers as it generates them. Overall, there are many short answer sentences in the questions, so the model tends to generate short answer sentences, but by keeping the paragraph marks and such in mind, we were able to see the model generate longer sentences in a contextualized way.

 

1. MonoQA: Rerank & Reader

As mentioned earlier, korQuAD2.0 requires you to infer the correct answer from a long document, unlike korQuAD1.0, which deals with short documents. Due to the structure of the Attention modules, it is not efficient to use long stretches of input, so we adopted the retrieval & reader method often used in ODQA (Open Domain Question Answering) tasks.

The retrieval & reader methodology searches for web documents that are relevant to the user’s query and uses the retrieved results as input to help the model answer the query. In a variation of this method, we viewed a long HTML document on the web and re-ranked the paragraphs containing the correct answer based on similarity to obtain the correct paragraphs, which we then used to train the reader model to generate the correct answer.

As a baseline, we used the cross-attention method with an encoder structure often used in re-rankers, which showed adequate performance in the experiments, but as the re-ranker and reader are divided into two models to learn each task, the information learned by the models is not shared, so the performance could not be improved beyond a certain level.

EXAONE Lab decided to adopt a single model structure called MonoQA^[13] to solve this problem.

Image 4. The overall framework of ORConvQA system (consisting of ConvDR & monoQA)^[13]

The MonoQA method utilizes a generative reader, such as T5, to perform both re-ranking and reading at the same time by determining if paragraphs are similar on a paragraph-by-paragraph basis and generating the correct answer if they are. With this method, the reader model also learns the information about re-ranking at the same time, which results in a slightly higher performance than the previous method.

5. Evaluation Results (Devset)

	EM	F1
Baseline	77.84	89.83
++HTML T5	79.26	90.92
++MonoQA	80.04	91.49

6. Limitation

The KorQuAD model using EXAONE LM has the advantage of being able to generate answers that do not exist in the document with parametric knowledge, but it is slower than the encoder base model as it generates all answers in an auto-regressive manner. However, many optimization libraries have recently addressed many of the limitations of the auto-regressive method, so we expect to see many improvements through research.

7. Conclusion

Through this leaderboard challenge, LG AI Research achieved the first rank in both KorQuAD2.0 and KorQuAD1.0 with a generative method using an encoder-decoder model. We think it is significant that we have consistently ranked first in the field of question answering, where encoder models are mainly used, by using the same generative pre-trained model. Furthermore, it was a good opportunity to introduce the performance of EXAONE, which we trained from scratch, to various researchers. We plan to continue our efforts to research large-scale AI model, EXAONE, so that it can boast good performance in the future.