The QASA Approach for Utilizing Language Models in Scientific Knowledge Reasoning

1. Instruction

LG AI Research proposes the advanced question answering on scientific articles (QASA) approach and benchmark for reasoning on scientific questions^[8]. Our intensive think-aloud study reveals that reading scientific articles not only raises surface questions but also induces deep questions that require complex reasoning. To answer such questions, a cognitive reasoning process is necessary: consisting of associative thinking to collect relevant pieces of knowledge and logical reasoning to compose grounding based on evidential rationales. While previous QA tasks and datasets^[1][2] neither require rich retrieval of associative information nor compose multiple pieces of evidence, we argue that far more sophisticated efforts are required.

Specifically, in constructing benchmarks for scientific question answering, we propose the following distinctions. We design a schema for questions based on the think-aloud study, categorizing them into three types: Surface, Testing, and Deep. We then collected various question-answer pairs from authors of scientific papers and AI experts. Furthermore, we requested both readers and authors of the papers to pose questions while reading the full papers. We also requested that they provide diverse rationale within the paper in response to those questions, then compose comprehensive final answers. The QASA benchmark includes 1798 question-answer pairs in which expert readers in the AI/ML domain pose questions and other experts or authors provide answers.

2. QASA Approach

We propose a new QA approach to answer deep and complex questions over scientific papers. The core idea of our approach is to decompose the QA task into three sub-tasks: (1) Associative Selection, (2) Rationale Generation, and (3) Systematic Composition. We finetune large language models (LLMs) on a mixture of the three sub-tasks through instruction tuning. As shown in [Figure 1], we insert the input in each task, and then get the final answer after three steps.

Figure 1. Multistep reasoning in QASA methodology
(The language model uses task-specific instructions for each step)^[8]

The above step-by-step reasoning will be explained in more detail below.

STEP1. Associative Selection

Research papers typically consist of multiple paragraphs (approximately 40 to 80 paragraphs). This first step is to extract associative knowledge from the paragraphs, corresponding to a question. In other words, when a question and K paragraphs are given, it selects the relevant paragraphs that contain the answer or supporting evidence for the question. While the selection in previous open-domain QA approaches is to extract answerable paragraphs that contain a short-form answer, we expand this to capture evidence, which is composed of multiple rationales and answers.

STEP2. Rationale Generation

In the second rationale generation step, we generate an evidential rationale on each selected paragraph, which could be part of a final long-form answer in the next step. According to the previous work^[3] on the discourse structure of answering complex questions, the evidential rationale can be the (1) main answer (main content of the answer which directly addresses the question), (2) elaboration (sentences which elaborate on the main answer), and (3) auxiliary information (background knowledge that could be helpful to the user). In the same way, the rationales generated in this step also could contain a main answer, explanation, and auxiliary information.

STEP3. Systematic Composition

The goal of this last composition step is to provide concise and readable information to users by systematically composing all the evidential rationales into a final comprehensive answer. Assuming that the answer is composed of multiple rationales, we aim to preserve all the rich rationales in the final answer, excluding duplicate text. The answer based on the given passage can be viewed as a comprehensive explanation of the question.

Details about the learning and data for each module can be found in this paper.

3. Experiment

3.1 Automatic Evaluation

For automatic evaluation, we considered pre-trained language models (open-source checkpoints or APIs) and fine-tuned language models. We compare the following language models.

1. Models: T5 (Version 1.1, LM-Adapted)^[4], T0^[5], GALACTICA^[6], InstructGPT (text-davinci-003), Flan-T5^[7]

 

[Table 1] shows the evaluation of the model on three subtasks. To independently evaluate each module, we provide oracle (or gold) contexts. As shown in [Table 2], for the full-stack QA, consider the results of previous task as input to the next task sequentially, which could propagate the errors of the previous steps.

As evaluation metrics, we used (Precision, Recall, and F1 scores) for selection task, and (Rouge-1, Rouge-2, and Rouge-L scores) for the other generation tasks.

 

Table 1. Performance of each system on QASA’s subtasks measured by Precision, Recall, F1, and ROUGE score^[8]

Table 2. Performance of each system on QASA’s full-stack tasks^[8]

 

Q: Which pre-learned language model shows the highest performance?
A: Among the pre-learned language models, InstructGPT (175B) outperformed other models. Especially in the rationale-generation task, it shows the best performance among all models. Among T5-based LMs, the number of downstream tasks used during training had a significant impact on the performances in full-stack QA, resulting in the order T5 < T0 < Flan-T5 in terms of results.

Q: Which fine-tuned language model shows the highest performance?
A: When comparing fine-tuned T0, T5, and Flan-T5, these models showed relatively minor performance differences across the three subtasks. However, in full-stack QA tasks, Flan-T5 outperformed all other LMs, and it even performed better than the latest model, InstructGPT (175B). Based on this observation, we suggest that the fine-tuned Flan-T5 model would serve as a good testbed for our QASA benchmark.

Q: Does our task indeed need rationale-generation?
A: Our three-step approach involves generating rationales and then composing them into the final answer. However, as an alternative, it is possible to skip the rationale generation step and directly generate the final answer from selected paragraphs. To compare this, we reported the results (“w/o Rationale Gen”) skipping the rationale generation step in [Table 2]. Skipping the second step showed a decrease in performance, highlighting the importance of the rationale generation step.

 

3.2 Human Evaluation

Although automatic metrics can measure crucial aspects of our task, they are not guaranteed to closely approximate the judgment of humans, whose satisfaction is an overarching goal of a QA system. Therefore, we conducted a pairwise evaluation scheme (AB Testing) between InstructGPT and our model, using the following four criteria:

1. Four criteria: Groundedness, Completeness, Specificity, Fluency.

 

The scoring system awards one point for a win and half a point for a tie in pairwise comparisons. The annotations were collected on 100 QA pairs by 9 experts.

 

Figure 2. Ours vs InstructGPT: A/B test results^[8]

 

According to the A/B test evaluation results in [Figure 2], our QASA approach shows higher performance in terms of completeness and groundedness, compared to that from InstructGPT. On the other hand, regardless of the reliability of the generated text, InstructGPT’s answers tend to be more fluent and specific.

 

4. Conclusion

The searching process for information requires a series of efforts, from finding relevant literature to reading and synthesizing the selected information. In particular, as scientific literature is expanding rapidly, there is a need for innovative tools for knowledge acquisition and reasoning. Generative AI can be an innovative tool, but there is still a lack of studies that focus on scientific literature.

Our work proposes a new benchmark and new approach. We have constructed various levels/types of question-answering benchmarks through experts and paper authors and addressed them through a full-stack reasoning process consisting of three multi-step methodologies: Associative Selection, Rationale Generation, and Systematic Composition.

Several experimental results have demonstrated that finetuned Flan-T5 model could serve as the best testbed for QASA. Through this study, it has been confirmed that we can move beyond traditional searching processes and harness language models to derive scientific reasoning and insights. In future research, we plan to continue designing LM-based approaches that integrate various external tools, including search engine, to assist in scientific knowledge exploration.

▶ QASA: Advanced Question Answering on Scientific Articles (Link)