1st Place in EHRSQL 2024: Practical Text-to-SQL System for Electronic Health Records

LG AI Research and KAIST received 1st place in the EHRSQL shared task at the 6th Clinical Natural Language Processing Workshop (ClinicalNLP 2024) at NAACL 2024. Our model accurately translates natural language questions into SQL queries for Electronic Health Records (EHRs) while reliably abstaining from answering unanswerable questions. This accomplishment showcases potential of LG AI Research and KAIST to improve healthcare decision-making through an accurate and practical text-to-SQL system.

---

Artificial intelligence (AI) is transforming the healthcare industry, offering solutions to enhance patient care, clinical research, and administrative efficiency. From early disease detection to personalized treatment recommendations, the ability of AI to analyze and interpret vast amounts of data is revolutionizing the way healthcare is delivered. One such critical application is in the accessibility of Electronic Health Records (EHRs), where AI models enable healthcare professionals to quickly extract valuable insights from complex datasets, thus improving clinical decision-making and, ultimately, patient outcomes.

Figure 1. Workflow of a reliable text-to-SQL model for Electronic Health Records (EHRs)^[2].
Natural language questions from healthcare professionals are translated into
SQL queries and then executed against an EHR database.
Answers are provided based on the query results.

The ability to convert natural language queries into SQL statements using text-to-SQL models has revolutionized the way healthcare professionals can access and interact with EHRs. By leveraging advancements in AI, particularly large language models (LLMs), these systems have become increasingly proficient at translating complex medical inquiries into accurate SQL queries, allowing clinicians without SQL expertise to retrieve relevant information efficiently.

Despite these significant advancements, one critical challenge in the healthcare sector is ensuring the reliability of text-to-SQL models. Misinterpreting complex or ambiguous queries can lead to misinformation, which is unacceptable in clinical settings. Therefore, it is essential to equip these systems with mechanisms to accurately detect unanswerable questions or uncertain predictions, thereby preventing potential errors in healthcare decision-making.

To rise to the challenge, the 6th Clinical Natural Language Processing Workshop (ClinicalNLP 2024) organized a shared task entitled “EHRSQL: Reliable Text-to-SQL Modeling on Electronic Health Records.” The goal is to develop a reliable text-to-SQL model for an EHR database that can accurately translate questions into SQL queries while avoiding incorrect responses. The questions cover a wide range of topics, including patient demographics, vital signs, and survival rates for diseases.

LG AI Research and KAIST participated in the shared task and received 1st place with our practical text-to-SQL system designed for EHRs. By leveraging a self-training strategy and comprehensive filtering mechanisms, the suggested model accurately translates natural language questions into SQL queries while reliably identifying and abstaining from answering unanswerable questions. The achievement highlights the potential of this system to revolutionize healthcare decision-making through more accurate and reliable text-to-SQL translations. By enhancing the reliability of these systems, we can empower healthcare professionals with accessible and trustworthy tools, further bridging the gap between technology and clinical practice.

1. Task: EHRSQL

Electronic Health Records (Electronic Health Records, EHRs)

Electronic Health Records (EHRs) are comprehensive relational databases that document the entire medical histories of patients within healthcare facilities. They capture various aspects of patient care, from admission and diagnosis to treatment and discharge, and serve as a vital source of clinical data. However, accessing this wealth of information requires proficiency in query languages like SQL, limiting its usability for healthcare professionals who lack such technical expertise.

EHRSQL

EHRSQL is a specialized text-to-SQL dataset designed to improve access to EHRs by translating natural language questions into SQL queries ^[1]. Constructed from the input of 222 hospital staff members, including physicians, nurses, and insurance review teams, the dataset reflects a wide range of information needs in a hospital environment. Questions cover topics like patient demographics, vital signs, and survival rates of diseases. It serves as a practical benchmark for developing question-answering models on structured EHR data, bridging the gap between text-to-SQL research and real-world deployment in healthcare.

Figure 2. Sample utterances in EHRSQL ^[1]

Figure 3. Demographics of the 222 respondents of EHRSQL by hospital departments
and the years of experience^[1]

Evaluation Metric

One of the most critical objectives for text-to-SQL models in healthcare is reliability, as misinformation can have serious consequences. EHRSQL aims for accurate SQL generation and reliable answers only. The model should abstain from providing incorrect answers by identifying unanswerable questions or uncertain predictions. For example, questions that lack relevant data or require external knowledge should be correctly identified as unanswerable to ensure the reliability of clinical decision-making.

To assess the accuracy and reliability of text-to-SQL models, the evaluation metric called Reliability Score (RS) was introduced^[3]. RS measures the model's ability to generate accurate SQL queries while abstaining from answering unanswerable questions ($Q_{una}$).

It aims to : 

RS is calculated based on whether the model chooses to generate SQL or abstain, and on execution accuracy ($Acc$). The term “c” determines the weight of the penalty imposed. RS can be used as an essential tool for developing and evaluating reliable text-to-SQL models in healthcare.

Figure 4. The formal definition of RS for a single data instance^[3].
g(x) denotes whether the model decides to generate SQL (g(x)=1) or abstains from answering the given question (g(x)=0).
Acc(x) represents whether the answer matches the ground truth. The term “c” represents the penalty weight.

2. Approach: Self-Training

To tackle the challenge of building a reliable text-to-SQL model for EHRs, we developed an approach that leverages a two-stage self-training paradigm. Our strategy involves training an initial seed model, using it to identify unanswerable questions, and then refining the model by retraining it with an augmented dataset. Finally, we apply filtering techniques to ensure the quality of the generated SQL queries.

This can be examined in more detail in Figure 5. The model is initially trained using the training set. A SQL query (or null) is then generated for each sample in the test set using the trained model. Subsequently, we select K null samples and add them to the training set, resulting in a null-augmented training set. This augmented dataset is then used to train the final model.

Figure 5. The Training process and SQL query generation of our approach

Self-Training

Self-training is a semi-supervised learning technique in which a model retrains itself using its own predictions on unlabeled data. While unlabeled data is typically abundant, labeled data is often scarce and expensive to obtain. By generating pseudo-labels and incorporating them into the training process, self-training leverages the large amount of unlabeled data to improve the model's accuracy and reliability.

Imagine deploying a real-world text-to-SQL system for EHRs in a hospital. While it's relatively easy to collect natural language queries typed by users like doctors and healthcare providers, matching them with accurate SQL statements is more challenging. Often, the only resources available are predictions generated by your own model. By creating pseudo-labeled unanswerable questions and retraining with this additional data, self-training can improve the model's ability to distinguish between answerable and unanswerable questions, resulting in more reliable SQL query generation.

Seed Model Training

We started by fine-tuning a seed model using the original training dataset, which included both answerable and unanswerable questions. This initial training gave the model a foundational understanding of the question types in the EHRSQL dataset.

Self-Training with Pseudo-Labeled Unanswerable Questions

The seed model was then used to identify unanswerable questions from the test set, effectively labeling them as unanswerable. These pseudo-labeled questions were incorporated into the original training set, creating an augmented dataset. The model was retrained with this data, improving its ability to differentiate between answerable and unanswerable questions.

Filtering Strategy

To refine the model's predictions, we applied a two-stage filtering process.

Combining self-training with this filtering strategy improved the model's reliability in distinguishing answerable from unanswerable questions, leading to more accurate SQL query generation.

3. Results

Our model PLUQ (Pseudo-Labeled Unanswerable Questions) leverages the self-training paradigm to improve the reliability of text-to-SQL models for EHRs. It achieved the highest performance in RS(10), the primary metric, earning us first place on the official leaderboard of the EHRSQL 2024 shared task. Notably, the minimal difference between RS(0) and RS(10) scores demonstrated the model's ability to minimize penalties by accurately categorizing uncertain outcomes in both answerable and unanswerable questions as "unanswerable." Despite a slight decrease in scores during the test phase compared to the development phase, PLUQ consistently outperformed other models across all RS metrics, ultimately securing our victory.

Figure 6. Results of the development and test phases on the official leaderboard. The best results are highlighted in bold.
The primary metric is RS(10) in this shared task.

4. Summary

The proposed model leveraged a two-stage self-training strategy and comprehensive filtering mechanisms to distinguish between answerable and unanswerable questions, leading to more reliable SQL query generation for Electronic Health Records. This approach resulted in the highest performance in the Reliability Score, ultimately awarding us first place in the EHRSQL 2024 shared task. By combining accuracy with reliability, LG AI Research and KAIST demonstrate the potential to transform healthcare decision-making through practical and dependable text-to-SQL systems.

LG AI Research plans to continue our research to further improve the performance and reliability of the Text-to-SQL system and LLMs. To this end, LG AI Research will focus on enhancing our ability to handle complex medical datasets, conducting validation experiments in real-world medical settings, and integrating other pre-trained models. Ultimately, LG AI Research expect that AI will strengthen the medical field, improving patient outcomes and the efficiency of medical practices.

▶[NAACL 2024] EHRSQL 2024: Self-Training Large Language Models with Pseudo-Labeled Unanswerable Questions for a Reliable Text-to-SQL System on EHRs^[5]