[ACL 2024] Developing a Large Language Model with Cross-Cultural Considerations

Natural Language Processing (NLP) has been advancing at a remarkably rapid pace in recent years. Models have grown exponentially in size, to the point where we now have to spread them across multiple servers to make inferences, and the data to train them has become exponentially larger.

Additionally, NLP researchers are training these models to mimic human behavior, thoughts, and preferences. But are all people one-size-fits-all? The truth is that a person’s behavior and thoughts are deeply connected to the culture to which they belong. Understanding how others in a group think and what values they respect are crucial cultural elements that cannot be overlooked when expressing or understanding people.

LG AI Research is currently one of the few organizations in Korea capable of training and deploying LLMs from scratch. We recently released the EXAONE 3.0 7.8B model, which performs well on both Korean and English benchmarks. Let’s take a look at what academics are doing to incorporate the aforementioned cultural factors into their models, and how we at LG AI Research are thinking about and developing LLMs, at the 2nd Cross-Cultural Consideration in NLP (C3NLP) workshop at ACL 2024, a prominent academic conference in the NLP field.

Benchmarks

While there are many benchmarks available to measure the general performance of LLMs, there is a relative lack of benchmarks to measure cultural alignment. Several benchmarks were presented at the workshop to fill this gap, some of which will be introduced here.

CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models^[1]

The values that people find important vary greatly depending on the culture they belong to. While we previously focused on universal values (helpfulness, honesty, harmlessness) to emulate people, the newly released CDEval was designed to measure the six cultural dimensions from Geert Hofstede's cultural dimensions theory: power distance index, individualism vs. collectivism, uncertainty avoidance, masculinity vs. femininity, long-term orientation vs. short-term orientation, and indulgence vs. restraint.

Image 1. Evaluation results on three cultural characteristics of various LLMs^[1]

KoBBQ: Korean Bias Benchmark for Question Answering^[2]

As social bias is usually deeply rooted in each culture, the previously published Bias Benchmark for Question Answering (BBQ) is not suitable for Korean culture. Therefore, this study proposed a BBQ benchmark for Korean culture (KoBBQ). It also explained the need for a culturally diverse social bias benchmark through a comparison between KoBBQ and BBQ.

Image 2. In the United States, people think “drugs are used by people with low socioeconomic status,”
but in Korea, we think the opposite: “drugs are used by people with high socioeconomic status.”^[2]

Modeling

While it is important to measure how culturally aligned an LLM is, it is equally important to know how it is aligned. Let’s look at some research to see how academics are tackling this.

Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?^[3]

Typically, language models are trained to align with human preferences before they are used by real users. However, most alignment datasets are in English, which means that they only represent the preferences of native English speakers. If we simply translate these datasets to train the model, can we also learn the preferences of people who speak that language?

This study focused on common sense morality, which is the ability to determine whether a sentence is morally acceptable or not. For example, determining whether sentences such as “I ignored my academic advisor’s sincere advice” or “I’m addicted to gambling” are morally acceptable.

Image 3. Results of generating a CALM2 model trained with GPT3.5 Turbo,
JCM (Japanese Commonsense Morality), and ChatbotArena-JA (ChatbotArena dataset machine-translated into Japanese)^[3]

“I'm addicted to gambling” is a sentence that is generally not morally acceptable in Japan. However, as shown above, GPT-3.5 Turbo is answering without any consideration for Japanese cultural morality. Given these differences in what is and is not morally acceptable in different cultures, would training a model by simply translating a dataset that represents preferences in English-speaking cultures be a good representation of preferences in other cultures?.

Image 4. Results for JCM and ETHICS of CALM2 fine-tuned with ChatbotArena-JA^[3]

Surprisingly, the model fine-tuned with ChatbotArena-JA, a machine translation of ChatbotArena into Japanese, outperformed the JCM (Japanese Commonsense Morality) dataset. There was a decrease in the ETHICS dataset, which the authors attributed to poorer English proficiency due to the Japanese fine-tuning.

Do Multilingual Large Language Models Mitigate Stereotype Bias?^[4]

Previous studies^[5,6] have shown that multilingual models have less bias than monolingual models in BERT-like models. This study explores what happens in larger decoder-based models.

Image 5. BBQ result heatmap^[4]

It can be seen that multilingual-trained models have less bias than monolingual-trained models, and among the open-source models, German Falcon in particular performed very well.

From Personal Inconsistencies to Cultural Biases

We introduced how academia is working on learning cultural alignment for LLMs. Now, let's take a look at how we at LG AI Research are working and evolving our model through some of our key research.

Model Persona and Inconsistency^[7]

In order to use LLMs as agents, understanding the persona, such as the characteristics and social tendencies of LLMs, and understanding how performance changes with changes in prompts may be more important than the performance of the model itself. .

LG AI Research has released MODEL-PERSONA, a benchmark for measuring personas, and through extensive experiments, found that slight changes in prompts cause performance changes, and most LLMs perform significantly differently, especially depending on whether negative words are included or not.

Image 6. Four consistency comparisons for each model^[7]

While order and option showed consistency, there was a significant drop in consistency when negative words like “no” and “not” were included, or when they were replaced with words that did not contain a negative word but had the opposite meaning.

Image 7. The difference between adding and not adding different personas to the prompt^[7]

What is even more interesting is that when we add personas to the prompt, the two negation consistencies, which were significantly inconsistent before, increase with the exception of a few outliers.

Impact of persona prompting^[]

If personas help with model consistency, do they also help with model performance? We measured the performance of four models with a total of 116 personas and 2,410 questions.

Image 8. Comparison of the performance of the top 5 personas and the worst 5 personas for each model.
Excerpt from Research Presented by Moontae Lee, Advanced ML lab leader at LG AI Research, at ACL 2024

There was some variation, but no noticeable differences in performance. Interestingly, most of the models did not well when given the AI persona.

Cultural sensitivity and biases^[8]

While NLP researchers have done a lot of work to incorporate general common sense into LLMs, cultural common sense has been less explored. Will LLMs respond well when asked about cultural common sense? Will they be biased toward the cultures that are more influential?

Image 9. Limitations of LLMs on cultural common sense^[8]

Nowadays, LLMs are actively used worldwide, making it ideal to train an LLM that is not only familiar with general common sense but also with the common sense of different cultures. However, most of the known large-scale training corpora used to train LLMs are very culture-specific. We naturally hypothesized that LLMs would have a better understanding of the common sense of their own culture, and this was confirmed by the cultural common sense QA.

Image 10. Accuracy of LLMs in cultural common sense^[8]

The LLM’s QA performance on cultural common sense was generally good, but it did not perform well on questions about Iran and Kenya, with an average performance drop of about 20% on Iran. It seems that the model was less accurate on cultural common sense, which was not widely present in the training corpus.

General common sense is common sense that any culture accepts as true. For example, “water freezes when it gets cold,” or “people want to eat when they are hungry.” When these generalizations are combined with cultural context, can LLMs answer well without cultural bias?

Image 11. Accuracy of LLMs to ensure that general common sense is true in the country^[8]

We can see that the scores are relatively consistent for the Vicuna, Falcon, and GPT models, but for LLAMA2, we can see a significant drop in scores simply by providing cultural context.

This study on understanding cultural commonsense of large language models, conducted by LG AI Research and the University of Michigan, won the NAACL 2024 Social Impact Award.

Beyond performance

Just because an LLM performs well in benchmarks does not mean that the real users who interact with it will be satisfied. Each person has their own style and preferences, which are closely connected to the culture of the group in which they live.

LG AI Research is not just looking for a model that performs well on benchmarks. We are trying to think about, research, and develop a model that takes into account the cultural specificity (not limited to Korea), usage, etc. of real users through various studies from academia and related studies within LG AI Research. Stay tuned to see how EXAONE will continue to develop in terms of performance, and continue to develop in the future!

ACL Trends Review Series

#1. [ACL 2024] Emerging Trends and Key Insights in LLM Research
#2. [ACL 2024] A New Approach to Chart Understanding and Reasoning
#3. [ACL 2024] Towards efficient Large Language Models
#4. [ACL 2024] AI-generated Text Detection Research Trends
#5. [ACL 2024] Research Trends in LLM Evaluation Methods for Faithfulness and LLM Efficiency