Inclusive and Context-Aware LLMs: Trends in Dialect Handling and Multilingual Performance

The recent research in the field of Large Language Models (LLMs) has been significantly focused on enhancing their performance across diverse linguistic and cultural contexts. A notable trend is the exploration of LLMs' robustness and fairness in handling minority dialects and code-mixing scenarios, which has been previously overlooked in many benchmarks. Innovations in prompting techniques and multilingual confidence estimation are also advancing the understanding of how LLMs perform in non-English languages, particularly in low-resource settings. Additionally, there is a growing emphasis on the sensitivity of LLMs to prompts, which impacts their performance and user satisfaction. Transfer learning approaches are being explored to improve NLP systems for rarely annotated languages, leveraging linguistic similarities between languages. Overall, the field is moving towards more inclusive and context-aware models that can better serve a global audience.

Noteworthy papers include one that introduces ReDial, a dialectal benchmark for evaluating LLMs' fairness and robustness to African American Vernacular English, and another that presents ProSA, a framework for assessing prompt sensitivity in LLMs, revealing variability in performance across datasets and models.

Sources

One Language, Many Gaps: Evaluating Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks

Code-Mixer Ya Nahi: Novel Approaches to Measuring Multilingual LLMs' Code-Mixing Capabilities

Sampling Strategies for Creation of a Benchmark for Dialectal Sentiment Classification

Findings of the WMT 2024 Shared Task on Chat Translation

ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs

MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models

Exploring transfer learning for Deep NLP systems on rarely annotated languages

Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings

NLIP_Lab-IITH Multilingual MT System for WAT24 MT Shared Task

HEALTH-PARIKSHA: Assessing RAG Models for Health Chatbots in Real-World Multilingual Settings

Built with on top of