What happened
arXiv:2608.07862v1 Announce Type: new Abstract: Existing safety evaluation datasets for large language models (LLMs) predominantly focus on English and Western contexts, often overlooking the linguistic diversity and cu
Analysis and impact
目前，主流的大语言模型安全评估数据集，如用于测试模型是否会产生有害、偏见或违规内容的题库，绝大多数基于英语构建。这导致模型在处理非英语、尤其是拥有复杂社会文化背景的语言时，其安全表现缺乏有效的衡量标准。SurakshaEval 正是针对这一缺陷而设计，它专门聚焦于印度的多种语言，构建了一个多语言的安全评测框架来源。
该基准测试的核心在于其“印度中心”的视角。它不仅覆盖了多种印度语言，更重要的是，其测试用例的设计融入了印度特有的文化规范、社会禁忌和法律环境。这意味着，一个在英语安全测试中表现良好的模型，在 SurakshaEval 中可能会暴露出因文化差异而导致的安全漏洞，例如对特定宗教或种姓议题处理不当。研究团队通过这种方式，试图推动大语言模型在全球化部署时，能够真正实现本地化的安全对齐，而非简单地将英语世界的安全准则套用到其他语言上。
从技术层面看，SurakshaEval 的发布为多语言模型的开发者提供了一个新的衡量维度。原文未提供该数据集的具体规模、覆盖语言种类以及详细的评估方法，但它的出现本身，标志着 AI 安全研究正从单一语言主导，向着更具包容性和文化敏感性的方向发展。这对于在印度等拥有庞大且多样化语言用户的市场中部署 AI 应用，具有重要的实践指导意义。
References
Original source text
arXiv:2608.07862v1 Announce Type: new Abstract: Existing safety evaluation datasets for large language models (LLMs) predominantly focus on English and Western contexts, often overlooking the linguistic diversity and culturally grounded safety risks present in other languages. To address this gap, we introduce SurakshaEval, a novel safety benchmark composed of human-written prompts spanning real-world scenarios, explicitly designed for ten major Indian languages - Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, and Telugu, along with English. SurakshaEval includes both generic prompts common across India and region- and language-specific prompts that capture localized sociocultural sensitivities. We benchmark a broad range of state-of-the-art LLMs on SurakshaEval, establish baseline safety performance, and identify recurring failure modes, including over-refusal, missed detection of implicit bias, and insufficient contextual awareness in regionally sensitive settings. Our results show that even strong multilingual LLMs struggle to reliably meet nuanced safety requirements when operating in Indic languages, particularly in native scripts. These findings highlight the urgent need for safety evaluation frameworks that incorporate region-specific data and structured assessment protocols, enabling the development and deployment of AI systems that operate securely, ethically, and in alignment with diverse societal values. Our code and data are available at https://github.com/debobanerjee/SurakshaEval. Warning: This paper contains text that may be offensive or unsafe.