FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios
| Authors | Yutao Hou et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2605.00706 |
| Download | |
| Categories | cs.CL |
Abstract
Large language models (LLMs) are increasingly applied in financial scenarios. However, they may produce harmful outputs, including facilitating illegal activities or unethical behavior, posing serious compliance risks. To systematically evaluate LLM safety in finance, we propose FinSafetyBench, a bilingual (English-Chinese) red-teaming benchmark designed to test an LLM's refusal of requests that violate financial compliance. Grounded in real-world financial crime cases and ethics standards, the benchmark comprises 14 subcategories spanning financial crimes and ethical violations. Through extensive experiments on general-purpose and finance-specialized LLMs under three representative attack settings, we identify critical vulnerabilities that allow adversarial prompts to bypass compliance safeguards. Further analysis reveals stronger susceptibility in Chinese contexts and highlights the limitations of prompt-level defenses against sophisticated or implicit manipulation strategies.
Engineering Breakdown
Plain English
This paper introduces FinSafetyBench, a bilingual benchmark with 14 subcategories designed to evaluate whether large language models can safely refuse harmful requests in financial contexts. The benchmark is grounded in real-world financial crime cases and compliance standards, testing both general-purpose and finance-specialized LLMs against three types of adversarial attacks. The authors discovered critical vulnerabilities in current LLMs that allow adversarial prompts to bypass compliance safeguards, revealing that even specialized financial models can be manipulated into producing outputs that facilitate illegal activities or unethical behavior. This work addresses a growing compliance risk as financial institutions increasingly deploy LLMs for customer-facing and internal applications.
Core Technical Contribution
The core contribution is FinSafetyBench itself—the first systematic bilingual (English-Chinese) red-teaming benchmark specifically designed for evaluating LLM safety in financial compliance scenarios. Unlike generic safety benchmarks that test for general harmfulness, this benchmark is grounded in 14 specific financial crime and ethics violation categories derived from real-world cases and regulatory standards, making it directly applicable to production financial systems. The authors also introduce a structured evaluation methodology that tests LLMs under three representative attack settings, allowing systematic identification of failure modes in both general and domain-specialized models. This fills a critical gap: while general LLM safety has been studied, financial compliance safety—with its specific legal and regulatory requirements—had no standardized measurement tool.
How It Works
FinSafetyBench operates as a red-teaming framework with three main components: (1) a curated benchmark dataset organized into 14 subcategories spanning financial crimes (e.g., money laundering, fraud, insider trading) and ethical violations (e.g., manipulation, discrimination), grounded in real financial crime cases and compliance standards; (2) three representative attack settings that simulate adversarial prompting techniques—presumably including direct requests, obfuscated requests, and jailbreak attempts; (3) an evaluation methodology that tests both general-purpose LLMs (like GPT-4, Llama) and finance-specialized models against these prompts, measuring their ability to refuse harmful requests. The input is a prompt designed to trick the model into producing a harmful financial output; the transformation involves the LLM processing the prompt under different adversarial conditions; the output is labeled as either safely refused or unsafely complied, with results aggregated across categories. The bilingual nature (English-Chinese) ensures the benchmark captures regional compliance differences and language-specific vulnerabilities.
Production Impact
For engineers deploying LLMs in financial services, this benchmark provides a concrete, standardized way to measure compliance safety before production deployment—directly addressing regulatory risk. Financial institutions can use FinSafetyBench to: (1) audit which attack vectors their models are vulnerable to, (2) prioritize safety improvements for the highest-risk violation categories, and (3) compare safety profiles across different LLMs to inform procurement decisions. The bilingual aspect is critical for global financial institutions operating in both English and Chinese markets, as it tests language-specific vulnerabilities that might not be caught by single-language evaluations. However, adoption requires careful integration: teams need to treat this as a pre-deployment safety gate (not a post-deployment monitor), likely adding 1-2 weeks to evaluation cycles, and they must invest in fine-tuning or RLHF-based safety improvements after identifying failures. The 14 subcategories enable granular auditing—you can identify that your model fails specifically on "insider trading requests" but handles "fraud" safely, allowing targeted remediation rather than blind safety improvements.
Limitations and When Not to Use This
The paper does not address how safety can be improved once vulnerabilities are identified—it is a diagnostic tool, not a solution tool. It assumes that adversarial attacks fall into the 14 predefined categories, but novel attack vectors outside these categories may exist in production (e.g., regulatory arbitrage between jurisdictions, emerging financial crime patterns). The benchmark is limited to refusal-based safety (i.e., saying no to bad requests) and does not evaluate whether models provide accurate, helpful compliance information when answering legitimate financial questions—a false-positive rate problem where over-cautious models hurt usability. Additionally, the paper does not discuss how quickly the benchmark becomes stale as adversarial techniques evolve, or how to maintain it as new financial crimes and regulations emerge. Finally, while the benchmark tests attack resilience, it does not evaluate the underlying safety mechanism used by models (e.g., RLHF, constitutional AI, rule-based filters), so two models may refuse equally well but for entirely different reasons, limiting insights into robust solutions.
Research Context
This paper builds on the broader LLM safety literature (e.g., work on jailbreaks, adversarial prompting, RLHF-based safety alignment) but applies it specifically to the financial domain—a high-stakes application area with regulatory requirements that generic safety research hasn't addressed. It extends red-teaming benchmarking methodology (similar to frameworks like ToxiGen or JailBreakBench) into the financial compliance space, creating a domain-specific evaluation tool analogous to how FinQA or FinBench evaluate financial understanding. The bilingual design reflects growing interest in multilingual AI safety, addressing the gap that most safety benchmarks focus on English. This work opens research directions in: (1) developing targeted safety improvements for finance-specific harms, (2) understanding why finance-specialized models aren't more robust to these attacks, and (3) creating automated safety vetting pipelines for regulated industries beyond finance.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::