Testing Insurance Software the Hard Way: LLM Agents Lift Black Box Test Reliability from 39 Percent to 84 Percent
Insurance software is one of the hardest things in the world to test: the rules are buried in legacy systems, the logic changes with each jurisdiction, and there is no single right answer to compare against. The ASI Lab's ACAT framework puts LLM agents inside a strict black box loop of planning, running, diagnosing and repairing browser tests. Across 50 real broker workflows it lifted how often generated tests could be re run, without ever touching the code behind the screen.
Insurance software is one of the hardest things in the world to test: the rules are buried in legacy systems, the logic changes with each jurisdiction, and there is no single right answer to compare against. The ASI Lab's ACAT framework puts LLM agents inside a strict black box loop of planning, running, diagnosing and repairing browser tests. Across 50 real broker workflows it lifted how often generated tests could be re run, without ever touching the code behind the screen.
](/news/img/agentic-constrained-testing.JPG)
Why it matters
Regulated domains break the classic assumptions of software testing. In a payments system you can predict the output and compare it. In an insurance platform, correctness depends on whether a quotation satisfies an actuarial rule, whether a claim workflow respects a jurisdictional regulation, and whether five separate components agree with each other. That is the insurance oracle problem, and no amount of script automation solves it. Black box testing teams still interpret requirements by hand, design test data by hand, and judge correctness by hand. This paper names that difficulty formally and then attacks it head on.
The team's insight is that an LLM agent should not be given a blank cheque to click around a system. Instead it should be fenced in by artifacts: the test plan it wrote, a set of generation rules, and a growing code book of previously validated repair patterns. Every click must flow through a tool server, never direct browser access, so the black box stays black. The result is a testing pipeline that is not just smarter, but auditable and reproducible, which is exactly what regulators and quality assurance auditors ask for. On 50 production derived use cases with eight human testers as the baseline, code book supported ACAT raised executability from 0.39 to 0.84 and found a genuinely different family of bugs.
Key findings
- Use cases The industrial study ran on 50 real world system level use cases drawn from live broker operational workflows, spanning 9 functional categories.
- Complexity spread Use cases were graded into 14 low, 21 medium and 15 high complexity workflows, all executed under an organizational administrator role.
- Headline gain Re run success, called executability, climbed from 0.39 under eight human testers to 0.58 for ACAT and 0.84 for the code book assisted ACAT+.
- Hardest tier On high complexity use cases, plain ACAT reached a perfect 1.00 executability, and ACAT+ held 0.86 across all three complexity tiers.
- Faster localization Median failure depth for high complexity cases fell from 0.44 under human testing to 0.14 under ACAT+, meaning broken tests reveal themselves early instead of stalling deep in a run.
- Bug discovery ACAT+ exposed 9 system level failures across 50 scripts, a 0.18 failure rate, versus 35 failures across 400 human written scripts at 0.09.
- Complementary findings Human testers caught bugs in 21 use cases and ACAT+ in 9, with only 3 overlapping cases, showing the two approaches see different parts of the system.
- Failure taxonomy ACAT+ failures concentrated in CRUD and UI frontend behaviour, while humans dominated on error handling, access control and observability issues.
- Architecture choice The pipeline sits on the Model Context Protocol, so every browser action is mediated by a tool server and the system under test is never exposed to the agent directly.
- Repair loop A Repair Agent classifies each failure as a test fault, a system defect or an environment anomaly, then either fixes the script within a bounded attempt budget or files a structured system bug report.
39 percent to 84 percent, with zero access to source code
- Code book assisted agents re ran successfully 0.84 of the time, more than double the human tester baseline of 0.39.
- Plain agents found no system bugs until the code book stabilised execution, then surfaced 9 distinct system level failures.
- Only 3 of 27 failing use cases overlapped between humans and agents, so the two approaches genuinely complement each other.
What it means for practitioners
For QA leads in banking and insurance, the practical message is that agentic testing belongs next to your exploratory testers, not instead of them. ACAT+ is a reproducible execution and repair layer: it keeps brittle Playwright scripts alive through interface churn, localises breakage quickly, and files structured bug reports. Human testers stay free to do what they do best, reading the domain, spotting the missing access control check, noticing when an error message is unhelpful.
Two design rules carry most of the weight. First, keep the agent bounded: let it reason, but route all execution through mediated tools so nothing happens off the record. Second, invest in the code book. The jump from 0.58 to 0.84 executability came entirely from reusing validated implementation patterns such as locator scoping, prerequisite navigation and synchronisation fixes. That artifact is small, auditable, and grows with every fix, and it is the piece most teams are missing today.
Related reading
Get the paper & cite it
Official citation: Hi Kuen Yu, Jacky W. Keung, Man On Wong, Yicheng Sun, Rachel Samantha Chandra, Yihan Liao, Xiaoxue Ma (2026). Artifact-Constrained Agentic Testing for Black-Box System Testing under Actuarial and Regulatory Constraints.. COMPSAC. DOI: 10.1109/COMPSAC69091.2026.00147.