Artifact-Constrained Agentic Testing for Black-Box System Testing under Actuarial and Regulatory Constraints.
Abstract
System-level testing of insurance software systems is challenging due to long-lived legacy architectures, distributed actuarial logic, regulatory-driven conditional behavior, and exception-heavy workflows, where correctness is defined by compliance with evolving domain constraints rather than deterministic outputs. Existing approaches either focus on artifactcentric validation, rely heavily on human-driven execution and diagnosis, or generate test artifacts in a one-shot manner without sustained adaptation during execution, limiting robustness and reproducibility at the system level. This paper proposes ArtifactConstrained Agentic Testing (ACAT), a multi-agent framework that structures black-box system testing as a closed-loop process of planning, execution, diagnosis, and repair, in which LLMbased agents operate under explicitly bounded capabilities and interact with the system under test exclusively through toolmediated execution. We evaluate ACAT through a controlled industrial study on an insurance broker management system using 50 real-world use cases. The results show that artifactconstrained agentic testing significantly improves test executability and execution stability compared to human-driven testing, while exposing a complementary subset of system-level failures. These findings suggest that agent-assisted testing can enhance system-level automation in complex, regulated software systems without replacing human expertise. Index Terms-Black-Box System Testing, Multi-Agent Frameworks, Automated End-to-End Testing, Large Language Model, Insurance Technology I. I NTRODUCTION Insurance-related software systems pose persistent challenges for system-level testing because they combine longlived legacy architectures [1]-[3], distributed actuarial logic, and regulatory-driven conditional behavior [4]-[6]. Unlike transactional systems with deterministic input-output relationships, correctness in insurance platforms depends on whether components satisfy actuarial rules, regulatory invariants, and business constraints [2], [4], [6]. Thus, system-level testing emphasizes constraint satisfaction and cross-component consistency rather than fixed expected outputs [5