SysTradeBench: An Iterative Build-Test-Patch Benchmark for Strategy-to-Code Trading Systems with Drift-Aware Diagnostics.
Abstract
Large Language Models (LLMs) are increasingly used as quantitative research copilots to translate natural-language strategy specifications into executable trading code. Yet most existing evaluations either focus on static financial knowledge or summarize performance with a single profitability metric, leaving a gap for benchmarking strategy-to-code trading systems as governed, auditable software. We introduce SysTradeBench (SysTB), an iterative build-test-patch benchmark that evaluates LLM-generated trading systems under drift-aware diagnostics. Given a standardized Base Strategy Doc and frozen semantics, each model must produce (i) a strategy card, (ii) executable code, and (iii) mandatory audit logs. A sandboxed harness runs determinism and anti-leakage checks, detects rule drift across iterations, and returns evidence bundles to support constrained patches. SysTradeBench reports multidimensional scorecards (D1: Spec Fidelity, D2: Risk Discipline, D3: Reliability, D4: OOS Robustness Indicators) and cost-effectiveness signals. We evaluate 17 models across 12 strategies: top models achieve ≥91.7% validity with 7.29-7.85 scores, but evidence-driven iteration induces code convergence (95.4% similarity by Iter2). These findings suggest that LLM iteration complements rather than replaces human Quantitative Researcher (QR) governance: LLMs excel at rapid prototyping and shallow bug fixes, while QR oversight remains essential for critical strategies requiring solution diversity and ensemble robustness. CCS Concepts • Information systems → Data mining; • Computing methodologies → Machine learning; • Mathematics of computing → Probability and statistics. Keywords datasets and benchmarks; large language models; code generation; strategy-to-code; auditability; trading system 1 Introduction Large Language Models (LLMs) have evolved from passive text assistants into tool-using systems that can retrieve information, write executable code, and perform multi-step decision making [1-6]. ∗ Corresponding author. Code repository: https://github.com/YgcCoder/SysTB This shift is particularly visible in software tasks: modern LLMs can generate code, follow API contracts, and i