HGARena Teaches AI Agents to Fix Real GitHub Bugs on a Budget
Real GitHub issue resolution is a different game from writing a single function: the system must read a bug report, hunt through thousands of files, edit across several of them, and prove the fix with executable tests. HGARena, a new role structured multi agent LLM framework from ASI Lab, sets up a lightweight typed graph over files, symbols, tests and edits, then runs a submitter, reviewer and executor loop under a hard 20 call budget. On the full 300 task SWE bench Lite benchmark it resolves 25.3 percent of real GitHub issues with GPT 4o, staying competitive with much heavier agent stacks.
Real GitHub issue resolution is a different game from writing a single function: the system must read a bug report, hunt through thousands of files, edit across several of them, and prove the fix with executable tests. HGARena, a new role structured multi agent LLM framework from ASI Lab, sets up a lightweight typed graph over files, symbols, tests and edits, then runs a submitter, reviewer and executor loop under a hard 20 call budget. On the full 300 task SWE bench Lite benchmark it resolves 25.3 percent of real GitHub issues with GPT 4o, staying competitive with much heavier agent stacks.
![![HGARena overview: task instance, graph retriever with four stages, and the submitter, reviewer and executor arena loop with its typed graph legend]](/news/img/hgarena.JPG)
Why it matters
Most public coding benchmarks ask an AI to fill in a function in isolation. Real maintenance work does not work that way. A bug report arrives as plain language, the evidence is scattered across issue threads, source files, function bodies, call sites, failing tests and stack traces, and the only authority on whether a fix is correct is the test suite. That is a search problem over a huge and messy edit space, and any weak choice of context burns an attempt. HGARena reframes repository level issue resolution exactly this way, as a budgeted search problem, so a software engineering agent must decide not only what patch to write but what evidence to look at next.
The framework treats the repository as a typed graph rather than a pile of text chunks. Files, symbols, tests and patch hunks become distinct node types, joined by definition, import, call, coverage and touch edges. Because failing tests and past edits sit in the graph as first class signals, the system can follow a failure from a test to the symbol it covers and onward to the code that symbol calls, which is exactly what a human engineer does by hand. On top of that, HGARena makes the reviewer a guide rather than a judge. LLM as a judge is known to be unstable, so reviewers here only point out likely failure causes and promising follow up edits, while Docker test execution remains the sole acceptance oracle. That combination is what makes the review loop trustworthy and reproducible in principle.
Key findings
- Resolve rate HGARena resolves 25.3 percent of the 300 SWE bench Lite tasks using GPT 4o, with every result confirmed by the official Docker test harness.
- Calls over SWE agent It beats SWE agent by 7.3 percentage points and AutoCodeRover by 6.3 percentage points on the same 300 task benchmark.
- Hard budget Each task runs under a fixed budget of 20 LLM API calls, and the average successful trajectory uses 13.0 calls.
- Iteration is the engine One shot generation solves only 5.3 percent of issues, while execution guided iteration lifts that to 24.7 percent, making iteration the main driver of repair ability.
- Graph halves token cost Adding the typed RepoGraph cuts token use from 410,456 to 211,299 per task, roughly a 49 percent reduction, while holding resolve rate near 24 percent.
- Best accuracy and cost balance HGARena reaches the highest resolve rate of 25.3 percent among the controlled variants at 204,995 tokens, less than half the tokens of free form iteration.
- Consistent across backbones Resolve rate ranges from 24.3 to 27.7 percent across GPT 4o, Claude Sonnet 4.5 and Gemini 2.5 Pro, a mean of 25.8 percent and a coefficient of variation of just 6.8 percent.
- Cheapest strong backbone Claude Sonnet 4.5 posts the best result at 27.7 percent while using only 143,658 tokens and 11.5 calls per task.
- Graph schema RepoGraph uses only 4 node types and 5 typed edge relations, built cheaply with Tree sitter parsers instead of heavyweight whole program analysis.
- Reviews in one call Three reviewer perspectives, failure analysis, patch validity and test alignment, are produced in a single LLM call to keep the budget tight.
One typed graph, three roles, twenty calls
- Builds a lightweight heterogeneous RepoGraph linking files, symbols, tests and edits with five typed relations
- Retrieves graph grounded context through lexical recall, graph reranking and conservative typed expansion
- Runs submitter, reviewer and executor under a 20 call budget with Docker tests as the only oracle
- Resolves 25.3 percent of SWE bench Lite with GPT 4o, ahead of SWE agent and AutoCodeRover
- Cuts iteration token cost by about half compared with free form iterative repair
- Holds a stable 25.8 percent mean resolve rate across three commercial LLM backbones
What it means for practitioners
For teams building software engineering agents, the practical lesson is that iteration matters far more than prompt polish, but unbounded iteration quietly drains your API budget. Adding a cheap structural index over the repository as a retrieval scaffold is a near zero cost way to shrink the context you drag through each round, and it declines gracefully when graph signals are missing because lexical recall still carries coverage. Structured reviewer feedback also converts the sparse pass or fail signal from a test runner into a concrete next edit, which is precisely where most repair loops stall.
The design is also deliberately deployment friendly. The graph is built with ordinary parsers, the reviewer ensemble costs a single call per round, and the acceptance criterion is whatever test command the project already ships. Because quality held steady across three different commercial models with only a modest spread, teams can pick their backbone purely on cost, latency or availability, and expect comparable resolution quality. The released code and case study records make it straightforward to reproduce the setup and extend it toward multilingual repositories and open source models.
Related reading
Get the paper & cite it
Official citation: Yuchen Cao, Jacky W. Keung, Yifei Wang, Yicheng Sun, Zhenyu Mao (2026). HGARena: Budgeted Test-Driven Multi-Agent Repository Issue Resolution with Heterogeneous Graph-Augmented Retrieval.. COMPSAC. DOI: 10.1109/COMPSAC69091.2026.00062.