DataTrace asks AI agents to find the single faulty database object behind a pipeline failure or incorrect result, repair it, and support the diagnosis with relevant evidence.
Each task represents database artifacts—such as schemas, data, queries, constraints, plans, tests, logs, and issue reports—as a dependency graph. Because the benchmark knows where the fault was introduced, it can score both root-cause accuracy and whether the agent inspected the artifacts that connect the observed problem to that root cause.
A database pipeline can produce an incorrect result without raising an error. The underlying problem may be an unenforced constraint, a renamed column, a corrupted source record, or a broken view several steps upstream. DataTrace checks both the reported cause and the evidence used to find it.
A correct answer receives full credit even when the agent did not inspect the artifacts needed to support it.
Every read action records the graph nodes it touched. A correct answer is compared with the reference evidence path π*. A correct answer with low evidence coverage is reported separately.
Frontier models on the identical frozen corpus (64 tasks × 2 repeats, n = 128 each; one scaffold, one parser). Ranked by RC@1ᵍ — correct and evidence-recall ≥ 0.5. The amber segment is the evidence gap (RC@1 − RC@1ᵍ): correct answers with insufficient evidence.
| # | Model | Grounded accuracy ▾ | CUR | ER | Worst-env |
|---|
95% CIs are environment-clustered bootstrap intervals (4000 draws over 13 environments). CUR = P(ungrounded | correct). ER = evidence recall. Worst-env = lowest RC@1ᵍ over environments with ≥ 6 episodes. Full table with Fix / EF1 / cost / steps in leaderboard.md.
Outcome-only scores can hide unsupported diagnoses. In these experiments, reference-path coverage also provided a useful training signal.
Across the controlled model ladder, root-cause accuracy climbs RC@1 0.18 → 0.82. Capability clearly helps agents reach the correct object.
Over the same model comparison, the share of all runs that are correct but have insufficient evidence rises 0.05 → 0.27. The conditional unsupported rate (CUR) remains present at every tier.
Rewarding reference-path coverage during training reduces CUR 0.163 → 0.054 while RC@1 0.292 → 0.367 rises.
The reference answer—the defective object, the reference evidence path π*, and lineage distance — is derived mechanically from the known fault and dependency graph, without a handwritten reasoning trace for each task.
Exact-match on the single defective object (outcome).
Correct and evidence-recall ≥ 0.5. The ranking metric.
RC@1 − RC@1ᵍ: the share of all runs that are correct but insufficiently supported.
Among correct answers, the fraction with insufficient evidence.
Artifacts inspected by the agent compared with the reference path π*.
The root cause is correct, but evidence recall is below 0.5.
Repaired outputs match the reference outputs and all tests pass.
Rating used in the optional evidence-aware self-play experiment.
Seven fault types introduce a known defect, run the pipeline again, and record the resulting failure or incorrect output. The dependency graph is then used to construct the reference evidence path. The frozen evaluation corpus contains 448 instances (mean reference-path length 5.1; mean dependency distance 3.0).
If DataTrace is useful in your research, please cite the project.
@misc{datatrace,
title = {DataTrace: Evidence-Grounded Root-Cause Localization for Databases},
author = {The DataTrace Authors},
howpublished = {\url{https://anonymous.4open.science/r/datatrace}},
year = {2026}
}