AutoXRDAutonomous LLM Agents and Comprehensive Evaluation for Powder Diffraction Analysis

Verifiable, stepwise powder X-ray diffraction analysis with executable crystallographic backends and scientific checks.

↗PaperarXiv PDF ◎XRDBenchInteractive benchmark </>CodeGitHub

134XRD tasks

10recent LLMs

1,340model–task runs

625.6Mtrajectory tokens

Overview

From plausible answers to verifiable XRD workflows

Powder X-ray diffraction is central to materials characterization, yet reliable end-to-end automation remains challenging. AutoXRD organizes powder-XRD analysis into stepwise refinement procedures, grounds actions in observed evidence, and applies deterministic crystallographic and physical checks before accepting a result.

XRDBench evaluates the resulting agents through 100 diagnostic questions and 34 executable workflows across eleven XRD task families. Across ten models, performance decreases from 61.9 on diagnostic QA to 53.7 on E2E workflows, exposing persistent limitations in quantitative reasoning, indexing, Rietveld refinement, and controlled termination.

Method

AutoXRD: evidence-preserving scientific execution

Planning, refinement backends, physical validation, and auditable trajectories in one agent framework.

Architecture of AutoXRD and XRDBench — **AutoXRD overview.** Skill-guided planning, executable XRD tools, deterministic checks, and evidence-preserving refinement trajectories.

Benchmark

XRDBench evaluates reasoning and execution

Two complementary tracks test isolated scientific decisions and their composition into executable XRD workflows.

XRDBench-QA100

Diagnostic tasks

Refinement decisions, residual interpretation, parameter recovery, phase analysis, and refinement-history assessment.

30 easy · 40 medium · 30 hard
Maximum 20 tool calls
Exact, objective, and scientific evaluation

XRDBench-E2E34

Executable workflows

State-changing analyses across eleven XRD task families with artifacts, execution gates, and physical validation.

Measured diffraction patterns
Maximum 50 agent steps
Artifact 15% · Metrics 55% · Judge 30%

XRDBench task composition across QA and E2E tracks — **XRDBench composition.** The two tracks contain 134 tasks spanning diagnostic capabilities and eleven executable XRD task families.

Evaluation

Large-scale evaluation of ten recent LLMs

1,340 model–task runs reveal a clear gap between bounded scientific reasoning and autonomous workflow execution.

Overall mean57.8

out of 100 across the two tracks

QA mean61.9

bounded diagnostic reasoning

E2E mean53.7

executable XRD workflows

Best overall81.1

GPT-5.6 Sol

Overall, QA, and E2E performance of ten language models — **Overall performance.** Scores combine XRDBench-QA and XRDBench-E2E with equal track weight.

Fine-grained analysis

Capability profiles expose different scientific bottlenecks

Fine-grained capability radar charts for QA and E2E tasks — **Capability profiles.** Models are relatively strong at refinement-history assessment and result acceptance, but remain weak at action selection, phase quantification, indexing, and Rietveld refinement.

Efficiency

Accuracy, execution time, tokens, steps, and cost

Workflow bottlenecks

Scientific bottlenecks and component effects

Heatmap of XRDBench-E2E scores, scoring-gate and completion rates, and effective evaluation components across eleven task families — **Figure 5: Scientific bottlenecks and component effects.** (a) XRDBench-E2E scores, scoring-gate and completion rates, and effective evaluation components across eleven task families, averaged over ten models. (b) Full-system score gains from six AutoXRD components.

Citation

AutoXRD

If you use AutoXRD or XRDBench, please cite our paper.

@misc{wu2026autoxrdautonomousllmagents,
  title={AutoXRD: Autonomous LLM Agents and Comprehensive Evaluation for Powder Diffraction Analysis},
  author={Yuetong Wu and Maojun Sun},
  year={2026},
  eprint={2609.00070},
  archivePrefix={arXiv},
  primaryClass={cond-mat.mtrl-sci},
  url={https://arxiv.org/abs/2609.00070}
}