Project Guide | Agent Orchestration Benchmark

Live Demo

• Open the public GitHub Pages demo
• Scope: credential-free, synthetic-data demo for AI platform teams and technical evaluators.

Standardized benchmark suite comparing LLM agent orchestration frameworks on a shared task. Measures reliability, latency, cost, and deterministic replay — the metrics that matter for production operators.

---

System Overview

A benchmark suite that lets teams compare orchestration runtimes before they commit to a fragile agent stack.

Evaluation Path

• Start here: Compare results/latest.md or the sample run before reading runner internals.
• Local demo: Run make install-dev && make bench; no model API key is required for the mock run.
• Checks: Run make test and make lint; rerender reports with make report.

Architecture Notes

• Architecture guide summarizes the system scope, first files to inspect, runtime commands, and known boundaries.
• Quality notes lists the local checks, CI surface, and release expectations for this repository.
• Enterprise readiness notes outlines security, data, operations, integration, and handoff expectations.

Why this exists

Teams picking between LangGraph, CrewAI, AutoGen, and home-grown orchestrators currently have no apples-to-apples comparison. Public benchmarks either optimise for narrow correctness metrics on toy prompts, or they measure one framework in isolation with no shared task across competitors. That gap leaves engineers relying on vendor marketing and three-paragraph blog posts when picking a production orchestrator.

This repository provides a reproducible benchmark: one dataset, one set of tools, one grading rubric, four runners. Every runner exposes the same BaseRunner interface, so adding a fifth framework is a fifty-line pull request. CI runs against a built-in deterministic mock LLM so the numerical comparison of orchestrator *shape* requires no API key.

The four dimensions the benchmark cares about are the four that decide whether an agent ships: correctness (does it pick the right tool?), quality (is the answer acceptable?), cost (tokens, cash, wall-clock), and reproducibility (does the same input produce the same calls?). Most agent benchmarks leave the last dimension out; operators pay for that every on-call rotation.

---

Quick start

git clone https://github.com/KIM3310/agent-orchestration-benchmark.git
cd agent-orchestration-benchmark
make install-dev
make bench          # mock-LLM run, no API key required


## If your default python3 is older than 3.11:
make BOOTSTRAP_PYTHON=/path/to/python3.11 install-dev

Results land under results/:

results/
  latest.json        # machine-readable
  latest.md          # human-readable markdown
  latest.html        # standalone HTML report with prompt and tool-call evidence

To re-render reports without re-executing the benchmark:

python -m scripts.run_bench --report-only --input results/latest.json

To request each framework adapter's live path:

export OPENAI_API_KEY=sk-...
make bench-live

The generated report labels this mode LIVE / CONFIGURED. That label records the requested adapter path; it does not by itself prove provider traffic. Provider credentials and integration depth differ by adapter, so confirm exceptions, token counters, and provider-side logs before making comparisons.

A working sample run is checked in at `results/sample_run_2026-04-16.json`. The numbers in Sample Results are rendered from that file.

---

Architecture

flowchart LR
    F["fixtures/benchmark_prompts.jsonl<br/>(20 prompts)"] --> R
    subgraph Runner
      R["BenchmarkRunner"] --> A1["stage-pilot-style"]
      R --> A2["LangGraph"]
      R --> A3["CrewAI"]
      R --> A4["AutoGen"]
    end
    A1 & A2 & A3 & A4 --> TOOL["Shared tools<br/>query_sales_data · summarize_trend"]
    TOOL --> M["MetricsCollector<br/>latency · tokens · retries · fingerprints"]
    M --> RPT["Report generators<br/>JSON · Markdown · HTML"]
    RPT --> OUT[("results/*")]

The runner is pluggable: each framework adapter implements src.runners.base.BaseRunner and is registered in scripts/run_bench.py. Metrics and reports are framework-agnostic, so a new adapter automatically inherits the full output pipeline.

---

Frameworks benchmarked

The fourth runner (stage-pilot-style) is a minimal in-house baseline. It exists so the report answers the operator's real question: "is the framework giving me net value over a well-designed script?"

All adapters default to a built-in mock LLM so CI runs are free and deterministic. Live mode requests each adapter's integration path and must be validated against that adapter's provider prerequisites.

---

Metrics

See `docs/methodology.md` for exact formulas, and ADR 002 for the rationale.

---

Sample results

Rendered from `results/sample_run_2026-04-16.json`. Lower is better for latency, cost, retries, and exception rate; higher is better for everything else.

Framework summary

How to read this

because it has no implicit state — every tool call is argument-checked and fingerprintable. It also wins on cost because the conversation pattern is shortest.

tool-calling loop rehydrates extra state into the prompt. Tool success is close to parity.

reporter roughly doubles the number of LLM calls per prompt. That also shows up as the highest retry count.

predictable than CrewAI's role negotiation but less economical than a single-assistant loop.

• stage-pilot-style wins on both correctness and replay stability
• LangGraph trails on cost and retry count mostly because its default
• CrewAI is the slowest; the role-based hand-off between analyst and
• AutoGen sits between LangGraph and CrewAI. Its proxy pattern is more

Numbers are mock-LLM to keep the comparison reproducible. Do not extrapolate their ranking to live providers without a separately verified live run.

Per-framework deltas at a glance

---

Extending

Adding a new framework is a four-step process:

src.runners.base.BaseRunner. Implement run_prompt(prompt) -> PromptObservation.

runners do; the benchmark guarantees determinism as long as you do.

one fixture prompt.

1. Create src/runners/<name>_runner.py that subclasses
2. In mock mode, call self.llm.complete(...) exactly as the existing
3. Register the runner in scripts/run_bench.py (FRAMEWORK_CHOICES).
4. Add a test file under tests/ that validates the runner on at least

A minimal adapter is roughly 40 lines. See `src/runners/autogen_runner.py` for the shortest existing example.

---

Methodology

The benchmark isolates orchestrator behaviour by holding prompts, tools, grading, and the mock backend constant in CI. Live mode requests adapter-specific integration paths, whose provider setup can differ. Every run freezes its prompt contracts and records per-prompt observations alongside the framework summary so the aggregate can be audited at the individual-call level. Full write-up: `docs/methodology.md`.

Key invariants the benchmark enforces:

must produce an answer that satisfies the same keyword + regex grader.

implementations in src/task.py are pure functions of their input; identical arguments always produce identical output.

config.seed. Frameworks do not get to bring their own completion strategy in CI.

orchestrator cannot distort aggregate metrics.

src/config.py is the only place that translates tokens to dollars.

1. Shared task. Every framework receives the same twenty prompts and
2. Shared tools. The query_sales_data and summarize_trend
3. Shared LLM. The MockLLM in src/runners/base.py is seeded with
4. Bounded loops. Every adapter caps its own loop count so a runaway
5. Single source of truth for scope. PRICING_USD_PER_MTOK in

Related reading:

• ADR 001: Framework Selection
• ADR 002: Metric Definitions
• ADR 003: Determinism Requirements
• Results Interpretation Guide

---

Project structure

agent-orchestration-benchmark/
├── README.md
├── LICENSE
├── pyproject.toml
├── requirements.txt
├── Dockerfile
├── docker-compose.yml
├── Makefile
├── .gitignore
├── .dockerignore
├── .github/
│   └── workflows/
│       └── ci.yml
├── src/
│   ├── __init__.py
│   ├── config.py              # models, scope, retry budgets, timeouts
│   ├── task.py                # standardized task + deterministic tools
│   ├── fixtures.py            # prompt loader + grading contract
│   ├── metrics.py             # metric primitives + aggregation
│   ├── runner.py              # BenchmarkRunner (drives adapters)
│   ├── report.py              # JSON / Markdown / HTML renderers
│   └── runners/
│       ├── __init__.py
│       ├── base.py            # BaseRunner protocol + MockLLM
│       ├── langgraph_runner.py
│       ├── crewai_runner.py
│       ├── autogen_runner.py
│       └── stage_pilot_style.py  # deterministic tool-calling loop, ~200 LOC
├── tests/
│   ├── __init__.py
│   ├── test_task.py
│   ├── test_metrics.py
│   ├── test_stage_pilot_style.py
│   └── test_report.py
├── fixtures/
│   └── benchmark_prompts.jsonl   # 20 standardized prompts
├── results/
│   └── sample_run_2026-04-16.json
├── docs/
│   ├── methodology.md
│   ├── results-interpretation.md
│   └── adr/
│       ├── 001-framework-selection.md
│       ├── 002-metric-definitions.md
│       └── 003-determinism-requirements.md
└── scripts/
    ├── __init__.py
    ├── run_bench.py              # python -m scripts.run_bench
    └── run_bench.sh              # convenience wrapper

---