Agent evaluation evidence: worked examples

Agent Evidence reviews imported task outcomes within one task, version and date window. It deduplicates sample labels and shows failures, exclusions and uncertainty. It helps review a supplier comparison; it does not run evaluations or verify reviewer identities.

Three fictional scenarios, calculated by the same engine as the tool. They illustrate behavior and failure cases, not customer outcomes or measured provider performance.

Repeated task samples

See how the evidence denominator changes after deduplication.

Providers: 1
Excluded records: 1
Sample sets: Aligned labels only

Task-specific observations
Provider	Accepted / samples	Distinct reviewers	Collapsed repeats	Conflicts	Observed rate	95% Wilson interval
Example A	1 / 2	2	1	0	50.0%	9.5% – 90.5%

Run this scenario Input JSON Calculated report

Inspect every input

{
  "task": "invoice-extraction",
  "version": "v1",
  "asOf": "2026-09-19",
  "maxAgeDays": 30,
  "records": [
    {
      "id": "e1",
      "provider": "Example A",
      "task": "invoice-extraction",
      "version": "v1",
      "sample": "case-1",
      "reviewer": "reviewer-a",
      "date": "2026-09-18",
      "accepted": true,
      "relation": "independent"
    },
    {
      "id": "e2",
      "provider": "Example A",
      "task": "invoice-extraction",
      "version": "v1",
      "sample": "case-1",
      "reviewer": "reviewer-b",
      "date": "2026-09-18",
      "accepted": true,
      "relation": "independent"
    },
    {
      "id": "e3",
      "provider": "Example A",
      "task": "invoice-extraction",
      "version": "v1",
      "sample": "case-2",
      "reviewer": "reviewer-a",
      "date": "2026-09-18",
      "accepted": false,
      "relation": "independent"
    },
    {
      "id": "e4",
      "provider": "Example B",
      "task": "invoice-extraction",
      "version": "v1",
      "sample": "case-1",
      "reviewer": "vendor-b",
      "date": "2026-09-18",
      "accepted": true,
      "relation": "self"
    }
  ]
}

Reviewers disagree

A conflicting result remains a conflict, not an extra success.

Providers: 1
Excluded records: 1
Sample sets: Aligned labels only

Task-specific observations
Provider	Accepted / samples	Distinct reviewers	Collapsed repeats	Conflicts	Observed rate	95% Wilson interval
Example A	0 / 2	2	1	1	0.0%	0.0% – 65.8%

Run this scenario Input JSON Calculated report

Inspect every input

{
  "task": "invoice-extraction",
  "version": "v1",
  "asOf": "2026-09-19",
  "maxAgeDays": 30,
  "records": [
    {
      "id": "e1",
      "provider": "Example A",
      "task": "invoice-extraction",
      "version": "v1",
      "sample": "case-1",
      "reviewer": "reviewer-a",
      "date": "2026-09-18",
      "accepted": true,
      "relation": "independent"
    },
    {
      "id": "e2",
      "provider": "Example A",
      "task": "invoice-extraction",
      "version": "v1",
      "sample": "case-1",
      "reviewer": "reviewer-b",
      "date": "2026-09-18",
      "accepted": false,
      "relation": "independent"
    },
    {
      "id": "e3",
      "provider": "Example A",
      "task": "invoice-extraction",
      "version": "v1",
      "sample": "case-2",
      "reviewer": "reviewer-a",
      "date": "2026-09-18",
      "accepted": false,
      "relation": "independent"
    },
    {
      "id": "e4",
      "provider": "Example B",
      "task": "invoice-extraction",
      "version": "v1",
      "sample": "case-1",
      "reviewer": "vendor-b",
      "date": "2026-09-18",
      "accepted": true,
      "relation": "self"
    }
  ]
}

Wrong agent version

A result from a different version cannot improve this cohort.

Providers: 1
Excluded records: 2
Sample sets: Aligned labels only

Task-specific observations
Provider	Accepted / samples	Distinct reviewers	Collapsed repeats	Conflicts	Observed rate	95% Wilson interval
Example A	1 / 1	2	1	0	100.0%	20.7% – 100.0%

Run this scenario Input JSON Calculated report

Inspect every input

{
  "task": "invoice-extraction",
  "version": "v1",
  "asOf": "2026-09-19",
  "maxAgeDays": 30,
  "records": [
    {
      "id": "e1",
      "provider": "Example A",
      "task": "invoice-extraction",
      "version": "v1",
      "sample": "case-1",
      "reviewer": "reviewer-a",
      "date": "2026-09-18",
      "accepted": true,
      "relation": "independent"
    },
    {
      "id": "e2",
      "provider": "Example A",
      "task": "invoice-extraction",
      "version": "v1",
      "sample": "case-1",
      "reviewer": "reviewer-b",
      "date": "2026-09-18",
      "accepted": true,
      "relation": "independent"
    },
    {
      "id": "e3",
      "provider": "Example A",
      "task": "invoice-extraction",
      "version": "v2",
      "sample": "case-2",
      "reviewer": "reviewer-a",
      "date": "2026-09-18",
      "accepted": false,
      "relation": "independent"
    },
    {
      "id": "e4",
      "provider": "Example B",
      "task": "invoice-extraction",
      "version": "v1",
      "sample": "case-1",
      "reviewer": "vendor-b",
      "date": "2026-09-18",
      "accepted": true,
      "relation": "self"
    }
  ]
}

A mistake worth catching

Treating two reviews of one case as two independent tasks overstates the evidence. This worksheet collapses repeated sample labels and exposes conflicts.

Use the method with your records

Filter to a dated task/version cohort. Exclude self/unknown relations and future/stale records. Collapse repeated sample labels, treating conflicts conservatively; show a descriptive 95% Wilson interval.

No public reputation ranking, identity verification, calibrated prediction or Sybil detection.

Read the complete method, sources and input contract. Updated 2026-09-19; by the AGI Scorecard team.

Continue your review

Route LabCompare bounded supplier sequences using quality assumptions, worst-case cost and latency constraints.Compute LensBring measured workloads and account for setup, egress, storage and failed outputs before choosing compute.