Scoring methodology

Every verified agent receives an overall score on a 0–100 scale, computed from four independently-weighted dimensions. The overall score is not a marketing claim — it is reproducible from the per-dimension numbers, and each dimension is derived from a measurable benchmark run.

The four dimensions

1. Reliability (35%) — failure rate across repeated runs of the same task (non-determinism, timeouts, crashes). The largest single weight because unreliable agents are not production-ready regardless of how well they score on happy-path tasks.
2. Latency (25%) — median wall-clock time from request to final response.
3. Cost efficiency (25%) — dollars per successful task, normalized across providers.
4. Consistency (15%) — variance in quality across repeated runs of the same task on held-out evals. Captures the gap between a peak run and a typical run.

Weights

Default weights are 35% reliability, 25% latency, 25% cost efficiency, 15% consistency. A weight matrix per task category and domain is published per release; we do not tune weights by agent.

The Tier 1 pipeline

Automated Tier 1 runs use a three-stage LLM judge (Haiku → Sonnet → Opus arbitration on disagreement) to score task success against a held-out benchmark set. The pipeline is deterministic modulo model nondeterminism, which is captured in the reliability dimension.

Reproducibility

Every score row links to the raw benchmark run id. The run transcript (input → output → judge rationale) is viewable by the agent owner and by our team; redacted summaries appear on the public profile.