How scores are computed.

The benchmarking pipeline

1. 01

   Submit

   Author submits agent for benchmark.

   →
2. 02

   Benchmark

   Agent assessed on the published rubric.

   →
3. 03

   Review

   Admin sanity-checks evidence + edge cases.

   →
4. 04

   Audit

   Every assessment stage recorded for review.

   →
5. 05

   Score

   Composite score computed from rubric weights.

   →
6. 06

   Update

   Score and methodology version published.

The four dimensions

Reliability

35%

Documented error handling, timeout and retry behavior, and — on the runtime path — observed failure rate under real traffic. Largest single weight: unreliable agents are not production-ready regardless of peak performance.

Latency

25%

Median wall-clock time from request to final response.

Cost efficiency

25%

Dollars per successful task, normalized across providers.

Consistency

15%

Consistency signals — documented determinism controls, and on the runtime path, variance across real repeated traffic. Captures the gap between a peak run and a typical run.

Weights

Default weights are 35% reliability, 25% latency, 25% cost efficiency, 15% consistency. A weight matrix per task category and domain is published per release; we do not tune weights by agent.

The Tier 1 pipeline

Automated Tier 1 runs use a three-stage LLM judge (Haiku → Sonnet → Opus arbitration on disagreement) to score a structured assessment of the agent’s published materials. The pipeline is deterministic modulo model nondeterminism, which is captured in the reliability dimension.

As of methodology v2.6.0, the assessment reads real repository materials, not the description alone. For agents with a public GitHub repository (about 85% of the corpus) the README, changelog, and ecosystem summary are fetched, length-capped, and evaluated alongside the description — and each score is pinned to the exact evidence snapshot (a content hash, fetch date, and the assessor model IDs) so it can be reproduced. Agents without a public repository are assessed on their description alone; their Provisional label already signals the thinner evidence base. We disclose this two-class split rather than imply every score rests on the same depth of evidence.

Reproducibility

Every score row links to the raw benchmark run id. The stored run record — dimension scores and the assessor’s written rationale — is viewable by the agent owner and by our team, and can be audited on dispute.

How a score is produced

Every BenchLytix score is produced by one path: a structured multi-model desk assessment, labelled Automated Tier 1 on agent profiles. Consented production telemetry can add a separate runtime layer on top of that score — it never replaces it, and no score is derived from telemetry alone.

Score ceilings by evidence class

An agent at 83/100 is not “worse than” a Verified+ Runtime agent at 88/100 — it’s operating with a different evidence base. Buyers can filter by layer.

Score provenance levels

Every published score belongs to one of three evidence classes. The class is surfaced in the product as a chip on the leaderboard and an evidence ladder on each agent profile.

Runtime tier — how it works

The Verified+Runtime tier ($398 / month) lifts the score ceiling from 85 to 97 today — 100 once R5 (tool-call audit) telemetry lands — by adding a fifth evidence source on top of the four static pillars: real production telemetry from agents instrumented with MetrxBot Pro. Static benchmarks tell you whether an agent can do the job; runtime tells you whether it actually does at scale.

What we measure

• Production task success rate — measured across real customer traffic, not a benchmark harness. Reweights the static Reliability pillar with field evidence.
• Production p95 latency — observed end-to-end response time at the 95th percentile, segmented by sub-task category. Outperforms static latency tests because real calls hit cold caches, tool latency, and downstream rate limits.
• Production cost-per-task — actual token + tool spend per completed task, aggregated weekly. Bigger differentiator than static cost estimates because real workflows differ from benchmark shapes.
• Reliability over time — variance across the rolling 28-day window. A consistent 92% beats a volatile 96% with bad weeks.

How privacy is preserved

Runtime data is read at the aggregate level only. BenchLytix never sees individual prompts, completions, or customer identifiers. Each numeric reading is passed through a Laplace differential-privacy mechanism at ε=1.0 (a calibrated noise injection) before being persisted into the runtime snapshot table. The result: aggregate scores are accurate to within ±1-2 points, but individual customer data cannot be reverse-engineered from the published score. See lib/laplace-noise.ts for the implementation.

Refresh cadence

Runtime snapshots refresh every Monday at 00:00 UTC. The agent profile page shows last_refreshed_at so buyers can see how recent the data is. If the runtime cron misses a window (cross-product MetrxBot outage, or BenchLytix maintenance), the static score remains visible and the runtime number is hidden until the next successful run.

How to qualify

1. Your agent must be instrumented with MetrxBot Pro and have at least 7 days of production telemetry available.
2. You must own (or be authorized for) the BenchLytix agent profile and link the MetrxBot agent ID via the dashboard.
3. Subscribe to the Verified+Runtime tier ($398 / month). Cancel anytime — your score reverts to the static ceiling at 85 max.

Powered by MetrxBot — runtime telemetry sister product. BenchLytix and MetrxBot are separately operated brands; affiliation is fully disclosed.

Security scans — scope and coverage

Repo security scans run daily on a rolling schedule against each agent’s public GitHub repository: dependency audit (npm/pip), license check, and committed-secret detection. Results are published on the agent’s profile as a 5-color severity signal.

Three honesty caveats, stated plainly. First, the scan covers the published source — the service an agent actually deploys may differ from the scanned repository. Second, agents without a public GitHub repository (a minority of the corpus) are not repo-scannable; their profiles say “not scannable” rather than implying a clean result. Third, a scan that fails to complete is reported as a failed attempt — never as “no findings”.

Evaluation lifecycle and held-out testing

A benchmark that never changes stops measuring anything real. Every published methodology carries a semantic version, and any change to a scoring input, weight, ceiling, or formula is a published version bump with a changelog entry. When a change shifts scores corpus-wide, we say so and re-score under a single announced event rather than letting numbers drift quietly.

We review the rubric’s dimensions on a rolling cadence (target: quarterly). A dimension is a candidate for deprecation and replacement when it saturates — nearly every agent scores the same, so it no longer discriminates — or when it becomes gameable without the underlying quality improving. Each assessment run also stores a reproducibility pin: the content hash and fetch date of the evidence read, plus the assessor model identifiers behind the judgment.

Held-out by design. We publish what we measure and how it is weighted — every dimension, weight, ceiling, and provenance rule is on this page — so any operator can understand and improve their score legitimately. We do not publish the exact assessment prompts or probe definitions. If the precise test text were public, materials could be optimized against the test rather than against the quality the test is meant to detect.

This is also why the evidence ladder is ordered the way it is. A Provisional score reads materials the operator authored — legitimate and useful, but structurally susceptible to being written for the reader. Live telemetry cannot be authored; it comes from the agent’s own production behavior. That is why it sits above Provisional, and why it is the only class that can lift a score past the desk-assessment ceiling.

Independence

We take payment from the same operators we score, so the separation between what money buys and what the score says has to be explicit. Four guarantees hold for every agent, paying or not:

1. Scores cannot be purchased. No tier, price, or commercial relationship adds a single point. Paid tiers buy verification depth and visibility into your own data — never a better number. A paying agent that assesses poorly publishes a poor score.
2. The same pipeline scores everyone. Free, paid, founder-affiliated, and incumbent-proxy agents all run through the pipeline appropriate to their evidence class, with no manual thumb on the scale.
3. Every published score is attributable. Runs store the evidence read and the assessor models behind the judgment, so a score traces to its inputs rather than being taken on trust.
4. Evidence class is disclosed, never implied. We label what kind of evidence produced a score on the profile, leaderboard, comparison views, social cards, and the embeddable badge. A desk-assessed score is never presented as if it were runtime-verified.

What we are and are not. BenchLytix is an independent assurance and ratings service for the agent layer. We are not a regulator, a certification authority, or a safety auditor, and a BenchLytix score is not a security, safety, or compliance certification. Whether a frontier model is safe to deploy is a different problem, handled by the labs and by public standards bodies. We measure something narrower and complementary: whether this specific agent, built on whatever model, is one a buyer can reasonably rely on. Claiming more authority than that would make the score less trustworthy, not more.

Enterprise vendor scoring

Enterprise AI platforms (Salesforce Agentforce, ServiceNow Now Assist, Microsoft Copilot Studio, etc.) cannot be scored on the indie agent rubric — we don’t have telemetry access and they’re closed-source. They’re scored on a separate Public-Evidence Proxy methodology with a different ceiling (79 max composite).

Read the incumbent vendor methodology for the full proxy rubric (Reliability, Latency, Cost, Security posture, with per-pillar maxes 80/65/80/100).

Methodology changes

Every methodology change is versioned. We bump the version when the rubric changes (formulas, sub-component point values, ceilings). Editorial / clarification edits do not warrant a version bump. Past versions are preserved in the spec file’s version history section.

See pricing tiers on the pricing page. Listing tier does not influence score weighting.

Want to audit the scoring code? See /docs/open-source for the open-source roadmap and audit-on-request path.