How scores are computed.

The benchmarking pipeline

1. 01

   Submit

   Author submits agent for benchmark.

   →
2. 02

   Benchmark

   Agent assessed on the published rubric.

   →
3. 03

   Review

   Admin sanity-checks evidence + edge cases.

   →
4. 04

   Audit

   Every assessment stage recorded for review.

   →
5. 05

   Score

   Composite score computed from rubric weights.

   →
6. 06

   Update

   Score and methodology version published.

The four dimensions

Reliability

35%

Documented error handling, timeout and retry behavior, and — on the runtime path — observed failure rate under real traffic. Largest single weight: unreliable agents are not production-ready regardless of peak performance.

Latency

25%

Median wall-clock time from request to final response.

Cost efficiency

25%

Dollars per successful task, normalized across providers.

Consistency

15%

Consistency signals — documented determinism controls, and on the runtime path, variance across real repeated traffic. Captures the gap between a peak run and a typical run.

Weights

Default weights are 35% reliability, 25% latency, 25% cost efficiency, 15% consistency. A weight matrix per task category and domain is published per release; we do not tune weights by agent.

The Tier 1 pipeline

Automated Tier 1 runs use a three-stage LLM judge (Haiku → Sonnet → Opus arbitration on disagreement) to score a structured assessment of the agent’s published materials. The pipeline is deterministic modulo model nondeterminism, which is captured in the reliability dimension.

As of methodology v2.6.0, the assessment reads real repository materials, not the description alone. For agents with a public GitHub repository (about 85% of the corpus) the README, changelog, and ecosystem summary are fetched, length-capped, and evaluated alongside the description — and each score is pinned to the exact evidence snapshot (a content hash, fetch date, and the assessor model IDs) so it can be reproduced. Agents without a public repository are assessed on their description alone; their Provisional label already signals the thinner evidence base. We disclose this two-class split rather than imply every score rests on the same depth of evidence.

Reproducibility

Every score row links to the raw benchmark run id. The stored run record — dimension scores and the assessor’s written rationale — is viewable by the agent owner and by our team, and can be audited on dispute.

How a score is produced

Every BenchLytix score is produced by one path: a structured multi-model desk assessment, labelled Automated Tier 1 on agent profiles. Consented production telemetry can add a separate runtime layer on top of that score — it never replaces it, and no score is derived from telemetry alone.

Score ceilings by evidence class

An agent at 83/100 is not “worse than” a Verified+ Runtime agent at 88/100 — it’s operating with a different evidence base. Buyers can filter by layer.

Score provenance levels

Every published score belongs to one of four evidence classes. The class is surfaced in the product as a chip on the leaderboard and an evidence ladder on each agent profile.

Runtime tier — how it works

The runtime tier is free. It lifts the score ceiling from 85 to 97 today — 100 once R5 (tool-call audit) telemetry lands — by adding a fifth evidence source on top of the four static pillars: real production telemetry from agents instrumented with Metrx. Static benchmarks tell you whether an agent can do the job; runtime tells you whether it actually does at scale.

What we measure

• R1 · Production latency — end-to-end response time observed in production at the 95th percentile, not in a benchmark harness.
• R2 · Real-world cost — token and tool spend per completed task on real workflows, which differ from benchmark shapes.
• R3 · Operational consistency — how steady latency and cost stay across the measurement window; a consistent agent beats a volatile one with bad weeks.
• R4 · Runtime reliability — task success rate across real customer traffic, re-weighting the static Reliability pillar with field evidence.
• R5 · Tool-call audit (pending) — whether tool calls were made and completed as the agent claims. Needs Metrx tool-call telemetry; shown as Pending, never as a zero, until it lands.

How privacy is preserved

Runtime data is read at the aggregate level only. BenchLytix never sees individual prompts, completions, or customer identifiers. Each numeric reading is passed through a Laplace differential-privacy mechanism at ε=1.0 (a calibrated noise injection) before being persisted into the runtime snapshot table. The result: aggregate scores are accurate to within ±1-2 points, but individual customer data cannot be reverse-engineered from the published score. See lib/laplace-noise.ts for the implementation.

Refresh cadence

Runtime snapshots refresh daily. The agent profile page shows last_refreshed_at so buyers can see how recent the data is. If telemetry lapses (a Metrx outage, or an agent that stops reporting), the static score remains visible and the profile marks the runtime layer as lapsed rather than silently hiding it.

How to qualify

1. Your agent must be instrumented with Metrx and have at least 7 days of production telemetry available.
2. You must own (or be authorized for) the BenchLytix agent profile and ask us to link the Metrx agent ID (hello@benchlytix.com). There is no self-serve link yet; BenchLytix sets it up.
3. There is nothing to buy. The runtime tier is free; if telemetry lapses, the score reverts to the static ceiling of 85 and the profile says so.

Powered by MetrxBot — the runtime telemetry product built by the same team as BenchLytix. The affiliation is disclosed wherever runtime data appears.

Security scans — scope and coverage

Repo security scans run daily on a rolling schedule against each agent’s public GitHub repository: dependency audit (npm/pip), license check, and committed-secret detection. Results are published on the agent’s profile as a 5-color severity signal.

Three honesty caveats, stated plainly. First, the scan covers the published source — the service an agent actually deploys may differ from the scanned repository. Second, agents without a public GitHub repository (a minority of the corpus) are not repo-scannable; their profiles say “not scannable” rather than implying a clean result. Third, a scan that fails to complete is reported as a failed attempt — never as “no findings”.

Evaluation lifecycle and held-out testing

A benchmark that never changes stops measuring anything real. Every published methodology carries a semantic version, and any change to a scoring input, weight, ceiling, or formula is a published version bump with a changelog entry. When a change shifts scores corpus-wide, we say so and re-score under a single announced event rather than letting numbers drift quietly.

We review the rubric’s dimensions on a rolling cadence (target: quarterly). A dimension is a candidate for deprecation and replacement when it saturates — nearly every agent scores the same, so it no longer discriminates — or when it becomes gameable without the underlying quality improving. Each assessment run also stores a reproducibility pin: the content hash and fetch date of the evidence read, plus the assessor model identifiers behind the judgment.

Held-out by design. We publish what we measure and how it is weighted — every dimension, weight, ceiling, and provenance rule is on this page — so any operator can understand and improve their score legitimately. We do not publish the exact assessment prompts or probe definitions. If the precise test text were public, materials could be optimized against the test rather than against the quality the test is meant to detect.

This is also why the evidence ladder is ordered the way it is. A Provisional score reads materials the operator authored — legitimate and useful, but structurally susceptible to being written for the reader. Live telemetry cannot be authored; it comes from the agent’s own production behavior. That is why it sits above Provisional, and why it is the only class that can lift a score past the desk-assessment ceiling.

Independence

BenchLytix takes no payment from the operators it scores. The separation between commercial relationships and what the score says is still stated explicitly. Four guarantees hold for every agent:

1. Scores cannot be purchased. There is no paid tier, and no price or commercial relationship adds a single point. Nothing buys verification depth either: an agent that assesses poorly publishes a poor score.
2. The same pipeline scores everyone. Every agent, founder-affiliated or not, and every incumbent proxy runs through the pipeline appropriate to its evidence class, with no manual thumb on the scale.
3. Every published score is attributable. Runs store the evidence read and the assessor models behind the judgment, so a score traces to its inputs rather than being taken on trust.
4. Evidence class is disclosed, never implied. We label what kind of evidence produced a score on the profile, leaderboard, comparison views, social cards, and the embeddable badge. A desk-assessed score is never presented as if it were runtime-verified.

What we are and are not. BenchLytix is an independent assurance and ratings service for the agent layer. We are not a regulator, a certification authority, or a safety auditor, and a BenchLytix score is not a security, safety, or compliance certification. Whether a frontier model is safe to deploy is a different problem, handled by the labs and by public standards bodies. We measure something narrower and complementary: whether this specific agent, built on whatever model, is one a buyer can reasonably rely on. Claiming more authority than that would make the score less trustworthy, not more.

Enterprise vendor scoring

Enterprise AI platforms (Salesforce Agentforce, ServiceNow Now Assist, Microsoft Copilot Studio, etc.) cannot be scored on the indie agent rubric — we don’t have telemetry access and they’re closed-source. They’re scored on a separate Public-Evidence Proxy methodology with a different ceiling (79 max composite).

Read the incumbent vendor methodology for the full proxy rubric (Reliability, Latency, Cost, Security posture, with per-pillar maxes 80/65/80/100).

Methodology changes

Every methodology change is versioned. We bump the version when the rubric changes (formulas, sub-component point values, ceilings). Editorial / clarification edits do not warrant a version bump. Past versions are preserved in the spec file’s version history section.

BenchLytix is free; see how it is funded. Nothing a listing can buy influences score weighting.

Want to audit the scoring code? See /docs/open-source for the open-source roadmap and audit-on-request path.