Sign in

Open source

BenchLytix is committed to open-sourcing the scoring stack so any vendor or buyer can independently audit how a score is computed. Methodology is public today (indie v2.6.0 and enterprise v1.1). The benchmark suite is live under Apache 2.0. The TypeScript SDK (@benchlytixai/sdk), Python SDK (benchlytix), and MCP server (benchlytix-mcp-server) are now published on npm and PyPI under MIT — source at github.com/ckpark123/benchlytix-sdks. The canonical incumbent scorer is also public at github.com/ckpark123/benchlytix-incumbent-scorer (MIT, published 2026-05-06).

Why open-source the scorer

Two reasons. First, methodology drift between the spec and the implementation has been a real risk during early development — published code makes drift externally observable, not just internally. Second, vendors disputing a score should be able to re-run the scorer against their own evidence and reproduce the result before opening a takedown thread. Open-source code closes that loop.

Repos

All licenses are permissive so vendors and integrators can fork, modify, and redistribute without legal friction. The benchmark-suite shipped under Apache 2.0; the SDK family (TypeScript, Python, MCP server) ships under MIT to match the thin-API-client shape and minimize legal review for downstream users.

benchmark-suite Published

The Tier 2 API benchmarking harness — test cases, judge rubrics, runner, and supporting scripts. Public today; clone, run against any agent with a public API, and reproduce a leaderboard score from public inputs.

Repository: github.com/ckpark123/benchmark-suite
License: Apache 2.0

Contents:

  • cases/ — task-specific test cases
  • rubrics/ — judge rubrics by dimension
  • src/ — runner + scoring code
  • scripts/ — helpers for local execution
  • README.md — install + run instructions

@benchlytixai/sdk (TypeScript) Published

Official TypeScript SDK for the BenchLytix Machine API. Published on npm as @benchlytixai/sdk (the @benchlytix npm scope was unavailable; @benchlytixai is the canonical scope). 17 passing tests including brand-separation regression gate.

Repository: github.com/ckpark123/benchlytix-sdks
Install: npm install @benchlytixai/sdk
License: MIT

Contents:

  • src/client.ts — BenchLytix class with leaderboard/agent/verifyStatus methods
  • src/types.ts — request/response interfaces matching the v1 API
  • src/errors.ts — typed error hierarchy (Auth/NotFound/RateLimit/Server)
  • README.md — install + 3-method quickstart

benchlytix (Python) Published

Official Python SDK for the BenchLytix Machine API. Published on PyPI as benchlytix. Sync + async clients, pydantic types, 18 passing tests including brand-separation gate.

Repository: github.com/ckpark123/benchlytix-sdks
Install: pip install benchlytix
License: MIT

Contents:

  • src/benchlytix/client.py — BenchLytix and AsyncBenchLytix classes
  • src/benchlytix/types.py — pydantic response models
  • src/benchlytix/errors.py — typed error hierarchy
  • README.md — install + sync/async quickstart

benchlytix-mcp-server Published

Model Context Protocol server wrapping the public API. Published on npm as benchlytix-mcp-server. Exposes get_leaderboard, get_agent_score, verify_agent, get_categories, compare_agents, get_methodology as tools to any MCP-compatible client (Claude Desktop, Cursor, VS Code). 16 passing tests.

Repository: github.com/ckpark123/benchlytix-sdks
Install: npx benchlytix-mcp-server
License: MIT

Contents:

  • src/index.ts — MCP server entrypoint (stdio transport)
  • src/tools.ts — 6 tool handlers + JSON-schema definitions
  • README.md — install + Claude Desktop config example

benchlytix-incumbent-scorer Published

The canonical 4-pillar Public-Evidence Proxy scorer (v1.0) used to rank Salesforce Agentforce, ServiceNow Now Assist, Microsoft Copilot Studio, and the rest of the enterprise vendor cohort. Pure function — no I/O, no LLM calls, deterministic output. Published as a public GitHub repo 2026-05-06; clone, drop your own evidence into the typed VendorEvidence shape, and reproduce the published score before opening a takedown thread.

Repository: github.com/ckpark123/benchlytix-incumbent-scorer
License: MIT

Contents:

  • canonical-scorer.ts — pure function applying the rubric
  • types.ts — VendorEvidence shape (one field per addendum sub-component)
  • evidence/ — per-vendor evidence objects with citation URLs (11 vendors)
  • README.md — boundary-case policies + calibration baseline

How to audit the scorer today

The canonical incumbent scorer (v1.0) is pure-function and stable. Until the public repo is live, audit access is available on request:

  1. Email legal@benchlytix.com with the subject "Source audit request — incumbent scorer".
  2. Identify yourself (vendor name + your role, or buyer organization). NDA is mutual and non-burdensome.
  3. We share the scorer source + per-vendor evidence files within 5 business days. The methodology page (/docs/incumbent-methodology) maps every line of code to a published rubric row, so review is a few hours of reading rather than reverse-engineering.

How to verify a score against your own data

For incumbent vendors specifically — even before the repo is public, you can replicate our scoring locally:

  1. Read the rubric at /docs/incumbent-methodology. Every sub-component, point cap, and pillar formula is listed with the exact wording used in the scorer's code comments.
  2. Score yourself against each sub-component using only the publicly available documentation we cite on your profile (link: /agents/<your-slug>, "Evidence" section).
  3. Apply the composite formula: ROUND((R + L + C + S) / 325 * 79). Result should match the published score within ±1 (rounding).
  4. If your manual reproduction differs from our published score by more than ±1, file a factual dispute (Section B of the vendor response template) — that's a real bug we want to fix.

Publication timeline

Status reflects what's actually live, not what's scheduled. The benchmark suite shipped 2026-04-18, the SDK family (@benchlytixai/sdk + benchlytix Python + benchlytix-mcp-server) shipped 2026-05-05, and the canonical incumbent scorer shipped 2026-05-06 at github.com/ckpark123/benchlytix-incumbent-scorer. Every repo published is MIT or Apache 2.0; the table above states the per-repo license.

Public API alongside

The public API is a separate, parallel transparency lever — a verified buyer can fetch any agent's published score without running our code. The API and the open-source scorer serve different audiences (integrators vs auditors) and ship on independent timelines. Note: API access requires an active $500/mo API Access subscription; the SDKs and MCP server above are MIT-licensed and free to clone / fork / read regardless of subscription state.