io.github.razilzul/llm-eval-harness

LLM Eval Harness

Is the LLM Eval Harness MCP server down? No. It answered initialize and tools/list in the probe of 2026-10-31.

Run a golden set and gate LLM model swaps on accuracy, with latency and cost per request.

alive probed 2026-11-15 · unchanged since 2026-10-31

2tools

596tokens in Claude Opus 5.5

0.3 %of Claude's 200k window

0.2×the median server

1.6 sto initialize

2 msto list tools

Run it: docker run -i --rm ghcr.io/razilzul/llm-eval-harness:1.0.0 (stdio, probed in a container with the network off)
Host: ghcr.io
Reports itself as: llm-eval-harness 1.0.0
Capabilities: tools
Instructions: none
Of a 1M window: <0.1 % of a 1M context window
Count: 596 tokens in Claude Opus 5.5, exact, measured 2026-10-31. With the Claude Code prefix mcp__…__ on each tool name: 622. The o200k_base estimate is 381. Counted again when the tool definitions change. Method.
Repository: https://github.com/razilzul/llm-eval-harness
Registry: active, updated 2026-10-18 · registry entry

Most expensive tools

The Claude column scales the o200k estimate of each tool by this server's ratio of exact to estimate, 1.56×.

Tool	≈ Claude tokens	o200k estimate	Share of this server
compare_models	354	226	59 %
run_golden_set	242	155	41 %

Other registry entries on ghcr.io, the ones next to this name; all of them.

pREST io.github.prest/prest · crashed on start
Proxus Industrial MCP Server io.github.proxusiiotplatform/proxus-mcp · crashed on start
makevideo io.github.random00000000/makevideo · alive
Crossplane io.github.ravibagri5/crossplane-mcp-server · needs env
Squire io.github.reidgoodbar/squire · alive
Audio Sonic MCP io.github.ripunjay-kashyap/audio-sonic-mcp · alive
pim-mcp io.github.robinvdvleuten/pim-mcp · needs env
io.github.rogiervdbrnk/mcp-real-estate-server · not found

Servers that mention harness.

Bring Your AI ai.bringyour/bringyour · alive
SignalHarness.ai ai.signalharness/mcp · server error
Webhound ai.webhound/webhound · needs auth
com.clauxel.clianythingharness/clianythingharness-mcp · no DNS
com.clauxel.jcodeharness/jcodeharness-mcp · no DNS
SuperSkill Registry com.onlyharness/registry · alive
Santismm Knowledge — Harness Engineering, Agentic AI & Governance com.santismm/knowledge · alive
Agent Reliability dev.agentreliability/agent-reliability · alive

The result of the latest probe, the tool count and the token cost. The feed has an entry for every probe that changed them.

Stops answering, comes back, tool set or cost changes. One confirmation mail first, a stop link in every mail.