Wiring the MCP conformance suite into local runs (and the client bug it caught)

PR #327 wires the official MCP conformance suite into the package's local development workflow, so the wire format is checked against the reference implementation instead of only against our own tests. It adds three composer scripts, a purpose-built fixture server, a baseline of known-failing scenarios, and — because the suite earned its keep on its very first run — two fixes to the client's protocol fallback. If you're upgrading from v0.9.6 there is nothing to migrate; the user-facing payload is that a 1.0 client connecting to a server predating server/discover falls back to initialize again instead of dying.

One command to run the whole spec

composer test:conformance runs both halves of the suite, with test:conformance:server and test:conformance:client for each side individually:

$ composer test:conformance:server

✓ tools-list: 3 passed, 0 failed
✗ tools-call-with-progress: 1 passed, 1 failed
...
Total: 113 passed, 57 failed
Baseline check passed: all failures are expected.
server conformance: 113/170 (66%)

Under the hood, tests/Conformance/run.sh boots testbench serve, polls the fixture endpoint at /conformance with a server/discover probe until it's ready, runs the reference runner via npx, and fails the run if anything outside conformance-baseline.yml breaks. Afterwards, score.php prints the pass rate and writes shields-style badge JSON (server-conformance.json, client-conformance.json). Port, pinned package, and revision are overridable via CONFORMANCE_PORT, CONFORMANCE_PACKAGE, and CONFORMANCE_REVISION.

The runner is pinned to @modelcontextprotocol/conformance@alpha for a reason: stable 0.1.16 only speaks up to the 2025-11-25 revision, which our server no longer accepts, and the handshake-less 2026-07-28 revision only landed in 0.2.0-alpha.x — the same line that added the --requirements flag the runner uses. Don't swap --requirements 2026-07-28 for --spec-version 2026-07-28; the latter scores a flattering 34/6 purely by dropping 30 pending and draft scenarios.

A fixture server built to be probed

The new ConformanceServer is mounted at /conformance in the workbench service provider:

Mcp::web('conformance', ConformanceServer::class);

It negotiates 2026-07-28, 2025-11-25, and 2025-06-18, and registers the response shapes the suite probes: tools returning text, image, audio, resource links, and mixed content (plus one that deliberately errors); static text, binary, template (test://template/{id}/data), and watchable resources; and prompts with required arguments, image output, and embedded resources.

What the baseline records — and what looks worth fixing

The opening server score of 113/170 (66%) is dominated by genuinely unimplemented features: tasks and MRTR, elicitation, sampling and roots round trips, progress notifications, embedded-resource content, and the client auth block that dominates the client-side baseline. Two failures are fixture gaps rather than package gaps (json-schema-2020-12 and http-custom-header-server-validation, each needing one more fixture tool — about six free passes).

Five scenarios look like real bugs, flagged as leads: server-stateless (undeclared client capabilities aren't rejected, and MissingRequiredClientCapabilityError should map to a 400), http-header-validation (Mcp-Name should tolerate surrounding whitespace per RFC 9110 §5.5), completion-complete failing despite CompletionComplete being implemented, sep-2164 (the error data should echo the requested URI), and dns-rebinding-protection (non-localhost Host/Origin aren't rejected). One gotcha if you extend the baseline: a WARNING fails the gate even though it never appears in the pass/fail summary.

The bug it caught on day one

The 1.0 client probes with server/discover first and is supposed to fall back to initialize when an older server rejects the probe. Against a reference-implementation server, that fallback never happened — the client surfaced "No message available from the HTTP transport" instead. Two independent causes:

• identifiesLegacyServer() only recognised PARSE_ERROR, INVALID_REQUEST, and METHOD_NOT_FOUND. The reference server rejects with a server-defined code in the -32000..-32099 band, which is now also treated as a legacy-server rejection.
• The response loop in attempt() waited for a response whose id matched the request. JSON-RPC mandates id: null on an error response when the id can't be honored, so the loop never matched, spun until the transport gave up, and the JsonRpcException the fallback could catch never surfaced. The loop now also terminates on an error response carrying id: null.

Both shapes have regression tests in HandshakeTest, asserting the wire order ends up server/discover followed by initialize. In practice:

$client = Client::web('https://legacy.example.com/mcp')->withTimeout(30);

$client->connect();

$client->protocolVersion();                      // ProtocolVersion::V2025_11_25
$client->discoverResult();                       // null — the probe was rejected
$client->initializeResult()?->serverInfo->name;  // populated via the handshake fallback

Upgrade impact

No public API changes, so application code is untouched. If you hit "No message available from the HTTP transport" pointing the 1.0 client at an older or reference server, that's fixed here. The new composer scripts are contributor-facing, and the results, logs, and generated badges under tests/Conformance/ are gitignored. There is deliberately no CI workflow in this PR — that's an announced follow-up.

Takeaways

• composer test:conformance (or :server / :client) runs the official suite locally and fails on anything outside conformance-baseline.yml.
• The runner is pinned to the @alpha package and the 2026-07-28 revision; use --requirements, not --spec-version.
• Connecting to pre-server/discover servers works again: legacy detection covers -32000..-32099, and id: null error responses no longer hang the read loop.
• Five baselined failures are leads on real bugs (DNS-rebinding protection and completion among them); the rest are unimplemented features or fixture gaps.
• Category: 2 — Feature.

Sources

• Pull request #327
• Release v1.0.1