Debugging, evals, conformance and security scanning are four jobs. Picking the wrong tool means a green run that proves nothing.
"How do I test my MCP server" has at least four answers, and they are not substitutes. One shows you what your server returns. One measures how models behave against it. One checks it against the specification. One looks for prompt injection. We build one of these, npx -y @yawlabs/mcp-compliance, and the most useful thing I can tell you about it is which of the four questions it cannot answer.
This is the part that outranks every feature comparison below. The MCP specification's own versioning page states that the current protocol revision is 2026-07-28. The previous revision, 2025-11-25, is the one most tooling in this space was built against, including ours.
That is not a cosmetic bump. Reading the 2026-07-28 changelog, the revision removes the initialize and notifications/initialized handshake entirely, in favour of each request carrying its protocol version in _meta. It removes protocol-level sessions and the Mcp-Session-Id header. It removes ping and logging/setLevel. It replaces resources/subscribe with subscriptions/listen, adds a mandatory server/discover RPC, moves the resource-not-found error code from -32002 to -32602, and deprecates Roots, Sampling and Logging.
Now run our tool and read the header, not the grade:
$ npx -y @yawlabs/mcp-compliance@latest test --only lifecycle --format json \
-- npx -y @modelcontextprotocol/server-everything
"schemaVersion": "1",
"specVersion": "2025-11-25",
"toolVersion": "0.17.1",
"grade": "A",
"score": 100An A against the revision before the current one. For a server that still speaks 2025-11-25, which is most of them today, that result means what it says. For a server built to 2026-07-28 only, a suite that expects an initialize handshake and a session header would report failures for behaviour the new spec deliberately removed. I have not run that experiment, because I do not have a 2026-07-28-only server to point it at; that is an inference from the changelog above, not a measurement. Support for the new revision is tracked in the open and is not shipped in 0.17.1. Ask every tool in this category the same question before you trust its output.
The official MCP Inspector is the reference tool here, and at 2.6.0 it ships three surfaces from one package: a web UI, a scriptable CLI, and a terminal UI. It is by a wide margin the most-starred tool in this list -- 10,863 stars against MCPJam's 2,196 when I read GitHub on September 11, 2026 -- and its documentation is refreshingly clear about its own boundary. From the repository's own smoke-testing guide: "It is deliberately not a conformance suite -- it is the check you run on every commit and every deploy, in a few seconds, with no browser."
That is the right description. Point it at a server, call a tool, read the JSON-RPC. When something is broken, this is where you find out what is actually on the wire.
MCPJam is the most interesting tool in this list and the one least like the others. It runs evals: test cases with expected tool calls, scored across what its README describes as 16 client configurations and 170-plus models, with accuracy tracked over time. It also ships a guided OAuth debugger covering protocol versions 2025-03-26, 2025-06-18, 2025-11-25 and 2026-07-28.
A spec-conformance pass and a behaviour eval answer different questions. Your tool descriptions can be perfectly schema-valid and still cause the model to call the wrong tool half the time. Only the eval catches that.
This is our lane, and there is one other credible entry in it. Ours is 88 tests over transport, lifecycle, tools, resources, prompts, error handling, schema validation and security, scored A to F with required tests weighted at 70 percent. The suite is transport-gated rather than uniform, which is easy to verify yourself:
$ npx -y @yawlabs/mcp-compliance@latest test --list --transport http
85 tests would run for transport=http
$ npx -y @yawlabs/mcp-compliance@latest test --list --transport stdio
55 tests would run for transport=stdioA stdio server is not marked down for failing CORS, TLS or session-header tests it has no business implementing. The grading rubric and the rule catalog are published under CC BY 4.0 so you can audit or fork the scoring rather than take the letter on faith, which we wrote up in grading MCP servers A to F.
RHEcosystemAppEng/mcp-validation is the Python option: handshake and JSON-RPC conformance, capability testing, stdio plus HTTP and SSE with full OAuth 2.0 dynamic client registration and PKCE, validation profiles, and JSON reports with a validation checklist. Its OAuth transport support is more thorough than ours. It was last pushed in November 2025, so check its spec assumptions before relying on it.
Snyk Agent Scan, which is where the well-known invariantlabs-ai/mcp-scan repository now redirects, discovers the agent components installed on a machine and scans them for prompt injection, sensitive-data handling and malware hidden in natural language. Its own README carries a warning worth repeating, because it surprises people: "Scanning MCP configurations will execute the commands defined in them." It requires a Snyk token and installs via uvx or a standalone binary.
A spec suite will not find a poisoned tool description that is nonetheless schema-valid. This is a different job, and it belongs in your process too -- alongside the rest of the MCP security checklist.
| Tool | What it is for | What it will not tell you | License |
|---|---|---|---|
| MCP Inspector | Interactive and scripted inspection: web, CLI, TUI | Whether you comply with the spec; it says so itself | MIT, per its package.json |
| MCPJam | Evals across clients and models, OAuth conformance, CI | Whether the protocol details outside its checks are right | Apache 2.0, with part of its server and evals code under a separate license |
@yawlabs/mcp-compliance | 88 graded spec tests, 85 on HTTP and 55 on stdio, CI gating | Whether your tools do the right thing, or behave well for a model | MIT |
| mcp-validation | Python handshake and capability validation, deep OAuth support | A graded score, or anything about spec revisions after its last update | MIT |
| Snyk Agent Scan | Prompt injection and malware scanning across agent components | Whether the server is spec-correct or usable | Apache 2.0, requires an account |
| Your own unit tests | Whether your tools return correct results | Everything above | Yours |
The Inspector should be the first thing you install, not ours. It is maintained by the MCP project, it is the tool everyone else's bug reports are written against, and when a server misbehaves the fastest path to understanding is watching the raw messages rather than reading a grade. If you only ever install one thing, install that one.
MCPJam is ahead of us on the thing this post opened with: its OAuth conformance checks already name 2026-07-28, and ours does not. It also does something we have no answer to at all, which is telling you that your server is spec-perfect and still confusing models. Its hosted app and team workspaces make it the easier sell for a team that does not want to run anything locally.
mcp-validation covers OAuth dynamic client registration and PKCE flows in more depth than our security category does, and it is the natural pick if your stack is Python. Snyk Agent Scan has a threat model we do not attempt.
What we do that the others do not is produce a single reviewable number with a published rubric behind it, gate a pipeline on it (--strict, --min-grade, SARIF output for code scanning), and cite the spec section behind every test so a failure is arguable rather than mysterious. That is a narrow lane. It is not the whole job.
One command against any server, HTTP or stdio. The one-click install is on our MCP servers page, or add it alongside your other servers with Yaw MCP:
npx -y @yawlabs/mcp-compliance@latest test https://my-server.com/mcp
npx -y @yawlabs/mcp-compliance@latest test -- npx -y @modelcontextprotocol/server-filesystem /tmp
# CI: fail the build when a required test regresses
npx -y @yawlabs/mcp-compliance@latest test https://my-server.com/mcp --strict --min-grade BSource: github.com/YawLabs/mcp-compliance. Read the specVersion field before you quote the grade.
The current revision is 2026-07-28, per the versioning page on modelcontextprotocol.io. Plenty of deployed servers and clients still speak 2025-11-25, so the practical answer is both, and the important part is knowing which one a given tool checked. Any conformance tool worth using tells you in its output; ours puts specVersion in every JSON report.
No, and its own documentation says so. The Inspector repository's smoke-testing guide describes the CLI workflow as deliberately not a conformance suite: it is the check you run on every commit and every deploy, in a few seconds, with no browser. It is the best tool in the ecosystem for seeing what your server actually returns.
Yes. A conformance suite checks that you speak the protocol correctly. It has no idea whether your search tool returns the right rows, whether your pagination cursor is stable, or whether your write tool wrote the right thing. Those are your tests, and nothing in this category replaces them.
Jeff Yaw, Yaw Labs. Follow along at tokenlimit.news for weekly notes on AI infrastructure.
Published by Yaw Labs.