Test any MCP server against the MCP specification and get a letter grade back - from the terminal, from CI, or from your agent.
@yawlabs/mcp-compliance is an 88-test suite that checks a server against the MCP specification - transport, lifecycle, tools, resources, prompts, error handling, schema validation and security - and returns an A-F grade. It works against HTTP endpoints and stdio servers alike, and ships three interfaces: a CLI for humans, an MCP server for AI assistants, and a programmatic API. Built and maintained by Yaw Labs, MIT licensed, published on npm.
npx -y @yawlabs/mcp-compliance@latestGitHub · npm · Add to Yaw MCP
MCP servers are multiplying fast, and most ship without compliance testing. Broken transport handling, missing error codes, malformed schemas and silent capability violations are common, and hand-rolling a test script for each one is tedious and incomplete. This is the test suite you point at a server - yours or someone else's - before you trust it.
From the terminal it is one command. From an agent, it is a conversation:
mcp_compliance_test runs the suite against a URL or a stdio command and returns the grade, the score and per-test results.mcp_compliance_explain explains a specific test ID and the spec section behind it.1. No API key needed. The suite talks to the server you point it at, not to a Yaw Labs service. For an authenticated target, pass credentials per run with --auth or -H, or per stdio process with -E / --env-file.
2. Add the server to your client. In Claude Code, one line:
claude mcp add mcp-compliance -- npx -y @yawlabs/mcp-compliance@latest mcpOr a .mcp.json in your project root. On macOS, Linux and WSL:
{
"mcpServers": {
"mcp-compliance": {
"command": "npx",
"args": ["-y", "@yawlabs/mcp-compliance@latest", "mcp"]
}
}
}On Windows, use the cmd /c form:
{
"mcpServers": {
"mcp-compliance": {
"command": "cmd",
"args": ["/c", "npx", "-y", "@yawlabs/mcp-compliance@latest", "mcp"]
}
}
}This config file is safe to commit - it carries no secrets.
3. Restart and approve. Restart Claude Code (or your MCP client) and approve the server when prompted. Then ask it something: "Run the compliance suite against node ./dist/server.js and tell me which required tests fail."
Running Yaw MCP? One click adds it to your local config, and it is then available in every Yaw Terminal session.
Tests are organized into eight categories, and --only and --skip each take a comma-separated list of categories or individual test IDs:
| Category | What it checks |
|---|---|
transport | HTTP POST handling, content types, session enforcement, batch rejection, SSE event fields, concurrency - plus the stdio framing, UTF-8 roundtrip and unknown-method recovery checks |
lifecycle | Initialize handshake, protocol version, capabilities, JSON-RPC conformance, ping, ID matching, re-init rejection, cancellation and progress, logging and completions when declared |
tools | tools/list shape, tools/call response format, cursor pagination, content item types |
resources | resources/list and resources/read, templates, pagination, subscribe and unsubscribe when the capability is declared |
prompts | prompts/list, prompts/get message validity, pagination |
| Error handling | Unknown methods, malformed JSON-RPC, the correct -32601 / -32700 / -32600 codes, missing params, unknown tool names, capability-gated methods, invalid cursors |
| Schema validation | Tool name and inputSchema validity, annotations, title and outputSchema fields, prompt and resource schemas |
security | Auth enforcement and WWW-Authenticate, TLS, session entropy, OAuth Protected Resource Metadata, CORS and Origin validation, command / SQL / path-traversal / SSRF resistance, oversized input, tool rug-pull and description poisoning, stack-trace and internal-IP leakage, rate limiting |
The grade is a weighted score: required tests are worth 70% and optional tests 30%.
| Grade | Score |
|---|---|
| A | 90-100 |
| B | 75-89 |
| C | 60-74 |
| D | 40-59 |
| F | 0-39 |
The MCP server exposes the same engine as tools: mcp_compliance_test runs the suite against a URL or a stdio command with auth, headers, env vars, timeout, retries and category filters, and mcp_compliance_explain explains what a specific test ID checks and why it matters.
readOnlyHint, destructiveHint, idempotentHint and openWorldHint let a client skip confirmation dialogs for safe operations.--concurrency defaults to one test in flight. Raising it reduces wall time but can perturb timing-sensitive servers, which is the tradeoff to make deliberately rather than by default.schemaVersion and the spec version it tested against, with a published JSON Schema shipped in the package. Pin against the version and reject unknown ones rather than guessing at the shape.The CLI is the primary interface. Point it at an HTTP endpoint, or pass a stdio command and its arguments Inspector-style:
# A remote HTTP server
npx @yawlabs/mcp-compliance@latest test https://my-server.com/mcp
# A local stdio server - pass the command directly
npx @yawlabs/mcp-compliance@latest test npx @modelcontextprotocol/server-filesystem /tmp
# A local build, with env vars
npx @yawlabs/mcp-compliance@latest test -E API_KEY=secret -- node ./dist/server.jsFor a pipeline, --strict exits 1 on any required-test failure and --min-grade exits 1 when the grade slips below a threshold. Reports come out as terminal, json, sarif for GitHub Code Scanning, github for inline PR annotations, markdown for a comment body, or html for a self-contained report you can share anywhere.
- uses: YawLabs/mcp-compliance@v0
with:
target: 'node ./dist/server.js'
format: github
strict: 'true'
min-grade: 'A'Around that core there is a working set worth knowing about: init scaffolds a committable mcp-compliance.config.json so CI and your dev loop run with no arguments; --list prints the test IDs a filter would run without connecting to anything; --watch re-runs on file changes for the stdio dev loop; diff compares two JSON reports and fails on regressions; benchmark measures latency; and --output badge.svg writes a local badge reflecting the real grade, for committing next to your README. There is a Docker image too, if you would rather not install anything at all.
It runs an 88-test suite against any MCP server and returns an A-F grade with a weighted score. Tests cover transport, lifecycle, tools, resources, prompts, error handling, schema validation and security, and every one of them links to the section of the MCP specification it validates. It works against HTTP endpoints and stdio servers alike.
No. The same package is a CLI, an MCP server and a programmatic API. Run mcp-compliance test against a URL or a stdio command from your terminal, drop it into a pipeline with --strict, or call runComplianceSuite from TypeScript. The MCP server exists so an agent can run the suite and explain individual test IDs in the same turn.
No. The suite is capability-driven: tests adapt to what the server declares. If it says it supports tools, the tool tests become required; if it never claims the feature, it is not marked down for it. Transport matters too - HTTP-specific tests like CORS, TLS, session headers and rate limiting are gated out when the target is a stdio server, so a stdio run executes fewer tests than an HTTP one.
Use the GitHub Action, or call the CLI directly. --strict exits 1 on any required-test failure and --min-grade exits 1 when the grade drops below a threshold you set. Output formats include json, sarif for GitHub Code Scanning, github for inline PR annotations, markdown for a comment body, and html for a self-contained report. There is also a diff command that compares two JSON runs and exits 1 if anything that was passing is now failing.
Yes. The testing methodology and the machine-readable rule catalog are published under CC BY 4.0, so anyone can inspect the scoring, build compatible tooling, or fork the rules. The project is explicit that this is not an authoritative conformance standard: the official MCP specification defines what servers must do, and the methodology describes how this tool verifies it.
Published by Yaw Labs.