Desvela AI Crawler Block Test — free is a remote MCP server that gives an AI agent 4 tools, among them crawler_block_test, preflight, search. It answers at https://registry.desvela.dev/mcp/server-card over streamable HTTP, and it listed its tools to an anonymous client, so no sign-in was needed for that. Our probes reached it 10 of 10 times since 5 September 2026; the last check, on 18 October 2026, succeeded.
What a domain's CDN actually serves each AI crawler, as opposed to what its robots.txt declares. crawler_block_test(domain) sends the real user-agent of GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and CCBot at the homepage and compares each response against a browser's: a site can allow GPTBot in robots.txt and still have its CDN answer it a 403, and robots.txt cannot show that. Verdict per crawler: ok, blocked, throttled (a 429 means too fast, not not-you), degraded (a 200 with a fraction of the bytes), or error. When the site refuses the control request too, measurable is false and no ve
Description as the publisher wrote it.
| Tool | What it does |
|---|---|
crawler_block_test | What the domain's CDN really does, as opposed to what its robots.txt says. Sends each AI crawler's real user-agent at the homepage and compares the response against a browser: a site can allow GPTBot in robots.txt and still have its CDN ans Inputs: domain |
preflight | What a domain publishes for AI agents before you touch it: ai-catalog.json (ARD), llms.txt, agents.md, and its robots.txt directives for AI bots (GPTBot, ClaudeBot, Google-Extended…). Served from the Desvela ARD index when fresh (<7 days), Inputs: domain |
search | Natural-language search over what Desvela has indexed: agents, MCP servers, registries — the same index as POST /search. Returns catalog entries per the ARD spec. When nothing matches, returns referrals: other ARD-conformant registries you Inputs: text |
watch | Subscribe to a domain: get a signed webhook (X-Desvela-Signature, HMAC-SHA256) when what it publishes for AI agents changes — ai-catalog.json entries added/changed/gone, llms.txt or agents.md edited or removed, robots.txt AI-bot directives Inputs: domain, webhook_url |
Every block below points at https://registry.desvela.dev/mcp/server-card. Pick your client and copy.
claude mcp add --transport http desvela-ai-crawler-block-test-fr https://registry.desvela.dev/mcp/server-card
{
"mcpServers": {
"desvela-ai-crawler-block-test-fr": {
"command": "npx",
"args": ["-y", "mcp-remote", "https://registry.desvela.dev/mcp/server-card"]
}
}
}{
"mcpServers": {
"desvela-ai-crawler-block-test-fr": {
"url": "https://registry.desvela.dev/mcp/server-card"
}
}
}{
"servers": {
"desvela-ai-crawler-block-test-fr": {
"type": "http",
"url": "https://registry.desvela.dev/mcp/server-card"
}
}
}{
"context_servers": {
"desvela-ai-crawler-block-test-fr": {
"url": "https://registry.desvela.dev/mcp/server-card"
}
}
}{
"mcpServers": {
"desvela-ai-crawler-block-test-fr": {
"serverUrl": "https://registry.desvela.dev/mcp/server-card"
}
}
}{
"mcpServers": {
"desvela-ai-crawler-block-test-fr": {
"url": "https://registry.desvela.dev/mcp/server-card"
}
}
}name: Desvela AI Crawler Block Test — free
version: 0.0.1
schema: v1
mcpServers:
- name: Desvela AI Crawler Block Test — free
type: streamable-http
url: https://registry.desvela.dev/mcp/server-card{
"mcpServers": {
"desvela-ai-crawler-block-test-fr": {
"httpUrl": "https://registry.desvela.dev/mcp/server-card"
}
}
}{
"mcpServers": {
"desvela-ai-crawler-block-test-fr": {
"type": "http",
"url": "https://registry.desvela.dev/mcp/server-card"
}
}
}{
"mcpServers": {
"desvela-ai-crawler-block-test-fr": {
"url": "https://registry.desvela.dev/mcp/server-card"
}
}
}goose://extension?url=https%3A%2F%2Fregistry.desvela.dev%2Fmcp%2Fserver-card&type=streamable_http&id=desvela-ai-crawler-block-test-fr&name=Desvela%20AI%20Crawler%20Block%20Test%20%E2%80%94%20free
It answered our last check on 18 October 2026 in 74 ms. Over 10 checks since 5 September 2026 it answered 10. Status here comes from Neuronto's own probes, not from the publisher.
Yes. It is published by desvela.dev and served from registry.desvela.dev, the vendor's own domain: https://registry.desvela.dev/mcp/server-card.
4, as returned by the server's own tools/list on 31 August 2026: crawler_block_test, preflight, search, watch.
It answered our last check on 18 October 2026 in 74 ms. Over 10 checks since 5 September 2026 it answered 10.
Put the server under mcpServers in ~/.cursor/mcp.json with the URL https://registry.desvela.dev/mcp/server-card. The exact JSON is above, with the same setup for Claude Code, Claude Desktop, VS Code, Windsurf, Zed, Gemini CLI and the rest.
It listed its tools to an anonymous client, so reading the tool list needs no key. Individual tools can still ask for credentials in their own inputs.
Publisher: desvela.dev. Endpoint host: registry.desvela.dev. ARD identifier: urn:air:desvela.dev:registry:crawler-block.
First indexed 31 August 2026. The tools are exactly what the server returned; nothing on this page
is inferred from its name. Ranking in Neuronto search is never for sale.