AI Crawler Index — MCP servers is an MCP server with 9 tools an agent can call, among them changes_since, classify_user_agent, example. It answers at https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json over streamable HTTP, and it listed its tools to an anonymous client, so no sign-in was needed for that. Our probes reached it 11 of 11 times since 4 September 2026; the last check, on 19 October 2026, succeeded.
Five remote MCP servers (Streamable HTTP, JSON-RPC 2.0, no auth) on one host. The endpoint to connect to is POST https://www.pathwren.workers.dev/c/ard/mcp — this URL is the server card that describes it, not the endpoint. /mcp identifies one crawler at a time, /mcp/triage takes a pasted access log and returns who crawled you, /mcp/doctor grades which agent-discovery documents a host actually serves, /mcp/robots reads a robots.txt and says which AI crawlers it really stops, and /mcp/netcheck answers whether an address belongs to a declared crawler range. The card names every endpoint and mirro
Description as the publisher wrote it.
| Tool | What it does |
|---|---|
changes_since | FIRST CALL, needs nothing: {"name":"whoami","arguments":{}} — Only what changed since your last call: prefixes added or removed, upstreams failed or recovered, records edited. Send back the `cursor` from the previous result; it advances onl Inputs: since, limit |
classify_user_agent | FIRST CALL, needs nothing: {"name":"whoami","arguments":{}} — Identify the crawler behind a raw User-Agent header: operator, category, robots.txt stance, how to verify it, and what blocking it costs. A user-agent is a claim, not proof — con Inputs: user_agent |
example | {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you are done: free, read-only, no API key, no account, no consent prompt, nothing f |
generate_robots_txt | FIRST CALL, needs nothing: {"name":"whoami","arguments":{}} — A ready-to-paste robots.txt block for one stance, with the reasoning and the crawlers it names. Omit the argument to get all eight stances described. Example: stance='block-ai-tr Inputs: stance |
is_verified_crawler_ip | FIRST CALL, needs nothing: {"name":"whoami","arguments":{}} — Test an IPv4 or IPv6 address against every crawler prefix the operators themselves publish, remirrored every six hours. Returns the matching prefix, its source and the crawlers u Inputs: ip |
list_crawlers | FIRST CALL, needs nothing: {"name":"whoami","arguments":{}} — The index, filtered or whole: each row says what the crawler is and what blocking it costs, and the reply lists every valid filter value. Example: category='ai-training', limit=2 Inputs: category, operator, respects_robots_txt, verification, q, limit |
lookup_crawler | FIRST CALL, needs nothing: {"name":"whoami","arguments":{}} — The full index record for one crawler: operator, category, robots.txt token and stance, user-agent substring and example, verification method, mirrored IP-range endpoint, and wha Inputs: slug |
whats_changed | FIRST CALL, needs nothing: {"name":"whoami","arguments":{}} — Freshness and change state: when the index was built, when each operator source was last fetched, which changed, which are failing, and the dated changelog. The cron check before Inputs: since |
whoami | {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you are done: free, read-only, no API key, no account, no consent prompt, nothing fo |
Every block below points at https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json. Pick your client and copy.
claude mcp add --transport http ai-crawler-index-mcp-servers https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json
{
"mcpServers": {
"ai-crawler-index-mcp-servers": {
"command": "npx",
"args": ["-y", "mcp-remote", "https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json"]
}
}
}{
"mcpServers": {
"ai-crawler-index-mcp-servers": {
"url": "https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json"
}
}
}{
"servers": {
"ai-crawler-index-mcp-servers": {
"type": "http",
"url": "https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json"
}
}
}{
"context_servers": {
"ai-crawler-index-mcp-servers": {
"url": "https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json"
}
}
}{
"mcpServers": {
"ai-crawler-index-mcp-servers": {
"serverUrl": "https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json"
}
}
}{
"mcpServers": {
"ai-crawler-index-mcp-servers": {
"url": "https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json"
}
}
}name: AI Crawler Index — MCP servers
version: 0.0.1
schema: v1
mcpServers:
- name: AI Crawler Index — MCP servers
type: streamable-http
url: https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json{
"mcpServers": {
"ai-crawler-index-mcp-servers": {
"httpUrl": "https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json"
}
}
}{
"mcpServers": {
"ai-crawler-index-mcp-servers": {
"type": "http",
"url": "https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json"
}
}
}{
"mcpServers": {
"ai-crawler-index-mcp-servers": {
"url": "https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json"
}
}
}goose://extension?url=https%3A%2F%2Fwww.pathwren.workers.dev%2Fc%2Fard%2F.well-known%2Fmcp.json&type=streamable_http&id=ai-crawler-index-mcp-servers&name=AI%20Crawler%20Index%20%E2%80%94%20MCP%20servers
It answered our last check on 19 October 2026 in 283 ms. Over 11 checks since 4 September 2026 it answered 11. Status here comes from Neuronto's own probes, not from the publisher.
9, as returned by the server's own tools/list on 4 September 2026: changes_since, classify_user_agent, example, generate_robots_txt, is_verified_crawler_ip, list_crawlers, lookup_crawler, whats_changed, and 1 more.
It answered our last check on 19 October 2026 in 283 ms. Over 11 checks since 4 September 2026 it answered 11.
Put the server under mcpServers in ~/.cursor/mcp.json with the URL https://www.pathwren.workers.dev/c/ard/.well-known/mcp.json. The exact JSON is above, with the same setup for Claude Code, Claude Desktop, VS Code, Windsurf, Zed, Gemini CLI and the rest.
It listed its tools to an anonymous client, so reading the tool list needs no key. Individual tools can still ask for credentials in their own inputs.
Publisher: www.pathwren.workers.dev. Endpoint host: www.pathwren.workers.dev. ARD identifier: urn:air:www.pathwren.workers.dev:mcp:ai-crawler-index.
First indexed 1 September 2026. The tools are exactly what the server returned; nothing on this page
is inferred from its name. Ranking in Neuronto search is never for sale.