CompletionKit is an MCP server with 54 tools an agent can call, among them agreements_create, agreements_list, datasets_create. It answers at https://completionkit.com/mcp over streamable HTTP, and it listed its tools to an anonymous client, so no sign-in was needed for that. Our probes reached it 10 of 10 times since 4 September 2026; the last check, on 18 October 2026, succeeded.
Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.
Description as the publisher wrote it.
| Tool | What it does |
|---|---|
agreements_create | Upsert an agreement for (run, response, metric, created_by). Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'. Inputs: run_id, response_id, metric_id, verdict, corrected_score, note |
agreements_list | List agreements. Filter by run_id, response_id, metric_id, or created_by. Inputs: run_id, response_id, metric_id, created_by |
datasets_create | Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, an Inputs: name, csv_data, tag_names |
datasets_create_from_url | Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is Inputs: name, url, tag_names |
datasets_delete | Delete a dataset Inputs: id |
datasets_get | Get a dataset by ID Inputs: id |
datasets_list | List all datasets |
datasets_update | Update a dataset Inputs: id, name, csv_data, tag_names |
judges_compare | Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics. Inputs: metric_id, metric_version_a_id, metric_version_b_id |
judges_replay | Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can co Inputs: name, metric_id, dataset_id, judge_model, output_column |
metric_groups_create | Create a metric group Inputs: name, description, metric_ids, tag_names |
metric_groups_delete | Delete a metric group Inputs: id |
metric_groups_get | Get a metric group by ID Inputs: id |
metric_groups_list | List all metric groups |
metric_groups_update | Update a metric group Inputs: id, name, description, metric_ids, tag_names |
metric_versions_dismiss | Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused — to demote a published version, publish a different one as current instead. Inputs: metric_version_id |
metric_versions_list | List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and timestamps. Inputs: metric_id |
metric_versions_publish | Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → current'. Transactionally flips current, demotes peers, and writes the version's instruction + rubric_ Inputs: metric_version_id |
metrics_create | Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or m Inputs: name, instruction, metric_type, rubric_bands, check_config, tag_names |
metrics_delete | Delete a metric Inputs: id |
metrics_get | Get a metric by ID Inputs: id |
metrics_list | List all metrics |
metrics_suggest_variants | Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft MetricVersion with source="suggestion". Returns the persisted drafts. Stripe-metering hooks fire vi Inputs: metric_id, count, model |
metrics_update | Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/equals), pattern (regex), json_path+expected (json_path_equals), min and/or max (length_bounds); valid Inputs: id, name, instruction, metric_type, rubric_bands, check_config |
promptfoo_import | Import a promptfooconfig.yaml. Creates a prompt, a dataset from the test vars, and metrics from the assert blocks (llm-rubric/g-eval become judge metrics; contains/equals/regex/is-json become deterministic check metrics). Returns a summary Inputs: config |
prompts_create | Create a prompt Inputs: name, description, template, llm_model, tag_names |
prompts_delete | Delete a prompt Inputs: id |
prompts_get | Get a prompt by ID Inputs: id |
prompts_list | List all prompts |
prompts_publish | Publish a prompt version, making it the current version Inputs: id |
prompts_suggest_improvement | Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and reviews, then returns reasoning plus a rewritten template (preserving {{variables}}) and persists it as Inputs: run_id |
prompts_update | Update a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing — promote it with prompts_publish — so an agent's edits don't go live without a gate. If it has no Inputs: id, name, description, template, llm_model, tag_names |
provider_credentials_create | Create a provider credential Inputs: provider, api_key, api_endpoint, api_version |
provider_credentials_delete | Delete a provider credential Inputs: id |
provider_credentials_get | Get a provider credential by ID (API key is not exposed) Inputs: id |
provider_credentials_list | List all provider credentials (API keys are not exposed) |
provider_credentials_update | Update a provider credential Inputs: id, provider, api_key, api_endpoint, api_version |
responses_get | Get a specific response Inputs: run_id, id |
responses_list | List responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are large: use "fields" to drop the bodies, "min_score"/"max_score" to isolate low scorers, and sort "sco Inputs: run_id, limit, offset, status, min_score, max_score |
runs_create | Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones. Inputs: name, prompt_id, dataset_id, judge_model, temperature, max_tokens |
runs_delete, runs_generate, runs_get, runs_list, runs_regrade, runs_rerun, runs_retry_failures, runs_update, tags_create, tags_delete, tags_get, tags_list, tags_update, usage_get
Every block below points at https://completionkit.com/mcp. Pick your client and copy.
claude mcp add --transport http completionkit https://completionkit.com/mcp
{
"mcpServers": {
"completionkit": {
"command": "npx",
"args": ["-y", "mcp-remote", "https://completionkit.com/mcp"]
}
}
}{
"mcpServers": {
"completionkit": {
"url": "https://completionkit.com/mcp"
}
}
}{
"servers": {
"completionkit": {
"type": "http",
"url": "https://completionkit.com/mcp"
}
}
}{
"context_servers": {
"completionkit": {
"url": "https://completionkit.com/mcp"
}
}
}{
"mcpServers": {
"completionkit": {
"serverUrl": "https://completionkit.com/mcp"
}
}
}{
"mcpServers": {
"completionkit": {
"url": "https://completionkit.com/mcp"
}
}
}name: CompletionKit
version: 0.0.1
schema: v1
mcpServers:
- name: CompletionKit
type: streamable-http
url: https://completionkit.com/mcp{
"mcpServers": {
"completionkit": {
"httpUrl": "https://completionkit.com/mcp"
}
}
}{
"mcpServers": {
"completionkit": {
"type": "http",
"url": "https://completionkit.com/mcp"
}
}
}{
"mcpServers": {
"completionkit": {
"url": "https://completionkit.com/mcp"
}
}
}goose://extension?url=https%3A%2F%2Fcompletionkit.com%2Fmcp&type=streamable_http&id=completionkit&name=CompletionKit
It answered our last check on 18 October 2026 in 472 ms. Over 10 checks since 4 September 2026 it answered 10. Status here comes from Neuronto's own probes, not from the publisher.
Yes. It is published by completionkit.com and served from completionkit.com, the vendor's own domain: https://completionkit.com/mcp.
54, as returned by the server's own tools/list on 10 September 2026: agreements_create, agreements_list, datasets_create, datasets_create_from_url, datasets_delete, datasets_get, datasets_list, datasets_update, and 46 more.
It answered our last check on 18 October 2026 in 472 ms. Over 10 checks since 4 September 2026 it answered 10.
Put the server under mcpServers in ~/.cursor/mcp.json with the URL https://completionkit.com/mcp. The exact JSON is above, with the same setup for Claude Code, Claude Desktop, VS Code, Windsurf, Zed, Gemini CLI and the rest.
It listed its tools to an anonymous client, so reading the tool list needs no key. Individual tools can still ask for credentials in their own inputs.
Publisher: com.completionkit. Endpoint host: completionkit.com. ARD identifier: urn:air:com.completionkit:mcp:evals.
First indexed 31 August 2026. The tools are exactly what the server returned; nothing on this page
is inferred from its name. Ranking in Neuronto search is never for sale.