ARD-Bench
A head-to-head retrieval measurement across the public ARD registries.
Ground truth is the publishers' own representativeQueries, so nothing is
hand-labelled, and the harness is open so the run can be reproduced or the task set
replaced.
- 120tasks
- 10results per query
- 6targets measured
| Registry | Coverage | recall@10 |
recall@10 when carried | MRR |
Median ms |
|---|
| neuronto (lexical only) | 1.0 | 0.9833 | 0.9833 | 0.9238 | 45 |
| neuronto (hybrid + federated) | 1.0 | 0.9667 | 0.9667 | 0.894 | 1912 |
| GitHub Agent Finder | 0.0083 | 0.0083 | 1.0 | 0.0083 | 1912 |
| WellKnown | 0.925 | 0.4833 | 0.5045 | 0.4211 | 153 |
| Hugging Face Discover | 0.0083 | 0.0083 | 1.0 | 0.0083 | 6299 |
| Desvela ARD Registry | 0.0917 | 0.0667 | 0.7273 | 0.0554 | 54 |
Read the conditioned column. `recall@k` is over all tasks and therefore mixes two things: whether a registry indexes the target at all, and whether it retrieves it. `coverage` is the fraction of targets a registry carries according to our own ingest records, and `recall@k_when_carried` is retrieval measured only on those. Compare the conditioned column. Our own coverage is 1.0 by construction, because tasks are built from our index, and that is an advantage of the task set, not a finding.
Known bias. queries are drawn from representativeQueries, and this registry weights that field highest in its own ranking (bm25 weight 9.0 of 9.0). Every registry indexes the same public manifests, so the field is equally available to all, but a registry that chooses not to weight it is penalised by our choice of ground truth. Read the absolute numbers with that in mind; the harness is published so the task set can be replaced.
Identifier matching. both sides normalised to the spec's urn:air: form before comparison, so a registry emitting urn:ai: is not scored as wrong
Harness:
https://github.com/neuronto/neuronto/blob/main/app/bench.py.
Machine-readable results:
https://neuronto.com/bench with
Accept: application/json.