Can an AI agent find out what the most visited websites offer, without scraping their pages? We check the Tranco top 10,000 (list V354N) every month for every file and record built for that purpose. Of the 6,935 domains that answered as a website, 74 publish a machine-readable agent file. Measured 2026-11-02 19:04:56 UTC.
| Signal | Top 100 48 sites | Top 1,000 598 sites | Top 10,000 6,935 sites |
|---|---|---|---|
| llms.txt /llms.txt, a Markdown summary of the site written for language models | 6 (12.5%) | 87 (14.5%) | 727 (10.5%) |
| MCP server card /.well-known/mcp.json or /.well-known/mcp/server-card.json | 1 (2.1%) | 7 (1.2%) | 50 (0.7%) |
| A2A agent card /.well-known/agent-card.json or /.well-known/agent.json | 1 (2.1%) | 2 (0.3%) | 11 (0.2%) |
| ARD manifest /.well-known/ard.json, the Agentic Resource Discovery manifest | 0 (0.0%) | 2 (0.3%) | 11 (0.2%) |
| ARD manifest, older path only /.well-known/ai-catalog.json, the name used before ARD v0.91 | 1 (2.1%) | 2 (0.3%) | 18 (0.3%) |
| API catalog (RFC 9727) /.well-known/api-catalog, a linkset of the site's APIs | 0 (0.0%) | 9 (1.5%) | 53 (0.8%) |
| Link: rel="ard" a header on the root pointing at the ARD manifest | 0 (0.0%) | 1 (0.2%) | 3 (0.0%) |
| SVCB _index._agents a DNS record naming where the ARD manifest is served | 0 (0.0%) | 0 (0.0%) | 11 (0.2%) |
| Any agent file ARD manifest, MCP card, A2A card or API catalog | 2 (4.2%) | 11 (1.8%) | 74 (1.1%) |
Shares are of the sites that answered over HTTPS. Of the 10,000 ranked domains, 2,090 answered no web request at all (most are CDN, DNS and telemetry hosts with no website) and 975 redirect to a different domain, so they are left out of every denominator.
596 of the 6,935 sites answer /.well-known/ard.json with
200 and an HTML page: their homepage or an app shell, served for any path. A scanner that counts status
codes reports every one of them as an ARD publisher. This index counts a file only when its content has the
right shape, which is why its numbers are small.
Every site in the population with each signal, with its Tranco rank.
cloudflare.com #2, github.com #34, adobe.com #62, wordpress.org #66, digicert.com #73, opera.com #77, sentry.io #102, intuit.com #115, samsung.com #122, dropbox.com #134, webex.com #144, gravatar.com #167, ubuntu.com #168, shopify.com #171, kaspersky.com #172, stripe.com #206, hcaptcha.com #228, weather.com #238, roku.com #239, amplitude.com #246, sourceforge.net #252, hubspot.com #260, linktr.ee #264, datadoghq.com #282, slack.com #304, etsy.com #334, unisys.com #337, bilibili.com #348, ea.com #350, dell.com #361, cm.com #366, paloaltonetworks.com #367, okta.com #394, name.com #399, wps.com #405, dynatrace.com #415, calendly.com #416, dailymotion.com #445, bluehost.com #470, taboola.com #475, businessinsider.com #477, shein.com #483, trendmicro.com #496, foxnews.com #507, braze.com #513, capcut.com #542, klaviyo.com #554, cloudns.net #565, dreamhost.com #579, newrelic.com #586, squarespace.com #594, atlassian.com #602, target.com #610, wix.com #613, sophos.com #637, onetrust.com #648, stackadapt.com #663, optimizely.com #666, bitrix24.ru #672, redhat.com #680, ring.com #682, surveymonkey.com #690, fontawesome.com #731, gumgum.com #741, asus.com #746, mailchimp.com #777, zoho.com #794, moloco.com #802, prnewswire.com #810, fiverr.com #818, allegro.pl #819, agora.io #841, auth0.com #849, frontiersin.org #870, unpkg.com #873, pendo.io #887, cloudinary.com #898, porkbun.com #909, lg.com #916, otto.de #925, seedtag.com #928, classlink.com #929, tailscale.com #930, repubblica.it #941, instructure.com #952, trendyol.com #985, flightradar24.com #1,000, xing.com #1,010, kwai.com #1,018, moe.video #1,060, netgear.com #1,063, typeform.com #1,135, kslawin.com #1,165, jimdo.com #1,188, siemens.com #1,205, coursera.org #1,213, one.com #1,214, intercom.com #1,215, home.pl #1,222, marktplaats.nl #1,226, savefrom.net #1,260, rajasthan.gov.in #1,282, polybuzz.ai #1,291, oxylabs.io #1,336, haberler.com #1,341, bookmyshow.com #1,378, sportybet.com #1,388, notion.com #1,411, protothema.gr #1,414, ilmeteo.it #1,425, coinbase.com #1,426, sozcu.com.tr #1,446, loopia.se #1,454, sondakika.com #1,496, revenuecat.com #1,504, programme-tv.net #1,528, markt.de #1,536, musicdel.ir #1,542, rakuten-sec.co.jp #1,549, alura.com.br #1,564, therapservices.net #1,586, chordtela.com #1,607, nustargame.com #1,621, adroll.com #1,636, eff.org #1,712, sambad.com #1,776, sattamatkano1.me #1,788, rapid7.com #1,791, bigcommerce.com #1,803, xfinity.com #1,813, dns-shop.ru #1,828, apkpure.com #1,830, pewresearch.org #1,834, hostinger.com #1,837, bitdefender.com #1,843, screenconnect.com #1,848, utorrent.com #1,853, wpastra.com #1,879, clevelandclinic.org #1,916, meetup.com #1,923, oreilly.com #1,939, surfshark.com #1,950, umeng.com #1,983, squareup.com #1,987, mongodb.com #1,997, trip.com #1,998, poste.it #1,999, sendbird.com #2,004, amocrm.ru #2,021, logitech.com #2,027, simpli.fi #2,053, customer.io #2,063, rustdesk.com #2,069, getbootstrap.com #2,082, yotpo.com #2,084, posthog.com #2,087, verify.hn #2,090, acronis.com #2,099, lenta.com #2,100, crowdstrike.com #2,119, coinmarketcap.com #2,145, iubenda.com #2,154, typing.com #2,158, americanexpress.com #2,174, unity.com #2,181, adkernel.com #2,195, mgid.com #2,200, nielsen.com #2,209, expedia.com #2,213, websupport.sk #2,221, nodejs.org #2,229, fliphtml5.com #2,246, gdemoideti.ru #2,267, dmac-solutions.net #2,269, sattamatka.email #2,283, adp.com #2,294, semrush.com #2,299, grafana.com #2,300, evernote.com #2,327, dondominio.com #2,346, verkada.com #2,389, ringcentral.com #2,394, mindbox.ru #2,405, goguardian.com #2,406, egnyte.com #2,432, canonical.com #2,446, dpbossss.com #2,456, gameanalytics.com #2,459, dpbosssss.mobi #2,495, su89-cdn.net #2,499, polymarket.com #2,506, kbz0pwvxmv.com #2,507, fotmob.com #2,513, atom.com #2,532, nordvpn.com #2,558, x-cdn-static.com #2,567, c6gj-static.net #2,568, digg.com #2,574, okx.com #2,578, ifeng.com #2,599, yg5sjx5kzy.com #2,625, tenable.com #2,640, parallels.com #2,664, slashdot.org #2,665, weplayapp.com #2,695, fullstory.com #2,707, zscaler.com #2,709, padlet.com #2,723, databricks.com #2,737, here.com #2,747, hindustantimes.com #2,750, cookieyes.com #2,756, malwarebytes.com #2,762, webflow.com #2,772, robinhood.com #2,778, dnsimple.com #2,781, suno.com #2,811, cutt.ly #2,831, rudderstack.com #2,840, guard.io #2,847, shazam.com #2,864, prezi.com #2,865, kpmg.com #2,879, newsbreak.com #2,914, clickup.com #2,919, gumroad.com #2,925, loom.com #2,938, redis.io #2,942, lokalise.com #2,948, tutu.ru #2,953, phys.org #2,958, fox.com #2,962, crealitycloud.com #2,968, crazyegg.com #2,980, libsyn.com #2,993, zapier.com #2,999, dantri.com.vn #3,025, ucoz.ru #3,029, ip.sb #3,033, sardine.ai #3,037, askona.ru #3,039, vercel.com #3,059, snapcraft.io #3,065, ihc.ru #3,079, drpc.org #3,092, cvent.com #3,107, fool.com #3,128, laracasts.com #3,170, imagemagick.org #3,174, cursor.com #3,202, walkme.com #3,221, monday.com #3,225, clickpetroleoegas.com.br #3,229, gldn.net #3,244, iterable.com #3,269, revcontent.com #3,271, elevenlabs.io #3,289, backblaze.com #3,291, clerk.com #3,294, proximus.be #3,311, hotmart.com #3,339, lavanguardia.com #3,342, razorpay.com #3,395, honor.com #3,396, postman.com #3,441, e-recht24.de #3,445, mobilefuse.com #3,449, twingate.com #3,456, jio.com #3,458, hootsuite.com #3,460, xero.com #3,472, joinmassive.com #3,506, pushwoosh.com #3,546, dinahosting.com #3,553, si.com #3,559, 17track.net #3,572, qnap.com #3,580, osano.com #3,582, sprinklr.com #3,585, voximplant.com #3,587, smartsheet.com #3,600, easydns.com #3,639, vkusvill.ru #3,642, bitwarden.com #3,652, arduino.cc #3,661, redbull.com #3,666, replit.com #3,670, demandbase.com #3,673, verisoul.ai #3,676, piano.io #3,684, lovable.dev #3,698, qualcomm.com #3,711, adyen.com #3,723, edx.org #3,727, dev.to #3,742, farlightgames.com #3,784, colorlib.com #3,786, familysearch.org #3,788, sanity.io #3,791, rokt.com #3,798, rome2rio.com #3,810, blackboard.com #3,832, domaintools.com #3,833, nazwa.pl #3,839, trimble.com #3,849, mux.com #3,880, sproutsocial.com #3,887, mysterium.network #3,893, elementor.com #3,904, nanit.com #3,907, remotepc.com #3,913, openrouter.ai #3,921, gamma.app #3,923, g2.com #3,930, admaster.cc #3,931, tum.de #3,939, mygaru.com #3,946, netlify.com #3,967, razer.com #4,012, elespanol.com #4,015, dramaboxdb.com #4,029, rapidssl.com #4,037, op.gg #4,053, lazada.com.ph #4,058, ctrip.com #4,077, tanium.com #4,097, similarweb.com #4,098, db.com #4,099, veeam.com #4,112, wiki.gg #4,135, garant.ru #4,151, grandviewresearch.com #4,155, ubs.com #4,163, helpshift.com #4,165, ts.net #4,200, xe.com #4,201, 1und1.de #4,209, mailerlite.com #4,220, giantpanda.com #4,224, zoneedit.com #4,225, lazada.sg #4,228, growthbook.io #4,268, livemint.com #4,285, producthunt.com #4,318, buffer.com #4,336, higgsfield.ai #4,364, hyundai.com #4,372, crypto.com #4,390, cloudflarestatus.com #4,392, victoriassecret.com #4,411, rediff.com #4,412, paylocity.com #4,431, picsart.com #4,435, ollama.com #4,464, 3ds.com #4,510, samsara.com #4,527, kaggle.com #4,534, terabox.com #4,564, lufthansa.com #4,565, vinted.co.uk #4,567, changeip.com #4,592, kayak.com #4,594, parentsquare.com #4,605, adspower.net #4,607, livechat.com #4,609, stlouisfed.org #4,634, recruit.co.jp #4,651, zoominfo.com #4,683, impact.com #4,684, salesmartly.com #4,690, yango.com #4,703, thetrainline.com #4,736, boredpanda.com #4,737, oppo.com #4,762, tally.so #4,767, westernunion.com #4,772, freshworks.com #4,778, dashlane.com #4,788, patch.com #4,806, vedomosti.ru #4,817, localytics.com #4,840, supabase.com #4,844, opencode.ai #4,864, kion.ru #4,869, thescore.com #4,913, wolfram.com #4,952, qualys.com #4,958, securly.com #4,960, dynadot.com #4,996, depositphotos.com #5,006, infobip.com #5,023, theweathernetwork.com #5,024, classdojo.com #5,025, zalando.de #5,115, framer.com #5,118, serverwatch.com #5,185, findmykids.org #5,214, airtel.in #5,241, bankrate.com #5,263, prodigygame.com #5,265, imperial.ac.uk #5,274, superbet.ro #5,295, zebra.com #5,303, uservoice.com #5,319, granola.ai #5,329, kijiji.ca #5,343, remitly.com #5,361, bittorrent.com #5,363, easybell.de #5,381, hola.org #5,384, skylinewebcams.com #5,402, yesstyle.com #5,409, bambuser.com #5,433, umt.edu #5,461, immowelt.de #5,477, lesoir.be #5,485, monster.com #5,490, narvar.com #5,504, tradedoubler.com #5,513, tradeinn.com #5,517, freeipapi.com #5,518, tuoitre.vn #5,543, techrepublic.com #5,545, vietnamnet.vn #5,568, nownodes.io #5,580, getcomposer.org #5,582, vuejs.org #5,583, outlookindia.com #5,602, glance.com #5,609, canadiantire.ca #5,612, swarovski.com #5,625, hostnet.nl #5,656, justgiving.com #5,659, imunify360.com #5,673, warwick.ac.uk #5,700, turbify.com #5,702, decodo.com #5,704, pubnub.com #5,710, uzh.ch #5,714, empik.com #5,716, maryland.gov #5,720, thinkific.com #5,724, credit-agricole.fr #5,729, mobalytics.gg #5,733, lazada.co.th #5,759, beehiiv.com #5,765, crossref.org #5,768, azbyka.ru #5,773, kucoin.com #5,778, zee5.com #5,801, pixai.art #5,815, groupon.com #5,826, n8n.io #5,847, ravelry.com #5,855, oilprice.com #5,857, milanuncios.com #5,863, ziprecruiter.com #5,866, boingboing.net #5,870, corel.com #5,872, mentalfloss.com #5,873, manageengine.com #5,882, airbridge.io #5,883, 9gag.com #5,912, netshort.com #5,924, mexc.com #5,929, thriftbooks.com #5,962, konghq.com #5,971, navyfederal.org #5,973, klook.com #5,975, wyze.com #5,984, formstack.com #5,989, transcend.io #6,011, ricoh.com #6,034, onamae.com #6,037, airwallex.com #6,055, cleverreach.com #6,060, ifixit.com #6,069, cosme.net #6,088, jooble.org #6,102, beacons.ai #6,119, solitaired.com #6,126, motorola.com #6,130, esri.com #6,140, goodnotes.com #6,162, dnb.com #6,185, playerok.com #6,202, aylanetworks.com #6,203, claro.com.br #6,209, camscanner.com #6,214, logitechg.com #6,225, genially.com #6,232, icims.com #6,262, uu.se #6,272, mynet.com #6,283, tsn.ua #6,295, piwik.pro #6,296, rtbhouse.com #6,301, 1984.is #6,315, tenjin.com #6,329, optum.com #6,347, td.com #6,359, muumuu-domain.com #6,365, fantasypros.com #6,378, tripo3d.ai #6,408, imo.im #6,411, dm.de #6,434, deepmind.google #6,460, mistral.ai #6,467, allegro.cz #6,484, hextom.com #6,487, thanhnien.vn #6,489, gmw.cn #6,504, webnode.com #6,508, wyndhamhotels.com #6,530, narto-drama.com #6,593, cimri.com #6,596, b92.net #6,632, pair.com #6,634, gameforge.com #6,645, cricketwireless.com #6,646, quran.com #6,666, prismic.io #6,680, nykaa.com #6,687, heute.at #6,716, openweathermap.org #6,717, smile.io #6,744, maersk.com #6,748, kiro.dev #6,773, yoast.com #6,780, onedio.com #6,834, 2dehands.be #6,843, orangelogic.com #6,867, tag24.de #6,869, drift-pixel.ai #6,880, mango-office.ru #6,887, me-qr.com #6,907, bamboohr.com #6,940, batch.com #6,942, trovaprezzi.it #6,958, claro.com.ar #6,971, programmaticx.ai #6,977, fastly.com #6,980, medicalxpress.com #6,986, noip.com #6,999, ixigo.com #7,007, qoo10.jp #7,010, carvana.com #7,045, tchibo.de #7,055, mql5.com #7,076, cafef.vn #7,080, bain.com #7,087, nic.ai #7,103, dellin.ru #7,112, almalinux.org #7,114, ledger.com #7,116, hackernoon.com #7,120, monetixads.com #7,137, amarujala.com #7,155, toptal.com #7,191, pik.ru #7,192, akc.org #7,240, suse.com #7,242, ngrok.com #7,260, starfieldtech.com #7,289, thredup.com #7,318, thehackernews.com #7,345, simple.life #7,385, vedantu.com #7,413, opentext.com #7,422, ukrainealarm.com #7,433, gitbook.com #7,438, streamlabs.com #7,447, apollo.io #7,450, palmettostatearmory.com #7,483, ig.com #7,518, dinamalar.com #7,545, ethereum.org #7,555, elfsight.com #7,597, link.me #7,612, booklive.jp #7,616, halodoc.com #7,650, seaart.ai #7,656, tuya.com #7,672, dxc.com #7,677, openart.ai #7,687, kenh14.vn #7,694, queensu.ca #7,702, survicate.com #7,716, trezor.io #7,723, getresponse.com #7,748, bibliaonline.com.br #7,751, react.dev #7,756, district.in #7,757, thenounproject.com #7,760, annas-archive.gl #7,795, mk.co.kr #7,825, catho.com.br #7,840, blackmagicdesign.com #7,842, make.com #7,854, telcel.com #7,860, cloudamqp.com #7,867, wordstream.com #7,881, imovelweb.com.br #7,884, soha.vn #7,934, pmi.org #7,945, sojern.com #7,963, complex.com #7,968, anker.com #7,974, icicidirect.com #7,986, autotrader.co.za #7,993, cashify.in #7,996, sbicard.com #8,013, alchemy.com #8,018, mattel.com #8,049, fikfap.com #8,070, umbc.edu #8,071, arm.com #8,079, dorzeczy.pl #8,088, yesplay.bet #8,096, cars24.com #8,099, storiesig.info #8,117, speakerdeck.com #8,123, webim.ru #8,161, yad2.co.il #8,164, uptimerobot.com #8,189, huanqiu.com #8,194, 1024tera.com #8,231, pocketfm.com #8,243, booksy.com #8,248, terabox.app #8,249, bikedekho.com #8,252, tubidy.com #8,304, allegro.hu #8,326, zougla.gr #8,372, timmehosting.de #8,394, doz.pl #8,407, anonymised.io #8,408, flexport.com #8,425, bebee.com #8,438, roborock.com #8,447, linear.app #8,452, linksys.com #8,461, linklist.bio #8,469, ilbe.com #8,482, onlinejobs.ph #8,484, getjobber.com #8,523, olist.com #8,526, pipedrive.com #8,532, autozone.com #8,597, anonyig.com #8,641, daisonet.com #8,646, surahquran.com #8,666, win2day.at #8,680, lcw.com #8,723, hyros.com #8,773, sssinstagram.com #8,796, harness.io #8,809, yoomoney.ru #8,814, freejobalert.com #8,841, decathlon.in #8,863, shop-pro.jp #8,865, abhibus.com #8,881, eufy.com #8,882, illumin.com #8,913, solitairegrandharvest.com #8,933, axismaxlife.com #8,951, sexiha.cam #8,968, delicious.com #8,976, banco.bradesco #8,980, startech.com.bd #8,981, qt.io #8,990, okbet.com #9,003, sexbebin.com #9,014, photoroom.com #9,035, javgg.net #9,048, sportybet.co.za #9,061, steamunlocked.org #9,063, savefrom.co.id #9,077, nobat.ir #9,091, apuestatotal.com #9,093, chiikawamarket.jp #9,103, vivastreet.co.uk #9,129, gumtree.com #9,147, pocketcasts.com #9,151, aniagotuje.pl #9,182, gogoroyal.com #9,210, beijing.gov.cn #9,264, megagroup.ru #9,277, arabnews.com #9,298, q10.com #9,299, rosszlanyok.hu #9,302, dg-datenschutz.de #9,304, creativethemes.com #9,323, modelcontextprotocol.io #9,337, alerts.in.ua #9,409, ecoflow.com #9,419, bezeq.co.il #9,450, havadurumu15gunluk.xyz #9,452, mxtoolbox.com #9,487, classificadosx.net #9,498, cleantalk.org #9,504, riyasewana.com #9,512, travesticomlocal.com.br #9,537, garotocomlocal.com.br #9,538, modaselvim.com #9,575, august.com #9,580, enloteria.com #9,697, artsy.net #9,705, petsmart.com #9,710, arabic-porn-tube.com #9,713, rozbano.com #9,714, underarmour.com #9,756, discoverpilgrim.com #9,777, nishatlinen.com #9,780, espacioapk.com #9,800, healthychildren.org #9,825, saya.pk #9,862, megaescort.info #9,870, zellbury.com #9,874, uncenz.com #9,878, faymas.in #9,882, elgato.com #9,895, wri.org #9,920, checkout.com #9,927, wakelet.com #9,962, gaiaonline.com #9,985, xxxza.com #9,991
cloudflare.com #2, sentry.io #102, forter.com #207, amplitude.com #246, linktr.ee #264, cm.com #366, zoom.com #833, twilio.com #1,137, intercom.com #1,215, oxylabs.io #1,336, notion.com #1,411, coinbase.com #1,426, hostinger.com #1,837, wpastra.com #1,879, kraken.com #1,967, customer.io #2,063, padlet.com #2,723, cookieyes.com #2,756, webflow.com #2,772, gumroad.com #2,925, crazyegg.com #2,980, zapier.com #2,999, monday.com #3,225, elevenlabs.io #3,289, clerk.com #3,294, razorpay.com #3,395, hootsuite.com #3,460, wistia.com #3,590, openrouter.ai #3,921, netlify.com #3,967, mailerlite.com #4,220, opensea.io #4,569, supabase.com #4,844, infobip.com #5,023, granola.ai #5,329, nownodes.io #5,580, glance.com #5,609, decodo.com #5,704, beehiiv.com #5,765, icons8.com #6,207, haveibeenpwned.com #6,669, openweathermap.org #6,717, cal.com #6,897, ngrok.com #7,260, gitbook.com #7,438, getresponse.com #7,748, alchemy.com #8,018, uptimerobot.com #8,189, pocketcasts.com #9,151, modelcontextprotocol.io #9,337
cloudflare.com #2, forter.com #207, medal.tv #1,850, cookieyes.com #2,756, monday.com #3,225, razorpay.com #3,395, opensea.io #4,569, glance.com #5,609, vidyard.com #7,816, modelcontextprotocol.io #9,337, alerts.in.ua #9,409
cm.com #366, bluehost.com #470, gameanalytics.com #2,459, cookieyes.com #2,756, hootsuite.com #3,460, elementor.com #3,904, openrouter.ai #3,921, netlify.com #3,967, supabase.com #4,844, pixai.art #5,815, arabnews.com #9,298
github.com #34, amplitude.com #246, huggingface.co #1,078, oxylabs.io #1,336, hostinger.com #1,837, bitdefender.com #1,843, padlet.com #2,723, rudderstack.com #2,840, airtable.com #2,929, zapier.com #2,999, vercel.com #3,059, clerk.com #3,294, beehiiv.com #5,765, haveibeenpwned.com #6,669, cal.com #6,897, ngrok.com #7,260, gitbook.com #7,438, alchemy.com #8,018
sentry.io #102, forter.com #207, amplitude.com #246, linktr.ee #264, cm.com #366, name.com #399, bluehost.com #470, redhat.com #680, zoom.com #833, huggingface.co #1,078, intercom.com #1,215, oxylabs.io #1,336, coinbase.com #1,426, hostinger.com #1,837, bitdefender.com #1,843, wpastra.com #1,879, kraken.com #1,967, customer.io #2,063, coinmarketcap.com #2,145, padlet.com #2,723, cookieyes.com #2,756, crazyegg.com #2,980, zapier.com #2,999, vercel.com #3,059, monday.com #3,225, elevenlabs.io #3,289, clerk.com #3,294, razorpay.com #3,395, hootsuite.com #3,460, wistia.com #3,590, sanity.io #3,791, openrouter.ai #3,921, netlify.com #3,967, mailerlite.com #4,220, zoneedit.com #4,225, buffer.com #4,336, cloudflarestatus.com #4,392, opensea.io #4,569, supabase.com #4,844, infobip.com #5,023, decodo.com #5,704, beehiiv.com #5,765, pixai.art #5,815, icons8.com #6,207, haveibeenpwned.com #6,669, onedio.com #6,834, cal.com #6,897, ngrok.com #7,260, gitbook.com #7,438, getresponse.com #7,748, uptimerobot.com #8,189, librarything.com #9,168, faymas.in #9,882
cm.com #366, netlify.com #3,967, pixai.art #5,815
hostinger.com #1,837, customer.io #2,063, padlet.com #2,723, cookieyes.com #2,756, coingecko.com #3,292, domain.com #4,207, opensea.io #4,569, 5ch.io #5,619, networksolutions.com #5,800, ngrok.com #7,260, faymas.in #9,882
4,928 of the 6,935 sites serve a robots.txt; 1,095 have a group that names at least one AI crawler and disallows the whole site to it. Named means a group names the crawler without blocking all of it.
| Crawler | Blocked | Share | Named, not blocked |
|---|---|---|---|
CCBot | 854 | 12.3% | 134 |
Bytespider | 839 | 12.1% | 82 |
GPTBot | 812 | 11.7% | 300 |
ClaudeBot | 763 | 11.0% | 225 |
Google-Extended | 681 | 9.8% | 247 |
meta-externalagent | 660 | 9.5% | 96 |
Applebot-Extended | 646 | 9.3% | 97 |
PerplexityBot | 416 | 6.0% | 274 |
ChatGPT-User | 335 | 4.8% | 306 |
OAI-SearchBot | 235 | 3.4% | 307 |
Claude-SearchBot | 219 | 3.2% | 135 |
curl -s https://example.com/llms.txt | head -3 curl -sI https://example.com/.well-known/ard.json | grep -i content-type curl -s https://example.com/.well-known/mcp.json curl -s https://example.com/.well-known/agent-card.json
Or run the free audit. To publish an ARD manifest, see how to publish or the manifest generator. How the sites that already publish one meet the new IETF drafts is tracked daily on the ARD IETF readiness tracker.
Download the full measurement as JSON (summary, history and one row per domain) or CSV, under CC BY 4.0. Please cite it as Neuronto Agent Readiness Index, 2026-11-02, https://neuronto.com/agent-readiness-index.
The Tranco top list by its permanent id, each domain checked once over HTTPS. A file counts only when its content has the right shape: JSON with an entries array for an ARD manifest, a name plus url or skills for an A2A agent card, a linkset for an API catalog, text starting with a heading for llms.txt. A 200 answered with an HTML page is a soft 404, not a file. Redirects are followed only within the domain. Percentages use the domains that answered HTTPS as the denominator. Each domain is requested once, over HTTPS, with an identifying user agent. A JSON file counts
as an ARD manifest when it has an entries array; as an A2A agent card when it has a
name and a url or skills; as an API catalog when it has a
linkset. An llms.txt counts when its first line is a Markdown heading and it is not HTML. A DNS
answer counts only when its record type is SVCB. robots.txt is read as RFC 9309 groups; a crawler counts as
blocked when a group naming it disallows /.