io.github.kriton-dev/ai-crawler-access-check
AI Crawler Checker — Robots.txt Policy for Six Bots
Is the AI Crawler Checker MCP server down? No. It is up and needs auth (HTTP 401), probed 2026-11-10.
Check declared robots.txt permissions for six AI crawlers at URL paths
needs auth probed 2026-11-14 · unchanged since 2026-11-10 · HTTP 401 · Bearer
Streamable HTTP error: Error POSTing to endpoint: {"error":"invalid_token","error_description":"Missing or invalid access token. Pass an Apify API token in the - Endpoint
- https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check (streamable-http)
- Host
- mcp.apify.com
- Getting access
- OAuth with dynamic client registration. A client such as Claude Code, Cursor or npx mcpcost auth https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check registers itself and opens the consent page; the publisher has nothing to set up for you.
Authorization server console-backend.apify.com, PKCE S256, grants authorization_code, refresh_token, urn:ietf:params:oauth:grant-type:device_code.
Scopes full_api_access.
Read from the server's public OAuth metadata, without a token. Method. - Website
- https://apify.com/hermeskd/ai-crawler-access-check
- Registry
- active, updated 2026-11-03 · registry entry
Compare this server with others.
More servers
Other registry entries on mcp.apify.com, the ones next to this name; all of them.
- Korea HS Code Import Requirements (세관장확인대상물품) io.github.kr-data-api/korea-hs-import-requirements · needs auth
- Korean Law in English — Statutes & Articles (법령 영문) io.github.kr-data-api/korea-law · needs auth
- Korea Medical Device Approval Lookup (MFDS 의료기기 품목허가) io.github.kr-data-api/korea-medical-device-approval · needs auth
- Rowfeed Search & Market Data io.github.kristerkb-pixel/rowfeed-seo-mcp · needs auth
- Article Extractor — Main Text, Dates & Markdown from URLs io.github.kriton-dev/article-extractor · needs auth
- Career Site Job Search — Verified Greenhouse & Lever Boards io.github.kriton-dev/ats-job-search · needs auth
- Speech to Text — File Transcripts, Speaker Labels & Subtitles io.github.kriton-dev/audio-to-transcript · not probed
- Broken Links Checker — Crawl Pages, Assets & Redirect Chains io.github.kriton-dev/broken-links · needs auth
Servers that mention crawler.
- Site SEO & AI Crawler Audit app.openkrill/site-check · alive
- AI Visibility Audit — Can AI crawlers read and cite your site? app.rowb.ai-visibility/audit · alive
- ActableSite AI Crawler Monitor com.actablesite/ai-crawler-robots-monitor · alive
- ActableSite AI Crawler Monitor com.actablesite/readiness · not found
- Apify MCP Server com.apify/apify-mcp-server · needs auth
- Brainiall Web com.brainiall/web · needs auth
- Chad Analytics com.chadanalytics/chad · not found
- Citevana AI visibility tools com.citevana/mcp · alive
Badge for the README
The result of the latest probe, the tool count and the token cost. The feed has an entry for every probe that changed them.
A mail when this server changes
Stops answering, comes back, tool set or cost changes. One confirmation mail first, a stop link in every mail.
For the publisher
- The server answers HTTP 401 to an unauthenticated initialize, so the probe cannot list its tools. That is correct for a private server. Some public servers answer initialize and tools/list without credentials and ask for auth on tool calls only; those get a token count here.
Fixed something? Probe it now, with the same steps as the daily run. This page updates with the next daily run.