Four free tools to check if AI engines can find, read and recognise a site. No API key.
Four free tools that answer one question: can AI engines find, read and recognise a website?
No account. No API key. No signup. Nothing here runs a language model, so nothing here costs you or us anything.
"Can ChatGPT read stripe.com?"
"Why does AI never mention my store?"
"Where is my brand missing outside my own site?"
Ask in plain language. Your AI client picks the right tool.
Requires Node.js 18+.
claude mcp add crawlbit -- npx -y crawlbit-mcp
Add to claude_desktop_config.json:
{
"mcpServers": {
"crawlbit": {
"command": "npx",
"args": ["-y", "crawlbit-mcp"]
}
}
}
Add to .cursor/mcp.json:
{
"mcpServers": {
"crawlbit": {
"command": "npx",
"args": ["-y", "crawlbit-mcp"]
}
}
}
Restart your client. Four tools appear.
| Tool | What it answers |
|---|---|
crawlbit_crawler_watch | Can GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended read this site? |
crawlbit_entity_check | Does this brand read as a clear entity AI can recognise? |
crawlbit_offpage_gaps | Where is this brand missing outside its own website? |
crawlbit_shopping | Can AI shopping agents understand and recommend these products? |
Check whether AI crawlers can read example.com
Does example.com read as a clear brand entity to AI?
Where is example.com missing off-page, and which gap should we close first?
Compare example.com and competitor.com on AI crawler access
The tools compose. A useful sequence is: crawler access first (a blocked crawler makes everything else pointless), then the entity check, then the off-page gaps.
Written plainly, because an SEO tool that overstates itself is worse than no tool.
crawler_watch reads robots.txt only. It cannot see blocking done at the CDN, firewall
or rate-limit layer. A site can pass here and still refuse the crawler in practice.entity_check reads what the site publishes about itself. It says nothing about the
brand's reputation across the rest of the web.shopping inspects published structured data. It cannot see a merchant feed submitted
privately to a platform.If a site scores well on all four and still is not cited, the answer is almost always the same and it is not technical: too few third-party pages mention it. These tools will tell you that honestly rather than sell you a fix that does not exist.
CrawlBit also runs a full technical audit and generates llms.txt. Both use a language model,
so both cost money per run, and neither is exposed here.
That is a deliberate line rather than a teaser. A free tool whose bill grows with its popularity
gets rate-limited, degraded or withdrawn the moment it succeeds, and the people who installed it
are the ones who pay for that. Everything in this server costs nothing to run, so nothing here
has to be taken back later. The audit and the llms.txt generator live at
crawlbit.app if you want them.
The API rate-limits per IP. Normal conversational use does not come close, and the server runs on your machine, so the budget is yours rather than shared. If you do hit it, you get a plain sentence rather than a stack trace.
| Variable | Default | Purpose |
|---|---|---|
CRAWLBIT_BASE_URL | https://www.crawlbit.app | Point the server at another instance |
The server runs locally and stores nothing. The only data leaving your machine is the URL you ask about, sent to the CrawlBit API to be analysed. There is no account, so there is nothing to tie a request to a person.
Built by CrawlBit, which measures AI visibility and does the off-page work that earns citations. The free tools here are the measurement half. The paid work is the other half, and you are not required to look at it to use this.
MIT licensed. Issues and pull requests welcome.
Source-derived launch command. Check the maintainer’s required arguments and credentials before running:
npx -y crawlbit-mcpMerge this template into ~/Library/Application Support/Claude/claude_desktop_config.json. Keep existing servers. Add any arguments, credentials, and permissions required by the maintainer; this template has not been install-tested.
{
"mcpServers": {
"io-github-amati032-dev-crawlbit": {
"command": "npx",
"args": [
"-y",
"crawlbit-mcp"
]
}
}
}Restart Claude Desktop completely for changes to take effect. Confirm the server appears connected in the client’s tool list, then try a read-only example from its documentation.
Claude Desktop setup referencecrawlbit-mcpnpmio.github.amati032-dev/crawlbit works with any MCP-compatible client. Copy the config snippet from the Configuration section above and add it to the file shown for your client, then restart the application.
~/Library/Application Support/Claude/claude_desktop_config.jsonRestart Claude Desktop completely for changes to take effect.~/.cursor/mcp.jsonRestart Cursor for changes to take effect..vscode/mcp.jsonReload VS Code window for changes to take effect.~/.codeium/windsurf/mcp_config.jsonRestart Windsurf for changes to take effect..mcp.jsonSave at the project root, then start Claude Code in that project and review the MCP server approval prompt. Keep real credentials out of shared files.