Crawlability
Also called Crawling, Crawler access, AI crawlers
Crawlability is whether a particular crawler can reach and fetch a page and the resources it needs.
Why it matters
A page can work in your browser while a firewall, authentication step, or robots rule prevents a crawler from reaching it.
Example (illustrative)
A public guide returns a normal page to people, but a security rule challenges automated requests.
What to check
Check response codes, robots rules, and security logs for the relevant crawler, and verify crawler identities against the provider’s documentation.
What this does not tell you
Access does not guarantee indexing or citation, and different crawlers can have different permissions and purposes.
Sources and scope
Documented concept · Sources checked
- Google Search Central: how Search works Platform documentation
Find Out How AI Sees Your Brand
AI is already shaping how buyers discover and evaluate your business. Start with a free AEO audit to see how ChatGPT, Perplexity, and Gemini represent your brand today.
Get Your Free AI Visibility Check
HubSpot Solutions Partner. Zero fluff.
