What Googlebot Checker checks
- robots.txt — whether the file lets Googlebot crawl this very page, and which rule applied.
- Server response — the code, redirects and response time for Googlebot for smartphones or for computers.
- Indexing — noindex and nofollow in meta robots and X-Robots-Tag, canonical, title, description and H1.
- Comparison with a browser — whether the site blocks the bot and whether it serves it different content.
- IP authenticity — whether an address from your logs belongs to Google or is a Googlebot fake.
Why Googlebot cannot crawl a page
Kinds of Googlebot
Googlebot for smartphones crawls most sites first because Google indexes the mobile version. Googlebot for computers, Googlebot Image, Video and News have their own User-Agents. Other robots, such as AdsBot or Google-InspectionTool, also sign themselves with Google’s name. They do not all follow the same robots.txt rules, so it matters which bot exactly came to the site. Any User-Agent string can be broken down by the User-Agent parser.
How to verify it is the real Googlebot
A User-Agent is easy to fake, so a bot is not recognised by it. Google recommends two checks. The first: do a reverse DNS lookup of the IP, make sure the name ends in googlebot.com or google.com, then look that name up forward and get the same address. The second: compare the IP with the range files that Google publishes in JSON format. Our tool does both checks. Fake bots posing as Googlebot are usually scrapers and are safe to block.
Limits of the check
We only send Googlebot’s User-Agent from our own server, so sites that let the bot in by IP address may respond to us differently than to the real Googlebot. We do not run JavaScript but read the response HTML: pages that build their content in the browser are rendered by Googlebot itself. The response headers are shown in detail by HTTP Headers Checker, and whether a page is in Google is checked by the page indexing check.
What else StayIndexed can do
This page is a free tool from the StayIndexed service. In your account you can save a list of URLs, refresh the check in one click and see when what Googlebot sees has changed (also free), while the main job of the service is to track how your pages are doing in Google.
How much it costs
The check is free, both on this page and in your account. Tokens are used to pay for the other services: indexing, speed and backlinks cost 3 tokens per check. 30 tokens are credited at sign-up, no card required.
Frequently asked questions
What is Googlebot?
It is Google’s search robot that visits pages, downloads their content and passes it to the index. Its behaviour is governed by robots.txt, response headers and directives on the page.
How do I check whether Googlebot can see a page?
Enter the address in the form above: we will open it as Googlebot and check robots.txt, the server response, noindex and canonical. The most accurate answer comes from the URL inspection in Google Search Console, but it only works for your own site.
How do I know a visitor in my logs is the real Googlebot?
Check its IP: the reverse DNS must lead to googlebot.com or google.com, and a forward lookup of that name must return the same address. Or compare the address with the ranges Google publishes in JSON format. The block above does both checks.
Why does Googlebot get an error while a browser does not?
Most often the site or its protection (Cloudflare, a WAF, a security plugin) blocks bots by User-Agent or by request rate. Allow the real Googlebot IP addresses, not just the User-Agent.
Is checking the User-Agent enough?
No. Anyone can put anything in a User-Agent string, and scrapers often pose as Googlebot. A bot can only be reliably recognised by its IP address.
How much does Googlebot Checker cost?
Nothing: the check is free both on this page and in your StayIndexed account. In your account you can save a list of URLs, refresh the check in one click and see the change history.