Public access comes first
A private browser visit can reveal a password wall, broken page or security challenge. It does not test every crawler. Keep customer areas and paid lessons protected. Improve a public description instead of exposing private content.
robots.txt and noindex do different jobs
robots.txt gives crawlers rules about requests. A noindex instruction tells supporting search services not to include a page in results, but they must be able to access that instruction. Blocking crawling is not a reliable way to keep confidential information private.
A checker finding no rule is not proof that every service has access. Network failures, firewall rules and individual crawler policies can differ. Search crawlers and training crawlers may also be controlled separately.
Ask for evidence from the service you care about
If you own the site, Google Search Console’s URL Inspection can help investigate Google’s view. It requires access to the property and does not measure other AI systems. A public page inspection is a useful starting point without an account, with a narrower scope.
Ask your website person to confirm the exact rule, affected page and crawler before changing access controls. Do not open all bots or remove security rules as a shortcut.
Made-up example / Course business
A hidden launch page
A creator’s public course sales page was left with a noindex setting after launch. Their website person confirms the instruction, removes it for that page and checks the published response. Paid lessons remain behind a login. Inclusion in search is still a separate decision.
One thing to try
Try it on one public page
- Open the intended public page while signed out.
- Check your platform’s search visibility settings for that page.
- If a report flags a rule, save the exact evidence and ask what it affects.
- Change only a confirmed unintended restriction with the site owner’s approval.
How to check the result
- Repeat the same request and inspect the affected rule.
- If you have Search Console access, inspect that URL there as a separate Google-specific check.
- Record unavailable crawler checks as unknown. An allowed request is not evidence of a future citation.
Keep the page address, date and evidence. A page improvement is not a promise of rankings or AI citations.
Words in this guide
Only if you want to look them up.
Sources and further reading
Official documentation for the general guidance. These sources are not inspection findings about your page.
Published Studio article, for further reading: Your Website Has Three Viewers Now. You Do Not Build Three Websites.
Want help with your own website?
Tell us what you’re stuck on →