Basic example
User-agent: *
Disallow: /private-preview/
Allow: /
Sitemap: https://example.com/sitemap.xmlImportant limitations
- A blocked URL may still be discovered through links.
- Agents may interpret or honor rules differently.
- Blocking can prevent a crawler from seeing a noindex directive.
- Sensitive content requires access control, not robots.txt.
Production review checklist
- Confirm the file is available at the root of each relevant hostname.
- Review rules by user-agent and deployment environment.
- Test important HTML, CSS, JavaScript, image, API, and rendering resources.
- Keep staging protections separate from production directives.
- Document why a path is blocked and who owns the rule.
- Review changes after migrations, CMS updates, security work, and launch automation changes.
Use the right control
Prevent crawling
Use robots.txt when the goal is to discourage compliant crawlers from requesting a path.
Prevent indexing
Use an index control on a crawlable response when the goal is to keep the page out of an index.
Protect information
Use authentication, authorization, and network controls. Robots directives are public instructions, not security.
Apply Robots.txt to a real website decision
Use robots.txt as a decision framework rather than a detached definition. Start with the user or business task, collect discovery paths, response codes, rendering, directives, canonicals, internal links, sitemaps, duplication, and selected URLs, document uncertainty, and decide what action is justified by the evidence.
Evidence
Page-template and url-pattern evidence rather than isolated examples.
Interpretation
Explain how the observed condition affects the intended outcome: make the intended pages discoverable, interpretable, internally connected, and maintainable.
Verification
Recrawl the affected patterns, inspect rendered output and directives, and monitor indexation and search behaviour over time.
Questions to answer about Robots.txt
- What user, search, commercial, compliance, or operational task does robots.txt affect?
- Which templates, URLs, entities, environments, or journeys are actually in scope?
- What evidence would distinguish a confirmed problem from the risk of changing directives or canonicals without understanding the intended URL model?
- Who owns the decision, implementation, approval, and follow-up?
- What measurement or retest will prove the change improved the intended outcome?
Turn robots.txt into an accountable record.
A documented decision or practice for robots.txt that can be applied to a real website.