Basic example
User-agent: *
Disallow: /private-preview/
Allow: /
Sitemap: https://example.com/sitemap.xmlImportant limitations
- A blocked URL may still be discovered through links.
- Agents may interpret or honor rules differently.
- Blocking can prevent a crawler from seeing a noindex directive.
- Sensitive content requires access control, not robots.txt.
Production review checklist
- Confirm the file is available at the root of each relevant hostname.
- Review rules by user-agent and deployment environment.
- Test important HTML, CSS, JavaScript, image, API, and rendering resources.
- Keep staging protections separate from production directives.
- Document why a path is blocked and who owns the rule.
- Review changes after migrations, CMS updates, security work, and launch automation changes.
Use the right control
Prevent crawling
Use robots.txt when the goal is to discourage compliant crawlers from requesting a path.
Prevent indexing
Use an index control on a crawlable response when the goal is to keep the page out of an index.
Protect information
Use authentication, authorization, and network controls. Robots directives are public instructions, not security.
Apply Robots.txt to a real website decision
Use robots.txt as a decision framework rather than a detached definition. Start with the user or business task, collect discovery paths, response codes, rendering, directives, canonicals, internal links, sitemaps, duplication, and selected URLs, document uncertainty, and decide what action is justified by the evidence.
Evidence
Page-template and url-pattern evidence rather than isolated examples.
Interpretation
Explain how the observed condition affects the intended outcome: make the intended pages discoverable, interpretable, internally connected, and maintainable.
Verification
Recrawl the affected patterns, inspect rendered output and directives, and monitor indexation and search behaviour over time.
Questions to answer about Robots.txt
- What user, search, commercial, compliance, or operational task does robots.txt affect?
- Which templates, URLs, entities, environments, or journeys are actually in scope?
- What evidence would distinguish a confirmed problem from the risk of changing directives or canonicals without understanding the intended URL model?
- Who owns the decision, implementation, approval, and follow-up?
- What measurement or retest will prove the change improved the intended outcome?
Expand the reach of Robots.txt
Search visibility and user value improve when robots.txt answers the real questions people bring to the page. For business owners, marketers, seo practitioners, and developers, that means covering the decision context, observable signals, implementation boundaries, and proof that the result works in production—not repeating a keyword or publishing a longer version of the same incomplete explanation.
Use the page as part of a connected topic cluster. Link the broad concept to focused implementation guides, definitions, checklists, examples, and the Asuruas workflow that can identify affected URLs. The goal is to help a reader move from discovery to a confident next action while giving search systems clear entities, relationships, and page purpose.
- Inspect crawl paths and status codes.
- Inspect indexability, canonicals, and robots directives.
- Inspect internal-link depth and sitemap coverage.
- Inspect metadata and search-intent alignment.
Strengthen the answer
Remove crawl waste and redirect chains.
Build the topic cluster
Strengthen discoverable topic clusters.
Prove the outcome
Verify the rendered production response after release.
Turn robots.txt into an accountable record.
A documented decision or practice for robots.txt that can be applied to a real website.