Plain-language meaning
An instruction affecting crawling or indexing through robots.txt, a robots meta tag, or an X-Robots-Tag header.
Why it matters
Robots directive affects how a website is discovered, understood, used, maintained, or evaluated. In an Asuruas report, the term remains connected to evidence and context.
Use “Robots directive” precisely
In an audit, robots directive should name a specific condition, object, relationship, or decision—not serve as a vague synonym for website quality. Attach the term to the exact evidence and affected scope so a reviewer can understand what was observed and what remains uncertain.
Evidence to inspect for Robots directive
- Discovery paths, response codes, rendering, directives, canonicals, internal links, sitemaps, duplication, and selected urls.
- Page-template and url-pattern evidence rather than isolated examples.
- Document how the term affects the outcome to make the intended pages discoverable, interpretable, internally connected, and maintainable.
- Recrawl the affected patterns, inspect rendered output and directives, and monitor indexation and search behaviour over time.
Decision and verification record
Scope
Name the website, environment, URLs, entities, templates, or user journeys included in the robots directive decision.
Decision
Record the chosen action, owner, priority, dependencies, approval, and the evidence that justified it.
Verification
Recrawl the affected patterns, inspect rendered output and directives, and monitor indexation and search behaviour over time.
Turn robots directive into an accountable record.
A shared working definition of robots directive used consistently across reports and operational records.