Place it at the website root
Publish the file at /robots.txt. Rules elsewhere will not act as the website’s robots file.
Free technical SEO tool
Robots.txt Generator & Tester
Create crawler directives, add your sitemap and test whether a path is allowed or blocked before publishing the file.
robots.txt preview
User-agent: * Disallow: /admin Disallow: /private Disallow: /search Allow: /
Test results follow common longest-match robots rules. Search engines may interpret unsupported directives differently.
Blocking is not de-indexing
A disallowed URL may still appear in search if other pages link to it. Use appropriate indexing controls for removal.