Robots.txt Generator
Build a robots.txt file with allow/disallow rules and a sitemap URL.
What is Robots.txt Generator?
A robots.txt file is a plain-text file placed at the root of a website (e.g. example.com/robots.txt) that tells search engine crawlers which parts of the site they're allowed to request. This tool builds one for you: set which paths are allowed or disallowed for a user-agent, optionally point crawlers at your sitemap, and get back a ready-to-use robots.txt file — no need to remember the exact directive syntax or line ordering. It runs entirely in your browser, so nothing you type is sent anywhere; you just copy the result and upload it to your site's root.
How to Use Robots.txt Generator
- Leave User-agent as "*" to target all crawlers, or enter a specific bot name (e.g. "Googlebot").
- List the paths you want crawlers to be allowed to access in the Allow box, one per line.
- List any paths you want to keep crawlers out of in the Disallow box, one per line (e.g. /admin/).
- Optionally add your sitemap URL so crawlers can discover it directly from robots.txt.
- Copy the generated text and save it as robots.txt at the root of your site (e.g. https://example.com/robots.txt).
Examples
Allow everything, point to a sitemap
Input: User-agent: *, Allow: /, Sitemap: https://example.com/sitemap.xml
Output: User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Block an admin area
Input: User-agent: *, Disallow: /admin/
Output: User-agent: * Disallow: /admin/
Common Mistakes
- Uploading robots.txt somewhere other than the domain root — crawlers only check /robots.txt, not any subfolder.
- Relying on Disallow alone to hide sensitive or private content — robots.txt is publicly readable and not an access-control mechanism; anyone can view exactly which paths you're trying to hide.
- Disallowing a path that's also referenced in your sitemap — that sends crawlers a mixed signal about whether the page should be indexed.
- Forgetting that Disallow paths are case-sensitive and prefix-matched — /Private/ will not block /private/, and /admin will also match /admin-panel/ since it's a prefix match.
Why Use This Tool
- No syntax to memorize — fill in fields and get correctly formatted output.
- Runs entirely client-side; nothing you type is sent to a server.
- Normalizes paths (adds a leading slash) so small typos don't produce a broken file.
- One-click copy of the finished robots.txt, ready to upload.