Ulvixorrobots.txt Generator
Create a custom robots.txt file for your own website, defining Allow and Disallow rules for each user-agent and adding your sitemap URL. Includes quick templates for the most common cases. Everything is generated locally in your browser, with no data sent to any server.
How it works
- Choose a quick template (allow everything, block everything, block common paths) as a starting point, or build your rules from scratch.
- Set the user-agent for each block (use * to apply the rule to all crawlers).
- Add Allow or Disallow rules with the corresponding path within each block.
- Add more user-agent blocks if you need different rules for specific crawlers.
- Optionally enter your sitemap URL, then copy or download the resulting robots.txt file.
Use cases
- Block crawlers from accessing administrative or private paths on your website.
- Allow full indexing of a new website while it doesn't yet have sensitive content.
- Set different rules for a specific bot (for example, an AI scraper) versus the rest of the crawlers.
- Declare the location of your sitemap.xml to make crawling and indexing your site easier.
Use cases
- Block crawlers from accessing administrative or private paths on your website.
- Allow full indexing of a new website while it doesn't yet have sensitive content.
- Set different rules for a specific bot (for example, an AI scraper) versus the rest of the crawlers.
- Declare the location of your sitemap.xml to make crawling and indexing your site easier.
Common mistakes
- Using Disallow thinking it keeps the page out of Google's search results.Disallow only tells the crawler not to visit that path; it doesn't stop the URL from being indexed if another site links to it. To prevent indexing, you also need a 'noindex' meta robots tag on the page.
- Accidentally blocking with Disallow: / on a site that actually wants to be indexed.That rule blocks every crawler across the entire site. Review each block before publishing the file; the 'Allow everything' template is a good starting point if you don't need restrictions.
- Defining rules only for a specific bot and forgetting to add a general block with User-agent: *.If you only define rules for one particular bot, every other crawler has no restrictions at all. Add a separate block with * if you also need general rules.
Frequently asked questions
No. The robots.txt file is built entirely in your browser using JavaScript. None of your rules or URLs are sent to any external server.
It should be uploaded to the root of your domain, for example https://yourdomain.com/robots.txt, so that crawlers find it automatically.
It doesn't guarantee that a page won't be indexed — it only tells crawlers not to visit that path. If you need to keep a page out of search results, also add a 'noindex' meta robots tag to that page.
Disallow tells crawlers not to access a given path. Allow explicitly permits access to a path, and is typically used to carve out an exception within a folder that would otherwise be blocked.
Alternatives
You can write a robots.txt file by hand in any text editor following the search engines' robots.txt specification, or generate one from an SEO plugin in your CMS. This tool helps you build the correct syntax with templates and multiple user-agent blocks without memorizing the exact format.