Developer

Robots.txt Generator

Write a robots.txt that welcomes crawlers to your public pages, blocks the paths you choose, and points them to your sitemap. Copy the starter file.

Robots.txt Generator

Create a robots.txt file with allow, disallow, and sitemap rules.

ShareX LinkedIn Facebook

Site URL

Allow all public paths by defaultInclude sitemap reference

Extra allow rules (one per line)

Disallow rules (one per line)

Generate a crawl-control file for your site.

robots.txt

—

Output actions

Ready to copy output once your result appears.

One User-agent group and a sitemap on the origin

Robots.txt Generator writes a single group for User-agent: *. Allow all public paths by default starts checked, so the file includes Allow: /. Include sitemap reference starts checked, and the sitemap line is the origin of Site URL plus /sitemap.xml, not the path you typed. Disallow rules starts as /admin and /drafts on two lines. Extra allow rules starts empty. Copy output copies the file. There is no download and no crawl-delay field.

How to write the robots file

1Leave Site URL at https://toolbench.online, or replace it with an absolute URL. The path on that URL is not copied into the sitemap line. Only the origin is.
2Leave the two checkboxes checked, or clear them. Put extra Allow paths in Extra allow rules, one per line. Edit Disallow rules the same way. The disallow box already contains /admin and /drafts.
3Press Generate robots.txt. Success says robots.txt generated. A site value that does not parse says Enter a valid site URL before generating robots.txt. and clears the file. Copy output copies the lines. There is no download button.

What it is good for

• Starting from Allow: /, two disallow paths, and a sitemap on the prefilled origin.
• Adding a path that you typed without a leading slash, and seeing the slash inserted.
• Dropping the sitemap line, or the Allow: / line, by clearing the matching checkbox.

Example

Input

Site URL https://toolbench.online
Allow all public paths checked
Include sitemap reference checked
Extra allow rules empty
Disallow rules:
/admin
/drafts

Output

User-agent: *
Allow: /
Disallow: /admin
Disallow: /drafts
Sitemap: https://toolbench.online/sitemap.xml

This is the file you get by pressing Generate robots.txt without editing the form. The disallow box is already filled.

What each output line is

The first line is always User-agent: *. If Allow all public paths by default is checked, the next line is Allow: /. Then each non-blank line of Extra allow rules becomes Allow: and the path. Then each non-blank line of Disallow rules becomes Disallow: and the path. If Include sitemap reference is checked, the last line is Sitemap:, a space, the origin, and /sitemap.xml. Blank lines in either path box are dropped. Each path is trimmed. A path that does not already start with / gets one / in front, so assets becomes /assets. A path that already starts with / is not given a second slash.

The sitemap URL ignores the path, query, and hash of Site URL. https://toolbench.online/blog/ still produces Sitemap: https://toolbench.online/sitemap.xml. A port is part of the origin, so http://example.com:8080/path produces Sitemap: http://example.com:8080/sitemap.xml when the sitemap box is checked. You cannot type a different sitemap path. Clearing Include sitemap reference omits the line entirely. Clearing Allow all public paths omits Allow: / and does not omit the extra allow lines or the disallow lines.

The site field, two checkboxes, and path boxes

Site URL starts at https://toolbench.online. Allow all public paths by default and Include sitemap reference both start checked. Extra allow rules (one per line) starts empty, with the placeholder /public-assets. Disallow rules (one per line) starts as /admin and /drafts. Generate robots.txt does not run when you edit a box. The status starts as Generate a crawl-control file for your site. After a success it says robots.txt generated.

Copy output copies the robots.txt pane. There is no Download output button, so nothing on this page names a file. The order of the lines is fixed: the user-agent, the optional Allow: /, the extra allows in the order you typed them, the disallows in the order you typed them, then the optional sitemap. A Disallow: / that you type is still printed after Allow: / if that checkbox is on. The page does not delete the contradiction and does not decide which line a crawler will obey.

When this file is the right starter

Use it when you want one group and a sitemap pointer at the origin. It does not crawl the site, does not check that /admin exists, and does not read an existing robots file. The sitemap XML itself is built on the XML sitemap generator, which turns URL rows into a urlset. This page only writes the Sitemap: line. It does not embed that XML.

A canonical and an Open Graph block for one page are the meta tag generator. A redirect from an old path to a new path is the redirect and rewrite rule generator. Neither of those outputs belongs inside this robots file, and this page will not convert a redirect into a Disallow.

Missing slashes, the origin, and one group only

Every allow and disallow path is a single field. The page does not parse wildcards for you, and it does not reject them. If you type /drafts/*, the line is Disallow: /drafts/*. If you type admin without a slash, the line is Disallow: /admin. Spaces at the ends of a line are removed. A line that is only spaces is dropped. The two path boxes are independent, so the same path can appear once as Allow and once as Disallow if you type it in both.

There is only one user-agent, and it is *. There is no second group, no crawl-delay, and no comment line. An unusable Site URL, including an empty field, clears the pane and says Enter a valid site URL before generating robots.txt. The previous file is not kept. The page does not contact the host. Copy output is the only way to take the text off the page.

Rules and limitations

The group is User-agent: *. Allow: / is included only while its checkbox is on. Path lines gain a leading slash when they lack one. The sitemap line is origin plus /sitemap.xml, and only while its checkbox is on. The path of Site URL is not part of that sitemap URL.

A bad site URL clears the file. Contradictory Allow and Disallow lines are both printed, in the order above. There is no crawl-delay, no second user-agent, and no download.

Common problems

If the sitemap points at the site root when you typed a path, that is the origin rule. https://toolbench.online/blog/ still announces https://toolbench.online/sitemap.xml. This page cannot name a different sitemap file.

If Allow: / and Disallow: / are both present, both boxes contributed a line. The generator does not remove either one. Edit the disallow box or clear Allow all public paths, then generate again.

Frequently asked questions

What is in the file before I edit anything?

User-agent: *, Allow: /, Disallow: /admin, Disallow: /drafts, and Sitemap: https://toolbench.online/sitemap.xml. The disallow box is prefilled. Press Generate robots.txt to print it.

Can the sitemap point at a path I choose?

No. The line is always the origin of Site URL plus /sitemap.xml. A port on the origin is kept. The path, query, and hash of Site URL are not.

What if I omit the leading slash?

One slash is added. assets becomes Allow: /assets or Disallow: /assets, depending on which box you used. A path that already starts with / is left alone.

Does this page save robots.txt?

No. Copy output copies the text. There is no download button, no crawl-delay, and no second User-agent group.

Is my input uploaded or saved?

This tool processes input in your browser and does not save its input or output to local storage. Copy and download are user actions. The site uses analytics for page and action metadata and may use ads and production error reporting; see the Privacy Policy for those services.

About this tool

Maintained by Toolbench. For large or deeply nested files, split the input into smaller batches and verify representative results before using them in an import or application. Report reproducible problems through Contact.

Next steps