Compose robots.txt rules per crawler with allow and disallow paths and a sitemap reference, without hand-writing the syntax.
Content last reviewed
Configure access for search engine bots.
Use * for all bots, or specify (e.g. Googlebot, Bingbot)
Adds `Allow: /` to the rule block.
Points search engines directly to your XML sitemap for better indexing.
Open robots editor
Monaco loads only after you open the editor.
User-agent: * Allow: / Disallow: /admin Disallow: /private Disallow: /login Sitemap: https://example.com/sitemap.xml
Set the user-agent (* for all bots), toggle full access, and list the paths to disallow, one per line.
Click Add User-Agent to give a specific crawler like Googlebot or GPTBot its own rule block.
Add your sitemap URL, copy the generated output, and save it as robots.txt at your site's root.
All bots allowed generally, with three utility paths excluded and a sitemap reference.
Input
User-agent: * · Allow full access: on · Disallow: /admin, /private, /login · Sitemap: https://motifuse.com/sitemap.xmlOutput
User-agent: *
Allow: /
Disallow: /admin
Disallow: /private
Disallow: /login
Sitemap: https://motifuse.com/sitemap.xmlA second rule block that disallows everything for a single named bot.
Input
Second block — User-agent: GPTBot · Allow full access: off · Disallow: /Output
User-agent: GPTBot
Disallow: /The robots.txt file is the first thing a crawler requests when it visits your domain — a plain-text set of rules saying which paths bots may fetch and which they should skip. The format looks trivial, but the stakes are not: one stray Disallow: / blocks your entire site from being crawled, and subtle formatting mistakes silently disable rules. This generator builds the file from a form, so the syntax is always valid.
Each rule block targets a user-agent — * for all bots, or a specific name like Googlebot or GPTBot. Within a block you can toggle an explicit Allow: / directive and list disallowed paths one per line, and you can add as many blocks as you need: a common pattern is generous rules for search engines alongside a stricter block for a specific crawler you want to keep out. A dedicated field appends the Sitemap: line, which tells search engines where to find your XML sitemap without waiting for them to discover it.
The finished file assembles live in a syntax-highlighted panel as you edit, ready to copy and save as robots.txt at the root of your domain. One distinction worth internalizing before you rely on the file: robots.txt controls crawling, not indexing — a blocked page can still appear in search results if other sites link to it. For keeping pages out of results entirely, a noindex directive is the right tool, and the comparison below spells out the difference.
In the first rule block, set the User-Agent: keep * to address all crawlers, or type a specific bot name such as Googlebot or Bingbot.
Toggle Allow full crawling access to include an explicit Allow: / line, and list any Disallow paths one per line — for example /admin or /private.
Click Add User-Agent to create additional rule blocks for other bots; each block gets its own access toggle and disallow list. Hover a block and use the trash icon to remove it.
Enter your Sitemap URL so the file points crawlers at your XML sitemap.
Review the live output in the code panel, click Copy, and save the contents as a file named robots.txt in the root directory of your website — it must be reachable at /robots.txt on your domain.
After deploying, test the file with Google Search Console's robots.txt report before relying on it.
Create separate rule sets per crawler — permissive for search engines, restrictive for bots you want to exclude — with add and remove controls.
A toggle adds Allow: / and a one-path-per-line textarea builds the Disallow list, so directive syntax is never typed by hand.
One field appends the Sitemap: line that helps search engines discover your pages faster.
The file assembles in a syntax-highlighted panel as you edit, with blocks separated by blank lines exactly as the format expects.
Set up sensible crawl rules and a sitemap pointer before search engines first visit.
Keep crawlers out of admin panels, login pages, cart flows, and internal search results that waste crawl budget.
Give a specific crawler its own block — for example disallowing GPTBot while leaving search engines unrestricted.
Rebuild a hand-edited file whose rules stopped working, with syntax guaranteed by the form.
The generator walks your rule blocks in order and emits the directives for each: a User-agent: line (defaulting to * if the field is blank), an Allow: / line when the access toggle is on, and one Disallow: line per non-empty path in the textarea, with whitespace trimmed. Blocks are separated by blank lines, which is how crawlers tell one rule group from the next. If the sitemap field is filled, a Sitemap: line is appended at the end.
When a crawler fetches the file, it looks for the most specific user-agent group that matches its own name and follows only that group's rules — which is why a Googlebot block completely replaces the * rules for Google rather than adding to them. Paths are prefix matches: Disallow: /admin covers /admin, /admin/, and everything beneath it.
Standard robots exclusion format: User-agent, Allow, and Disallow directives in blank-line-separated groups, plus a Sitemap reference.
Free text — * for all bots or a specific crawler name. A blank field falls back to *.
Toggle that adds an explicit Allow: / directive to the block.
One path per line; empty lines are skipped and whitespace is trimmed.
Any number of rule blocks; the remove control appears when more than one exists.
Appended as a Sitemap: line when filled; omitted when empty.
A Disallow: / line in the * block tells every crawler to skip everything. Use / as a disallow path only in blocks targeting bots you genuinely want to exclude completely.
Expected behavior — robots.txt prevents crawling, not indexing. A disallowed URL can be indexed from external links, appearing with no description. Use a noindex meta tag or header to keep a page out of results.
Crawlers follow only their best-matching user-agent group. If you added a Googlebot block, Google ignores the * block entirely — repeat any general disallows inside the specific block too.
It must be served at exactly /robots.txt on the domain root. A file in a subdirectory, or on a different subdomain than the pages, is never consulted.
The generator covers the widely supported directives — User-agent, Allow, Disallow, and Sitemap. It does not emit Crawl-delay, wildcards inside paths, or comment lines, though you can add those by hand after copying. Output is copy-only: you save it as robots.txt yourself. And the format itself has inherent limits worth remembering — directives are advisory (well-behaved crawlers honor them, malicious ones do not), blocking crawl does not remove indexed URLs, and the file is publicly visible, so disallow paths reveal what you are trying to keep quiet.
Rule editing and file generation happen entirely in your browser; the paths and URLs you enter are not uploaded or stored by the tool. Note the separate, inherent point about robots.txt itself: once deployed, the file is public on your domain, so anyone can read the paths you list in it.
Test the deployed file in Google Search Console's robots.txt report before trusting it.
Use noindex, not Disallow, to keep a page out of search results — robots.txt only controls crawling.
Repeat general disallows inside bot-specific blocks; a matched block replaces the * rules, not adds to them.
Never list secret or sensitive paths — the file is public and effectively advertises them.
Keep the Sitemap line current so crawlers always find your latest sitemap.
Start from what should be crawled, not what should be blocked. Most sites need only a handful of disallows — admin areas, cart and checkout flows, internal search pages — and blocking more than that usually hurts discovery without any benefit. If your instinct is to disallow half the site, the actual problem is usually indexing (solved with noindex) or access control (solved with authentication).
Structure blocks deliberately. Keep the * block as your baseline policy, and add named blocks only for bots that need a genuinely different policy — remembering that a named block fully replaces the baseline for that bot, so copy the shared disallows into it.
After deployment, verify twice: fetch /robots.txt in a browser to confirm it is served as plain text, then check the parsing in Search Console. Re-test after site migrations — robots.txt files pointing at pre-migration paths are one of the most common silent SEO regressions.
The file is assembled client-side from the rule-block state: each block contributes a User-agent line (blank input falls back to *), an optional Allow: / line, and trimmed Disallow lines with empty entries filtered out; blocks are joined with blank-line separators and the Sitemap line is appended last. Output renders in a lazily loaded read-only Monaco editor. Nothing is uploaded, and the tool performs no validation of whether paths exist on your site.
Disallow (robots.txt) | noindex (meta tag / header) | |
|---|---|---|
| Controls | Whether bots fetch the page | Whether the page appears in results |
| Blocked page can still be indexed? | Yes, via external links | No |
| Requires the page to be crawlable? | No | Yes — the crawler must see the directive |
| Best for | Utility paths, crawl-budget waste | Pages that must stay out of search |
The two interact awkwardly: a page that is both disallowed and noindexed keeps its noindex invisible, because the crawler never fetches it. Pick the one that matches your actual goal.
Webmasters and developers setting up crawl rules for new sites, SEO practitioners fixing crawl-budget waste or accidental blocks, and site owners who want a specific crawler excluded without risking hand-written syntax. It suits anyone who knows which paths should be off-limits but does not want to memorize the directive format.
At the root of your domain, reachable at the path /robots.txt. Crawlers request exactly that address — a file in a subfolder is never read. Each subdomain needs its own file.
The asterisk is a wildcard matching every crawler that does not have a more specific block of its own. Rules under it form your default policy; a named block like User-agent: Googlebot overrides it entirely for that bot.
No — it stops the page being crawled, but the URL can still be indexed from links elsewhere and shown without a description. To keep a page out of results, let it be crawled and add a noindex meta tag or X-Robots-Tag header.
You can add rule blocks for their published user-agent names — GPTBot is a common example — with Disallow: /. Compliance is voluntary, as with all robots.txt rules, but the major AI crawlers document their user-agents and state that they honor the file.
Not strictly — crawling is allowed by default for anything not disallowed. The explicit line does no harm and makes the intent of a block obvious to human readers, which is why the toggle exists.
Crawlers cache robots.txt — Google typically refreshes it within about a day. After deploying a change, expect a delay before behavior shifts, and verify with Search Console rather than by watching server logs minute to minute.