WordPress generates a robots.txt file automatically, and most site owners never think about it again after launch. That’s fine right up until you need to block a staging subdomain from getting indexed, stop a crawler from hammering a resource-heavy search results page, or add a directive your SEO plugin’s basic interface doesn’t expose. At that point you need to know the difference between a virtual robots.txt and a physical one, because WordPress will silently ignore your manual edits if you don’t understand which one you’re dealing with.
Virtual vs. Physical: The Distinction That Trips Everyone Up
If there’s no actual robots.txt file sitting in your WordPress root directory, WordPress generates one dynamically on request through the do_robotstxt action, pulling from a default template and any modifications applied through plugin filters. Visit yoursite.com/robots.txt on a fresh install with no file present, and you’ll see WordPress’s generated output even though no file actually exists on disk, it’s assembled by PHP at request time, every time.
The moment a real, physical robots.txt file exists in your site’s root directory, WordPress’s virtual generation stops entirely. The web server serves the physical file directly, bypassing PHP and WordPress altogether, which also means any plugin filters hooking into robots_txt stop having any effect, since that filter only runs as part of the virtual generation process that a physical file supersedes. This is the single most common cause of “I edited robots.txt through my SEO plugin and nothing changed”, a leftover physical file from a previous plugin, a migration, or a manual edit is silently overriding everything the plugin tries to output.
Check which situation you’re in before doing anything else: connect via FTP or your host’s file manager and look directly in the WordPress root directory (the same folder containing wp-config.php) for a file literally named robots.txt. If it’s there, that’s the file actually being served, full stop, regardless of what any plugin setting says.
Method 1: Direct File Edit via FTP or File Manager
This is the most reliable method precisely because it sidesteps any ambiguity about virtual generation, plugin filters, or caching interference, you’re editing the actual file the server hands out.
Step 1: Connect to Your Site’s File System
Use an FTP client like FileZilla with credentials from your hosting provider, or use your host’s built-in File Manager if they offer one through the control panel (cPanel, Plesk, or a custom host dashboard). Either route gets you to the same place: direct access to the files WordPress lives in.
Step 2: Navigate to the WordPress Root
This is the directory containing wp-config.php, wp-admin, wp-content, and wp-includes as siblings, commonly public_html, though the exact path depends on your host’s directory structure. robots.txt, if it exists, sits alongside those, not inside wp-content or any subdirectory.
Step 3: Check for an Existing File and Handle It Accordingly
If a robots.txt file is already present, download a copy first as a backup before editing, in case you need to revert. If none exists, you’re creating a new physical file, which will immediately take over from WordPress’s virtual generation the moment it’s uploaded.
Step 4: Edit or Create the File Locally
Open the downloaded (or new, blank) file in a plain text editor, Notepad, TextEdit in plain-text mode, Sublime Text, VS Code, anything that won’t add rich-text formatting characters that corrupt the file. A minimal, correct robots.txt looks something like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap_index.xmlAdd specific Disallow lines for anything you want crawlers to skip, being careful with paths, since a Disallow rule blocks that path and everything beneath it. A stray Disallow: / at the top of the file blocks your entire site from every well-behaved crawler, which is a mistake worth triple-checking before you upload, since it’s happened to more than a few site owners who didn’t notice until organic traffic quietly cratered weeks later.
Step 5: Upload and Overwrite
Via FTP, drag the edited file into the root directory and confirm the overwrite when prompted. Via cPanel File Manager, use the upload function and confirm replacement of the existing file if one is present.
Step 6: Verify
Visit yoursite.com/robots.txt directly in a browser (not through wp-admin, the actual public URL) and confirm the content matches what you just uploaded. If caching is aggressive on your host or through a CDN like Cloudflare, you may need to purge cache for that specific URL to see the update reflected immediately, since robots.txt is a static file path that caching layers happily cache like any other.
Method 2: Through an SEO Plugin’s Built-In Editor
Yoast SEO, Rank Math, and All in One SEO all include a robots.txt editing interface under their respective tools sections, and this is the friendlier route for anyone not comfortable with FTP. Importantly, these plugins handle the virtual-vs-physical distinction for you in a specific way: if no physical file exists, editing through the plugin modifies WordPress’s virtual output via the robots_txt filter. If a physical file already exists on the server, most of these plugins will detect it and either offer to let you edit that physical file’s contents directly through their interface, or warn you that a physical file is present and taking priority over anything the plugin would otherwise generate.
Rank Math’s implementation, for example, explicitly reads and writes to a physical robots.txt file when it detects one, rather than trying to inject content through the virtual generation path, specifically to avoid the split-brain problem where the plugin’s settings say one thing and the actual served file says another. Whichever plugin you’re using, check its documentation for exactly how it behaves in the presence of an existing physical file, since this differs between plugins and has changed across versions.
Method 3: Custom Code in functions.php
For control that survives without depending on manually maintaining a static file, hooking into the do_robotstxt action lets you generate the content programmatically, which is useful if your robots.txt rules need to vary based on environment (staging vs. production) or site conditions:
add_action('do_robotstxt', 'custom_robots_txt');
function custom_robots_txt() {
echo "User-agent: *\n";
echo "Disallow: /wp-admin/\n";
echo "Allow: /wp-admin/admin-ajax.php\n";
echo "Sitemap: " . home_url('/sitemap_index.xml') . "\n";
}This approach only works, again, if no physical robots.txt file exists in the root directory, since the do_robotstxt action is part of the virtual generation path that a physical file bypasses entirely. If you’re going this route, make sure there’s no leftover physical file that would silently override the code you just wrote.
A more targeted alternative uses the robots_txt filter to modify the existing virtual output rather than replacing it wholesale, which is safer if you want to add one or two custom rules without rewriting WordPress’s sensible defaults from scratch:
add_filter('robots_txt', 'add_custom_robots_rule', 10, 2);
function add_custom_robots_rule($output, $public) {
$output .= "Disallow: /staging-preview/\n";
return $output;
}This is the recommended approach for most custom robots.txt needs, because it composes with WordPress’s default output and with any other plugin also filtering the same hook, rather than replacing everything and risking losing rules another plugin depends on.
Common Directives Worth Knowing
Blocking a staging environment entirely. User-agent: * followed by Disallow: / on a staging subdomain keeps search engines from indexing a duplicate copy of your production content, which matters because duplicate content across a staging and production domain can genuinely confuse search rankings and cannibalize your production site’s authority.
Blocking specific crawlers by name. If a particular bot is hammering your server with excessive requests, User-agent: BadBotName followed by Disallow: / targets just that crawler rather than blocking everyone. Note that robots.txt is an honor-system request, not an enforcement mechanism, well-behaved crawlers (Googlebot, Bingbot) respect it, but a genuinely malicious scraper ignoring robots.txt entirely needs actual server-level blocking (a firewall rule, fail2ban, or your host’s bot mitigation) rather than a polite request in a text file.
Pointing crawlers to your sitemap. A Sitemap: directive at the bottom of robots.txt is one of the standard ways search engines discover your XML sitemap without you needing to manually submit it through Search Console, and most SEO plugins add this automatically.
Preventing indexing of internal search results. Disallow: /?s= stops crawlers from indexing your internal search results pages, which tend to create low-value, near-duplicate content in search results and are generally worth excluding.
Mistakes That Cause Real Damage
Blocking CSS and JavaScript files. Older robots.txt conventions sometimes blocked /wp-includes/ or asset directories entirely, which used to be considered good practice for reducing crawl budget waste. Google explicitly recommends against this now, since Googlebot needs to load your CSS and JavaScript to properly render and evaluate the page the way a real visitor would; blocking those assets can hurt how your pages get evaluated for ranking rather than help.
Confusing Disallow with noindex. A Disallow rule tells crawlers not to fetch a page’s content, but a page can still appear in search results (title and URL only, no snippet) if other pages link to it, since Google can index the existence of a URL without ever crawling its content. If your actual goal is keeping a page out of search results entirely, use a noindex meta tag on the page itself (which requires the page to be crawlable so the crawler can see the noindex instruction) rather than, or in addition to, a robots.txt Disallow.
Forgetting that robots.txt is publicly visible. Anyone, including anyone with bad intent, can view your robots.txt file. Listing sensitive directory paths in Disallow rules as a way to “hide” them from view actually broadcasts exactly where those paths are to anyone looking, since the file is plain text sitting at a predictable, public URL.
Editing through a caching plugin without clearing cache afterward. If a caching plugin or CDN is caching the robots.txt response itself, an edit that looks correct in the file may not actually be what crawlers are receiving until the cache clears, which can create a confusing gap between what you changed and what search engines are actually respecting.
Robots.txt on Multisite Installs
WordPress multisite handles robots.txt with an extra layer of nuance worth understanding before you edit anything. In a subdirectory multisite install (example.com/site2/), there’s only one robots.txt for the entire network, served from the main site, because robots.txt as a standard only recognizes one file per domain regardless of how many WordPress “sites” live underneath it. Any per-subsite rules need to be handled through per-page noindex tags rather than robots.txt itself, since you can’t have a separate robots.txt for a path-based subsite.
In a subdomain multisite install (site2.example.com), each subdomain is technically its own domain from robots.txt’s perspective, so each subsite can have its own distinct robots.txt file, either through its own physical file (subdomain-specific hosting configurations permitting) or through the do_robotstxt hook firing per-site if you’re using virtual generation and checking get_current_blog_id() to vary the output by subsite.
Using WP-CLI for a Faster, Scriptable Edit
If you manage several WordPress installs and find yourself repeating this task, WP-CLI offers a faster path than manually connecting via FTP each time. There’s no dedicated wp robots command in WP-CLI core, but you can write the file directly through a shell command once you’re connected via SSH:
wp eval "file_put_contents(ABSPATH . 'robots.txt', \"User-agent: *\nDisallow: /wp-admin/\nAllow: /wp-admin/admin-ajax.php\nSitemap: \" . home_url('/sitemap_index.xml') . \"\n\");"This writes a physical robots.txt file directly using WordPress’s own ABSPATH and home_url() functions, which keeps the sitemap URL correct even across environments with different domains (useful if you’re running the same deployment script against staging and production, where the sitemap URL needs to differ automatically). For anyone managing a portfolio of WordPress sites where robots.txt needs to stay consistent, wrapping this in a small deployment script and running it across every site via WP-CLI’s --path flag for each install saves a meaningful amount of repetitive manual work compared to logging into each site’s file manager individually.
What Happens If You Get It Wrong
The failure modes here range from mildly annoying to genuinely damaging, and it’s worth understanding the actual severity of each before treating every mistake as equally urgent.
A typo in a specific Disallow path that doesn’t match your intended URL structure is low-severity: crawlers just won’t respect a rule that doesn’t match anything, so the practical effect is usually “the rule silently does nothing” rather than “something breaks.” Annoying to debug, not damaging.
An accidental site-wide Disallow (Disallow: / under the wildcard user-agent) is high-severity and time-sensitive: this tells every well-behaved crawler to stop crawling your entire site, and while it won’t instantly deindex existing pages, extended blocking eventually leads to search engines treating your content as stale or removed, and new content simply never gets discovered and indexed in the first place. If you catch this quickly (checking Search Console’s coverage report periodically is the way you’d notice), reverting the file and requesting a recrawl through Search Console limits the damage. Left in place for weeks, the recovery takes considerably longer than the mistake did to make.
A robots.txt file that returns a server error (500-series response) instead of valid content is treated cautiously by Google specifically: historically, Google’s crawlers would pause crawling entirely if robots.txt was unreachable, on the assumption that an error might be temporary and erring toward caution made sense. Current guidance has softened this somewhat for short-lived errors, but a robots.txt file that’s broken for an extended period (a PHP fatal error in a custom do_robotstxt hook, for instance) can meaningfully suppress crawling until it’s fixed. This is one more reason to prefer a static physical file for critical production robots.txt content over dynamic PHP generation that could break due to an unrelated code error elsewhere in your custom functions.
Common Questions
Does WordPress overwrite my custom robots.txt on update? No. WordPress core updates never touch a physical robots.txt file sitting in your root directory, and they don’t modify the virtual generation logic in a way that would silently override custom filters you’ve added through functions.php or a plugin. The only things that touch robots.txt are your own edits, an SEO plugin’s editor if you use one, or another plugin explicitly designed to modify it.
Can I have different robots.txt rules for different user agents in the same file? Yes, that’s standard robots.txt syntax. Each User-agent: line starts a new block of rules that apply only to that agent (or to all agents, for the * wildcard), so you can have one block disallowing a specific bot from a specific path while a separate block allows everyone else full access.
Why does my edited robots.txt still show old content after I uploaded the new file? Almost always a caching issue, either from a CDN like Cloudflare caching the response, a caching plugin treating robots.txt as a static asset to cache, or your browser itself caching the previous response. Purge relevant caches and check in an incognito window or with a cache-busting query parameter to confirm.
Should I just leave robots.txt at WordPress’s default and not touch it? For a lot of sites, yes, the default virtual output (disallowing wp-admin while allowing admin-ajax.php, which most sites need crawlable for certain AJAX-dependent front-end functionality) is entirely adequate. Custom edits are worth making specifically when you have a concrete reason: blocking a staging environment, excluding low-value internal search pages, or dealing with a specific misbehaving crawler. Editing it just because you can, without a specific goal, is more likely to introduce a mistake than to improve anything.
Testing Your Changes
Google Search Console’s robots.txt Tester (found under the legacy tools section, or via a direct URL check) lets you paste your file’s contents and test specific URLs against your rules before or after deployment, catching syntax errors and unintended blocking before they affect real crawl behavior. It’s worth running any significant robots.txt change through this before assuming it’s working correctly, since a small syntax mistake (a missing colon, an incorrectly cased directive) can silently break the intended behavior without any obvious error message.
Interesting Reads:
Can You Make a Living with WordPress on Upwork?