How to edit robots.txt for individual sites in a WordPress Multisite network

Last Updated:

The robots.txt file is created dynamically in WordPress. If you want, you can upload a robots.txt file onto the root directory via FTP. You can edit robots.txt for individual sites in a WordPress Multisite network.

Editing robots.txt for individual sites in a WordPress Multisite network

If you do not upload a robots.txt file manually, the file will be created dynamically by WordPress.

The default robots.txt file will contain the following content:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://your-site.com/wp-sitemap.xml

Since WordPress version 5.5, XML sitemaps are included in the CMS' core code. The URL of the native WordPress sitemap is https://your-site.com/wp-sitemap.xml.

Depending on the WordPress sites, the Sitemap line will not be included in the dynamically created robots.txt.

If you activated an SEO plugin such as Yoast SEO or Rank Math, the Sitemap url will be changed accordingly.

Modify robots.txt with a WordPress hook (filter)

If no actual robots.txt file is present in the root folder, you can modify the robots.txt file with the following code snippet:

// Edit robots.txt for individual sites in a WordPress Multisite installation

function fbn_custom_robots( $output, $public ) {
	$site_id = get_current_blog_id();
	if ( $site_id == 1 ) {
		$output .= "Allow: /wp-content/uploads/\n";
		$output .= "\nSitemap: https://your-site.com/sitemap_index.xml";
	}
       elseif ( $site_id == 2 ) { 
       // Your code here
        }

	return $output;
}

add_filter( 'robots_txt', 'fbn_custom_robots', 20, 2 );

Please add the above code snippet to the functions.php file of your theme. Of course, you need to install a child theme to prevent your custom code from being deleted when the theme is updated.

Edit robots.txt using a SEO plugin

If you are using a premium version of Yoast SEO or Rank Math, it's possible to edit the robots.txt file within Dashboard.

For example, you can edit robots.txt by going to Rank Math > General Settings > Edit robots.txt when using Rank Math Pro.

How to edit robots.txt for individual sites in a WordPress Multisite network

However, a robots.txt file is present at the root directory of your site, it's not possible to edit it using Rank Math's Edit robots.txt section. You need to delete the actual robots.txt uploaded under the root folder.

Edit robots.txt using a plugin: Robots.txt Editor

Robots.txt Editor, a free WordPress plugin, allows you to create and edit the robots.txt file on your site. It works with multisite network on Subdomains.

How to fix robots.txt 404 Page Not Found

If the robots.txt file shows 404 error, please check the followings:

  • Deactivate all plugins;
  • Update WordPress, theme and plugins to the latest version;
  • Make sure that the permalink setting is not set to 'Plain';

See Also...