If you are wondering what is WordPress robots.txt file, you have come to the right place.
In this guide, I will explain what is a robots.txt file, where it is, how to create a robots.txt file for your WordPress site.
Let’s start!
What is the robots.txt file?
Robots.txt file is a text file to instruct robots on how to crawl a website. In general, it is created by the webmaster to tell search engine bots which part to crawl & index and which part to avoid.
Why use robots.txt?
A Robots.txt file tells crawlers which area of your website crawler can or can not access. It has a strong influence on SEO but it is very dangerous if you accidentally disallow search engines to crawl your entire site.
Some common uses of robots.txt are:
- Controlling crawler access to a certain area of the website like ‘wp-admin’.
- Preventing duplicate content from indexing in search engines.
- Specifying a crawl delay.
- Specifying the sitemap location.
- Restrict search engines to index certain files on your website (PDFs, images, etc.)
- Restrict search engines to index your internal search result pages.
Where is my robots.txt file and how to access it?
WordPress creates a virtual robots.txt file if there is no physical robots.txt file in the site root folder. So it is possible you might see robots.txt file at yoursite.com/robots.txt but no physical file in your site’s root folder.
The robots.txt file resides in the site’s document root folder (top level of directory of your webserver). The document root is a folder where websites files for a hosted domain name are stored.
You can access this file through an FTP client like FileZilla. If you don’t know how to access it with FTP client, you can easily access this from your WordPress admin with the Yoast SEO plugin.
WordPress Admin > Yoast SEO > Tools > File Editor
How to create robots.txt file?
If you don’t have robots.txt file on your root folder of the website, you can create it using a plain text editor of your choice.
Here are simple steps to create and uploaded robots.txt file via FTP:
- Open a plain text editor of your choice.
- Add syntax.
- Save the text file as robots.txt
- Upload the file to your server.
Here are steps to create or edit robots.txt file in WordPress Dashboard using the Yoast SEO plugin:
- Log in to your WordPress site.
- Click on ‘SEO’. You can find this on the left-hand side in the menu section.
- Click on ‘Tools’.
- Click on ‘File Editor’
- Make the necessary changes to your file.
- Save it.
What to hide with robots.txt file?
The most common use of robots.txt file is to exclude certain directories or pages from search engines.
Here are some pages/folders you should disallow using your robots.txt file:
- Admin pages/folder
- Thank you pages
- Account pages (SaaS and eCommerce sites)
- Shopping cart (eCommerce sites)
- Dynamic products (eCommerce sites)
- Paginations/sorting pages
- Duplicate content pages
Blocking Robots and Search Engines from Crawling
For a reason, if you want to stop bots and search engines to crawl your site, use this code:
#Blocks robots and search engines User-agent: * Disallow: /
Please note that you will lose all your search engine ranking if you block search engines.
You can block robots and search engines to crawl a certain part or folder of your website. Bots will not crawl any object/file/URL in that folder. Check the code:
#Blocks robots and search engines User-agent: * Disallow: /wp-admin/
The above code allows robots to crawl the WordPress site except for files from the ‘wp-admin’ folder.
Adding Your XML Sitemap to Robots.txt File
If you are creating an XML sitemap for your WordPress site you can add it to your robots.txt file manually like this:
Sitemap: http://www.example.com/sitemap.xml
If you are using YOAST SEO Plugin or similar WordPress plugin to generate XML sitemap, then your plugin will automatically add (in most cases) your sitemap into your robots.txt file.
We recommend you to add your sitemap manually to Google and Bing Webmasters Tools.
