How to Upload a Txt File to Wordpress Root
Recently one of our readers asked us for tips on how to optimize the robots.txt file to improve SEO.
Robots.txt file tells search engines how to crawl your website which makes it an incredibly powerful SEO tool.
In this commodity, nosotros will testify you lot how to create a perfect robots.txt file for SEO.
What is robots.txt file?
Robots.txt is a text file that website owners can create to tell search engine bots how to clamber and index pages on their site.
It is typically stored in the root directory, also known as the primary folder, of your website. The basic format for a robots.txt file looks like this:
User-agent: [user-agent name] Disallow: [URL string non to exist crawled] User-agent: [user-agent proper name] Permit: [URL string to be crawled] Sitemap: [URL of your XML Sitemap] You can accept multiple lines of instructions to let or disallow specific URLs and add multiple sitemaps. If you do not disallow a URL, and so search engine bots assume that they are allowed to clamber information technology.
Here is what a robots.txt example file can look like:
User-Agent: * Permit: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Sitemap: https://example.com/sitemap_index.xml In the above robots.txt case, we have allowed search engines to crawl and index files in our WordPress uploads folder.
After that, we accept disallowed search bots from crawling and indexing plugins and WordPress admin folders.
Lastly, nosotros have provided the URL of our XML sitemap.
Do You Need a Robots.txt File for Your WordPress Site?
If you don't have a robots.txt file, then search engines volition still crawl and index your website. However, you lot will not exist able to tell search engines which pages or folders they should not clamber.
This will non have much of an bear on when you lot're kickoff starting a blog and do non have a lot of content.
However as your website grows and you have a lot of content, then you would likely want to have better control over how your website is crawled and indexed.
Hither is why.
Search bots accept a crawl quota for each website.
This means that they crawl a certain number of pages during a clamber session. If they don't finish crawling all pages on your site, and then they will come back and resume crawl in the adjacent session.
This can tiresome down your website indexing rate.
You can fix this past disallowing search bots from attempting to clamber unnecessary pages like your WordPress admin pages, plugin files, and themes folder.
By disallowing unnecessary pages, you save your crawl quota. This helps search engines clamber even more pages on your site and index them as quickly as possible.
Another good reason to use robots.txt file is when y'all want to end search engines from indexing a mail or page on your website.
It is non the safest way to hide content from the general public, but it will help yous prevent them from actualization in search results.
What Does an Platonic Robots.txt File Look Like?
Many popular blogs apply a very elementary robots.txt file. Their content may vary, depending on the needs of the specific site:
User-amanuensis: * Disallow: Sitemap: http://www.instance.com/post-sitemap.xml Sitemap: http://www.case.com/page-sitemap.xml This robots.txt file allows all bots to index all content and provides them a link to the website'due south XML sitemaps.
For WordPress sites, we recommend the following rules in the robots.txt file:
User-Amanuensis: * Let: /wp-content/uploads/ Disallow: /wp-admin/ Disallow: /readme.html Disallow: /refer/ Sitemap: http://www.instance.com/mail-sitemap.xml Sitemap: http://www.example.com/page-sitemap.xml This tell search bots to alphabetize all WordPress images and files. It disallows search bots from indexing WordPress admin area, readme file, and cloaked affiliate links.
By adding sitemaps to robots.txt file, you brand it easy for Google bots to find all the pages on your site.
Now that y'all know what an ideal robots.txt file look similar, let's have a look at how you can create a robots.txt file in WordPress.
How to Create a Robots.txt File in WordPress?
There are two means to create a robots.txt file in WordPress. You can cull the method that works best for yous.
Method 1: Editing Robots.txt File Using All in One SEO
All in One SEO also known as AIOSEO is the best WordPress SEO plugin in the market used by over 2 million websites.
It's piece of cake to employ and comes with a robots.txt file generator.
If yous don't accept already accept the AIOSEO plugin installed, y'all tin come across our step by step guide on how to install a WordPress plugin.
Note: Gratis version of AIOSEO is also bachelor and has this feature.
Once the plugin is installed and activated, you tin can use information technology to create and edit your robots.txt file direct from your WordPress admin area.
Simply become to All in One SEO » Tools to edit your robots.txt file.
First, you'll need to turn on the editing pick, past clicking the 'Enable Custom Robots.txt' toggle to blueish.
With this toggle on, yous tin create a custom robots.txt file in WordPress.
All in 1 SEO will testify your existing robots.txt file in the 'Robots.txt Preview' section at the lesser of your screen.
This version will testify the default rules that were added by WordPress.
These default rules tell the search engines not to crawl your core WordPress files, allows the bots to index all content, and provides them a link to your site's XML sitemaps.
Now, you can add your own custom rules to amend your robots.txt for SEO.
To add a rule, enter a user agent in the 'User Agent' field. Using a * will apply the dominion to all user agents.
Then, select whether you want to 'Allow' or 'Disallow' the search engines to crawl.
Side by side, enter filename or directory path in the 'Directory Path' field.
The rule will automatically exist applied to your robots.txt. To add together some other rule clicks the 'Add Rule' push.
We recommend adding rules until yous create the ideal robots.txt format we shared above.
Your custom rules will expect like this.
Once you're done, don't forget to click on the 'Salvage Changes' button to shop your changes.
Method 2. Edit Robots.txt file Manually Using FTP
For this method, you volition demand to use an FTP client to edit robots.txt file.
Simply connect to your WordPress hosting account using an FTP client.
Once inside, you will be able to encounter the robots.txt file in your website's root folder.
If you don't see one, then you likely don't have a robots.txt file.
In that instance, you lot tin just get ahead and create one.
Robots.txt is a plain text file, which means you can download it to your reckoner and edit it using any plain text editor like Notepad or TextEdit.
After saving your changes, you tin upload it back to your website'south root folder.
How to Exam Your Robots.txt File?
Once y'all have created your robots.txt file, it's always a skillful idea to test information technology using a robots.txt tester tool.
At that place are many robots.txt tester tools out there, merely we recommend using the one inside Google Search Console.
First, you'll demand to have your website linked with Google Search Console. If you lot haven't done this notwithstanding, run across our guide on how to add together your WordPress site to Google Search Console.
So, you can employ the Google Search Console Robots Testing Tool.
Simply select your property from the dropdown list.
The tool will automatically fetch your website'south robots.txt file and highlight the errors and warnings if it found any.
Final Thoughts
The goal of optimizing your robots.txt file is to preclude search engines from crawling pages that are not publicly available. For example, pages in your wp-plugins folder or pages in your WordPress admin folder.
A mutual myth among SEO experts is that blocking WordPress category, tags, and annal pages will improve clamber rate and result in faster indexing and higher rankings.
This is not truthful. It's also against Google's webmaster guidelines.
We recommend that y'all follow the to a higher place robots.txt format to create a robots.txt file for your website.
Nosotros hope this commodity helped you learn how to optimize your WordPress robots.txt file for SEO. You may also want to see our ultimate WordPress SEO guide and the best WordPress SEO tools to abound your website.
If you liked this commodity, then please subscribe to our YouTube Aqueduct for WordPress video tutorials. You can also find us on Twitter and Facebook.
Disclosure: Our content is reader-supported. This means if you click on some of our links, then we may earn a committee. See how WPBeginner is funded, why it matters, and how you can support usa.
desrochershateref.blogspot.com
Source: https://www.wpbeginner.com/wp-tutorials/how-to-optimize-your-wordpress-robots-txt-for-seo/
0 Response to "How to Upload a Txt File to Wordpress Root"
Post a Comment