Most site and blog owners are willing to do a lot to have their sites and blogs indexed and appear in search results, and this is normal, as search engines are considered the largest and most effective source of traffic. However, you may sometimes want to prevent search engines from indexing your site or blog or de-index it if it… It is already indexed in order to make some modifications in the background that are not visible to users.
In this article, we will shed light on the reasons that might prompt you to prevent or de-index your site or blog, and then we will show you how to do that easily using methods that you can apply, some of which can be applied to your WordPress site.
Why do you need to prevent search engines from indexing your blog?
There are many reasons that may prompt you to cancel the indexing of your site, but because in most cases the site administrator can know when his site or blog should not appear in the search results, we will mention here only the most important of these reasons:
- Site under development : When your site is not fully built yet or is under maintenance, there may be some errors in it, so it is better not to let search engines show it in the results for potential visitors.
- The blog is personal or family : If you want to create a blog where only family members interact with each other, or if you want to create a private blog for personal needs, you will prefer that the blog not appear to strangers, and you will need to prevent search engines from showing it in search results.
- Project management websites: WordPress or another website management tool can be used as a business project management platform or as an internal data management system . Project owners certainly do not want their business documents to appear to the public, so they must prevent search engines from indexing their websites.
- Test sites : If you are using a copy of your site to test the additions, themes, and modifications you make before applying them to the main site, you will need to prevent search engines from indexing the copy of the site you are using for testing
There are also reasons why you might want to prevent only some of your site’s pages from being indexed in search results, and not all of its pages, and we will highlight how to do this as well in a separate paragraph after explaining how to de-index the entire blog or site.
Ways to de-index the site in search results
There are many methods that allow you to prevent search engines from indexing your site and showing it to potential visitors in search results. Some methods are easier and better to apply than others, and some of them have different features than others. Therefore, we will explain to you here the best and most important of these methods, and you can choose the method that you prefer. You see that it is more suitable for you yourself.
De-index the site from the WordPress panel
The WordPress system includes a custom option that allows you to cancel and prevent search engines from indexing your site or blog, so you select this option to carry out the task, but this method does not force search engines not to index, but rather asks them not to do so, and the decision to index the site is made by not It eventually gets indexed back to the search engine’s crawler, and it will either honor your request not to index your site, or ignore it and index it anyway.
Although it is not possible to deindex your site or blog from all search engines using this method, it works with the most popular search engines such as Google and Bing, as these engines usually respect your request and do not index the site when you ask them to do so.
You can de-index your site or blog from the WordPress control panel by going to your site’s control panel, then going to the reading settings by clicking on the word ( Reading ) from the drop-down menu that appears when you place the mouse cursor on the ( Settings ) option located in the side menu.
You will see a page containing many reading settings, and at the bottom you will find the option ( Prevent search engines from archiving this site ), under the ( Visibility to search engines ) section.
If you want to ask search engines not to index your site, put a check mark (√) in the box next to the option, then click the ( Save changes ) button, as the following image shows:
After you activate this option in WordPress, the search engines that will honor your request will not index your site’s pages if they have not yet indexed them, but if your site’s pages were indexed before that, it will take some time to remove them from search results.
Also, if you disable Prevent Indexing in the Future to allow search engines to index your site, it will take some time for it to appear in search results again.
This method will not always be appropriate for you to use to remove your site pages from search results, because using it does not mean that your site will not appear in search results 100%. If your site is a blog that includes personal matters, or contains sensitive information, you do not want to risk someone seeing it. What is this content.
So if it’s not very important to keep your site hidden from everyone, such as when it’s a test site that doesn’t contain sensitive information, you can use this method, but in cases where you’re likely to have problems with someone viewing your site’s content, you should use a better method to hide it. Your site for search engines.
It is worth noting that when you activate the option (Prevent search engines from archiving this site), WordPress modifies the robots.txt file of your site or blog and asks the crawlers using those modifications not to index the site, and it also inserts the following code in the header of the website pages ( The head element may also delete specific codes from your site pages:
Prevent indexing of the site by protecting it with a password
The previous method usually hides your site’s pages from search results and does the job perfectly, but in accordance with Google’s advice which says that the robots.txt file should not be used to hide the site, and based on the drawbacks of this method that we mentioned, sometimes a better method should be used. To de-index the site.
Although there are many other ways to hide a website or blog from search engines, the best way you can be completely sure that your site will not be indexed by any crawler is by protecting the server that includes its files with a username and password. In this case, no person or program will be able to access your site without providing the correct login data that you specify.
You can protect the server (hosting) with a password by going to the cPanel web hosting control panel for your hosting, then clicking on the ( Directory Privacy ) tool, which you will often find in the ( Files ) tools section.
Note: You will find the Directory Privacy tool called Password Protect Directories in some versions of cPanel.
After clicking on the tool, a page will appear that enables you to protect any folder on your hosting with a username and password. To protect your site with a password, you must click on the word ( Edit ) next to the folder that contains the site, which is usually ( public_html ) or another folder within it if you have more than one WordPress site installed on the hosting within this folder.
You will now see a page asking you to enter the name of the protected folder with which the folder will be dealt with after it is protected, in addition to a checkbox that allows you to use a password to protect the folder with. You have to put a check mark in the check box, enter a name for the folder, and then click the ( Save ) button.
You will now be taken to a data entry page that will allow the person or crawler to access the site upon submission. Enter the user name and password, and re-enter the password in the fields designated for that, then click the ( Save ) button.
Thus, you have prevented search engines from indexing your site by protecting it with a password through the use of the web hosting control panel. If someone has a link to your site and enters it in the browser bar to go to the site, he will be asked to enter the login data. If he enters it correctly, it will allow He can browse the site, and if he cannot enter it correctly, the browser will not display the site content to him.
Prevent search engines from indexing a specific page on the site
Most of the time, you need to de-index or prevent search engines from indexing a specific page or pages of your site instead of the entire site, and there are many ways that enable you to do this, but the easiest and best way that does not require installing any additional components on the WordPress site is By protecting the article or page with a password through the component editor or the classic editor, WordPress includes an option where you can protect pages and articles that you do not want everyone to see with a password.
You can lock an article or page on your WordPress site with a password to prevent it from being indexed by search engines by entering the article or page editing page, then choosing the ( password protected ) option from the drop-down list for the ( appearance ) option, then entering the password you want to protect the article. Or the page in it, as the following image shows.
Notice that the word ( general ) that appeared next to the word ( appearance ) disappears and the phrase ( protected with password ) appears in its place after you enter a password in the designated field.
After entering the password, and the phrase ( password protected ) appears next to ( appears ), you must click on the ( Update ) button to apply the changes.
Thus, you have locked the article and prevented it from being archived by search engines. If anyone tries to visit it, he will be asked to enter the password. If he is able to enter it correctly, the content of the article or page will appear to him.
It is worth noting that there are other methods that enable you to prevent or cancel the indexing of one of the pages of your website or blog by search engines without the need to lock it with a password, but this method is the best and most secure, so always use it if the page information is sensitive.
If the page does not include sensitive information, you can use WordPress plugins such as the Yoast SEO plugin , which provides the ability to disallow the article from being indexed from the advanced SEO settings for the article, which you will find below the editing area on the article editing page.
This concludes our explanation of how to prevent and cancel indexing of a website or blog and not allow its pages to appear in search results. We would like to point out and emphasize once again the necessity of applying methods that include password protection if you want to prevent indexing of sensitive content, in order to prevent crawlers. Allow search engines and people alike to access that content to prevent it from being completely leaked to anyone you do not want to give access to.
