What to Do If You Accidentally clicked “Discourage search engines from indexing this site.”
It’s easy to do. Maybe you were moving from a development area to a live environment? Maybe you hired a WordPress “expert” who had no idea what he was doing? Maybe you just forgot to uncheck the box after your site was all set up? In any case, if you find yourself in a position where Google has not indexed your site or has started to de-index your site because of this WordPress setting, then this is the post for you. Read on!
When this “Search Engine Visibility” box is checked, WordPress essentially tells Google and other search engines NOT to crawl your site. If Google users do happen to find your site, Google will not provide them a meta description. Instead, they will see a message that reads, “A description for this result is not available because of this site’s robots.txt.” Eventually though, Google will likely begin to de-index your website’s pages. This will almost always result in a decline of web traffic, which makes sense. If Google is blocked from crawling your pages, it cannot be confident that your page offers a good response to a searcher’s keyword search. Therefore, they will start to de-index your pages and not send you any more traffic.
Okay but how do I fix it?
The first thing to check is to navigate to Settings > Reading in your wp-admin. At the bottom of this page, you will see a box labeled “Search Engine Visibility.” You will need to uncheck this box so and then click “Save Changes.”
Unfortunately, Google will not automatically or instantly re-add all of your pages. However, there are several things that you can that usually help sites get indexed:
- Publish new content on your site – New content gives Google something new to add to it’s index on your site and as they crawl this page, they sometimes will crawl through to other pages and re-add them to their index also. By default, WordPress automatically notifies a few update services that you have new content. These Update Services often publish links to blog content from thousands and thousands of blogs and Google crawls them frequently which increases your changes of Google crawling your new post and possibly other pages on your site along with it. (Read more about WordPress’s Update Services here.) You can edit these by going to Settings > Writing and again scroll down to the bottom where you will see this. Here is where you can add more services.
- Submit your site and/or individual pages on the site to blog search engines, bookmarking sites, and other such sites. Doing so should help your pages get “found” again by Google. This works great even if you don’t have new content to add to your site right away. You can manually go ping-o-matic and submit your site. Also, while not as popular as they used to be, bookmarking services such as Delicious allow users to post and share links on the web. This gives you the opportunity to link to and share your posts and pages in the hopes that Google will find them and revisit your site.
- Link to your pages from other websites. If you have friends with sites and/or any connections to industry blogs, online magazines, or any other site, offering to write an article for them that links back to your site, particularly to pages that have been de-indexed, can help Google find them again.
Unfortunately, getting re-indexed is not an automatic process, but taking some of the steps in this brief guide should significantly improve your site’s changes of being re-indexed by Google. Please note that depending on your site, your competition’s sites, and several other factors, it could take weeks for Google to re-index your entire site and it could take even longer for you to regain any rankings you once had.
Of course, if you would like help speeding up the process, please get in touch with us about getting help.
