Preventing Search Engine Indexing

Large numbers of web sites have portions of their site configured to utilize SSL. This permits the transfer of information among the server and the browser to get place over an encrypted connection. URLs of this type of pages start with https rather than http to point to the secure protocol.

You may practice grave canonical difficulties if the secure portions of your site have been fully indexed beside with your standard site.

search engines from indexing

These difficulties happen only if you have the secure seo pages inside the similar sub domain as your standard pages. But, if you have a secure sub domain, then that part of your site can be barred from indexing utilizing the robots.txt file in the root folder for the same sub domain. And if there is a single page within a site require the use of SSL. It may be viewed as a simpler alternative in this type of cases to have the secure page inside your standard site structure. Just the protocol is required to be altered in this case, not the sub domain or directory.

Though, this technique can result in a search engine optimization indexing the secure page, with following links from the page. For example if relative links, i.e. to index.html, these would be interpreted as links to secure versions of your standard pages.

Google, and other SEO Company , could observe this as duplicated content, therefore reducing your page ranking in their search results. Once indexed, Google will continue to visit these pages, unless barred by a robots.txt file or special Meta tags in the head of each file.

Prevent Indexing

If you ever search SEO yourself in this position, it may seem like there is no simple way to get out. There is a method to redirect secure requests for the robots.txt file to a secondary file that will eliminate web crawling programs from your secure pages. In turn for this solution to work you should be via an Apache web server with mod_rewrite enabled.

Firstly, you should make a second robots.txt, calling it robots_ssl.txt, making certain it blocks every one spider. You upload this file to the root level of your domain.