Google says URL case is also important for Robots.txt


We are known as Google and can handle the same URL but with different cases. Thus, vs can be seen by Google as different URLs. But Google seems to be stricter on this rule when it comes to the robots.txt file.

With vs, Google can tell if the pages are the same and will canonize the URL of one of them, so both will not appear in search and Google will be able to also consolidate signals.

But maybe if you have a robots.txt directive for but not for, maybe Google won’t use the Robots directive?

John Mueller of Google posted a video on URL Case Sensitivity and Mark 1:08 said “Another place where the exact URL plays a role is robots.txt. In the robots.txt file you can indicate which parts of a website should not be crawled. The robots.txt file also uses exact URLs. So if you have entries that refer to a version of a URL, they would not apply to other versions of this URL. ”John added that it is“ rare for this to cause problems ”.

So keep this in mind not only for how Google can canonize your URLs, but also for the handling of robots.txt.

Discussion forum at WebmasterWorld and Twitter.

Leave A Reply

Your email address will not be published.