.Google's John Mueller responded to a concern regarding why Google marks pages that are actually disallowed coming from crawling through robots.txt and also why the it's secure to ignore the similar Browse Console files regarding those creeps.Robot Traffic To Question Criterion URLs.The individual talking to the concern recorded that robots were actually making web links to non-existent question parameter URLs (? q= xyz) to pages along with noindex meta tags that are actually likewise blocked in robots.txt. What cued the concern is that Google is creeping the web links to those webpages, obtaining shut out by robots.txt (without noticing a noindex robotics meta tag) then getting reported in Google.com Browse Console as "Indexed, though blocked through robots.txt.".The individual asked the complying with inquiry:." However right here's the large question: why would certainly Google.com index webpages when they can't also find the web content? What's the advantage during that?".Google.com's John Mueller confirmed that if they can not creep the web page they can't view the noindex meta tag. He additionally helps make an exciting mention of the site: search operator, advising to neglect the end results since the "ordinary" consumers will not find those outcomes.He wrote:." Yes, you are actually appropriate: if our company can't crawl the page, we can't see the noindex. That said, if our company can't creep the web pages, at that point there is actually certainly not a whole lot for our team to mark. So while you might observe a few of those webpages with a targeted website:- query, the typical consumer will not observe them, so I wouldn't bother it. Noindex is actually also alright (without robots.txt disallow), it simply implies the Links will certainly find yourself being actually crawled (as well as end up in the Browse Console document for crawled/not recorded-- neither of these standings induce concerns to the rest of the web site). The important part is actually that you don't make all of them crawlable + indexable.".Takeaways:.1. Mueller's response validates the limits in operation the Site: hunt evolved search operator for diagnostic reasons. Some of those reasons is actually because it is actually not linked to the regular search index, it is actually a separate factor completely.Google's John Mueller discussed the internet site search operator in 2021:." The quick solution is that a site: question is actually not suggested to become comprehensive, nor utilized for diagnostics objectives.An internet site concern is actually a specific sort of search that confines the end results to a particular internet site. It is actually basically just the word web site, a bowel, and afterwards the website's domain.This concern confines the end results to a specific website. It is actually not meant to be a comprehensive assortment of all the webpages from that web site.".2. Noindex tag without using a robots.txt is great for these type of scenarios where a robot is linking to non-existent pages that are actually acquiring found through Googlebot.3. URLs with the noindex tag will create a "crawled/not catalogued" entry in Search Console which those will not have an adverse impact on the rest of the site.Read through the concern as well as address on LinkedIn:.Why would certainly Google index webpages when they can not also find the information?Included Photo through Shutterstock/Krakenimages. com.