.Google.com's John Mueller answered a concern regarding why Google marks pages that are prohibited coming from creeping through robots.txt and why the it is actually risk-free to dismiss the similar Explore Console reports about those creeps.Crawler Traffic To Query Parameter URLs.The person talking to the inquiry chronicled that crawlers were actually producing hyperlinks to non-existent question specification Links (? q= xyz) to pages with noindex meta tags that are likewise blocked out in robots.txt. What urged the question is actually that Google is actually crawling the hyperlinks to those webpages, obtaining obstructed by robots.txt (without envisioning a noindex robotics meta tag) at that point receiving shown up in Google Search Console as "Indexed, though shut out by robots.txt.".The individual talked to the following inquiry:." But listed here's the big concern: why would certainly Google index web pages when they can't also view the information? What is actually the conveniences because?".Google.com's John Mueller affirmed that if they can't crawl the page they can not observe the noindex meta tag. He likewise creates an appealing acknowledgment of the web site: search driver, encouraging to neglect the end results considering that the "common" consumers will not observe those results.He created:." Yes, you are actually proper: if our experts can not crawl the page, our experts can't find the noindex. That pointed out, if our experts can not crawl the web pages, after that there's certainly not a great deal for our team to index. So while you might view several of those pages with a targeted website:- concern, the ordinary customer will not find them, so I wouldn't fuss over it. Noindex is likewise alright (without robots.txt disallow), it only suggests the Links will definitely wind up being crept (as well as end up in the Explore Console report for crawled/not indexed-- neither of these standings lead to concerns to the remainder of the website). The vital part is that you do not make them crawlable + indexable.".Takeaways:.1. Mueller's response validates the restrictions in operation the Web site: search advanced hunt operator for diagnostic explanations. Some of those factors is since it's not attached to the normal search mark, it's a different thing completely.Google.com's John Mueller talked about the internet site hunt operator in 2021:." The quick response is that a website: inquiry is actually not implied to be complete, neither used for diagnostics functions.A web site inquiry is actually a particular kind of hunt that confines the end results to a certain internet site. It's basically only words web site, a bowel, and then the website's domain.This concern restricts the end results to a certain website. It is actually certainly not meant to become a thorough collection of all the pages from that site.".2. Noindex tag without utilizing a robots.txt is great for these sort of conditions where a robot is linking to non-existent web pages that are acquiring found out through Googlebot.3. URLs along with the noindex tag are going to generate a "crawled/not listed" item in Search Console and that those will not possess an adverse effect on the rest of the internet site.Review the inquiry and also respond to on LinkedIn:.Why would certainly Google.com mark pages when they can't even find the information?Featured Graphic by Shutterstock/Krakenimages. com.