Google’s Gary Illyes mentioned on the final Search Off the File podcast that Google can crawl sure sections of your website extra continuously and likewise infer the standard of sure sections of your website in another way.

This got here up on the 9:09 minute mark into the podcast however Glenn Gabe summarized it tremendous effectively on Twitter. Glenn mentioned “Google can infer from a website total which areas they could must crawl extra continuously. E.g. if there is a weblog subdirectory & there are indicators that it is widespread/essential, then Google would possibly wish to crawl there extra.” ” And it isn’t simply replace frequency, it is also about high quality. E.g. if G sees a sure sample is widespread (folder), & persons are speaking about it & linking to it, then it is a sign that ppl like that listing,” he added.

Right here is the video embed:

Right here is the transcript of this part on crawl frequency by part of the location:

Yeah. As a result of like we mentioned, we do not have infinite area, so we wish to index stuff that we think– effectively, not we– however our algorithms decide that it may be looked for in some unspecified time in the future, and if we do not have indicators, for instance, but, a couple of sure website or a sure URL or no matter, then how would we all know that we have to crawl that for indexing?

And a few issues you possibly can infer from– for instance, when you launch a brand new weblog in your most important website, for instance, and you’ve got a brand new /weblog subdirectory, for instance, then we are able to form of infer, primarily based on the entire website, whether or not we wish to crawl quite a bit from that /weblog or not.

Then right here is the part on high quality:

Nevertheless it’s not simply replace frequency. It is also the standard indicators that the primary website has.

So, for instance, if we see {that a} sure sample could be very widespread on the web, like a slash product could be very widespread on the web, and folks on Reddit are speaking about it, different websites are linking to URLs in that sample, then it is a sign for us that folks like the location usually.

Whereas if in case you have one thing that persons are not linking to, after which you are attempting to launch a brand new listing, it is like, effectively, individuals do not like the location, then why would we crawl this new listing that you simply simply launched?

Discussion board dialogue at Twitter.