Crawling December: HTTP caching
Allow us to cache, pretty please. As the internet grew over the years, so did how much Google crawls. While Google’s crawling infrastructure supports heuristic caching mechanisms, in fact always had, the number of requests that can be returned from local caches has decreased: ...