SEO

The Relationship Between Googlebot Crawl Rate, Impressions and Rankings

Marvin Magusara

Marvin has over 7 years of experience working with some of the largest global brands and digital agencies. Despite his vast experience, Marvin is still obsessed with learning and testing SEO theories to stay ahead of the curve.

Search engines use crawlers to find new and updated content. Content can range from web pages to PDF files and images. What all content has in common, however, is that it's discovered by links.

The most important crawler is Googlebot, Google's crawler, because more than 90% of searches are performed via Google. It works by visiting web pages on its index and then following the links within those pages to find new pages. Thus, Googlebot can discover new and updated content by following links, which is why backlinks are still a key ranking factor. 

Clicks and Rankings

When you search on Google, Google will check its index for relevant content and rank it to answer your query. In theory, the highest-ranked result will be the best and most relevant to your query. However, in practice, that's often not the case.

Our SEO data analyst set out to establish how Googlebot crawls affect page ranking and the dependency between crawl frequency and URL performance. Here is what they discovered. 

The Study Dataset

The analysis is based on a UK-based site's actual data with average monthly organic search impressions of 1.2M and 3.5k indexed pages.

For this analysis, the team combined data from Googlebot crawl events (GET requests from google.com/bot) and Google Search Console's daily URL performance for a period of 15 days in September 2020. 

For each URL, they calculated the number of crawls per day. Then, the frequency of crawls for each URL was calculated by dividing 15 (the total number of days) by the number of days on which there was at least one crawl.

Predictably, the more frequent the crawls, the higher the number of daily impressions and clicks were. The ranking of these pages was also higher. 

In this study, the number of URLs crawled every day was 81, which had 211 daily impressions and 3.59 clicks on average, and an average position of 15. Meanwhile, 1866 URLs were crawled once every two weeks, which had 2 daily impressions, 0.01 clicks, and an average position of 27.

Average Crawl Frequency (Once In)

Nr Of Urls

Avg Daily Impressions

Avg Daily Clicks

Avg Position

Everyday

81

211

3.59

15

Once in 2 days

540

25

0.42

19

Once in 3 days

162

11

0.08

22

Once in 4 days

343

4

0.02

27

Once in 5 days

677

3

0.02

28

Once in a week

1068

2

0.02

26

Once in 2 weeks

1866

2

0.01

27

frequency of crawls vs impression vs avg position

Introducing Pearson’s Coefficient of Linear Correlation

You'll agree that with age, children tend to get taller. A statistician would say there was a very strong positive relationship between age and height. The Pearson coefficient of correlation would range from 0.70 to 1, depending on the study sample. 

Likewise, the higher the speed of a car, the shorter the traveling time; if you drive faster, you'll arrive at your destination sooner. We then speak of a very strong negative relationship between speed and travel time. In this case, the coefficient would range from -0.70 to -1.

The team calculated the Pearson correlation coefficient between the number of days with crawls and the average daily impressions, a metric quantifying how many times a page is displayed on the search engine results pages (SERPs). The coefficient between the number of crawls, clicks, and position (ranking) was calculated too. 

Header

Nr Of Crawls

Avg impressions

0.69

Daily clicks

0.66

Position

-0.69

The coefficient between the number of crawls and average impressions was 0.69, which shows a strong positive relationship between them. It was 0.66 between the number of crawls and the clicks per day and -0.69 between the number of crawls and the position of the URL. This last negative value shows that high URL crawling frequency is associated with a low average position number. In this case, this means more often crawled URLs are closer to the top SERP position.

The team compared two types of URLs: high-ranking URLs with low impressions (average position 1-5 and average daily number of impressions <=5) and low-ranking URLs with high impressions (position >20 and daily number of impressions >=50). They found low-ranking URLs with more impressions were crawled almost twice as often: 8.7 vs. 4.6. This led them to conclude impressions were a more important factor for crawl frequency.

How are Crawl Frequency and Page Ranking Connected?

To determine how Googlebot affects page ranking in the days after a crawl, the analyst combined data from three sources: a list of crawls and their frequency for a period of two weeks in September 2020 and the avg daily position of each keyword and the corresponding URL. 

They noted the position of each keyword and URL; 

  • on the day before the last crawl
  • on the day of the crawl
  • and each day for five days after the crawl

They filtered out URLs that were crawled more frequently than once every three days because these could have been affected by previous crawls, potentially resulting in data inaccuracy. They also filtered out keywords that had no position data on the day before the crawl.

Position Change

Avg Dtd Change

Day Of Last Crawl

1

Day After Last Crawl

2

Day After Last Crawl

3

Day After Last Crawl

4

Day After Last Crawl

5

Day After Last Crawl

Avg DtD change

1.1

1.6

3.5

2.2

1.3

1.2

1.6

Avg % DtD change

3.71%

5%

6%

4%

3%

1%

3%

Avg change to the day before crawl

Cell

1.6

3.0

3.5

3.2

2.4

2.9

Avg %change to the day before crawl

Cell

5%

8%

11%

11%

8%

10%

The share of URLs with position improvement after the crawl was 64%. The position was higher than before the crawl on the third day after it. The share of URLs, whose position deteriorated after the crawl, was 36%.

High-ranking URLs were less volatile. For first, second, and third-ranked URLs, the average day-to-day change was just 0.13 compared to 2.08 for URLs ranking 21st or lower. 

Avg Dtd Change In Position Depending On Avg Position

Avg Dtd Change

Avg position 1-3

0.13

Avg position 4-10

0.15

Avg position 11-20

0.64

Avg position >21

2.06

What do These Findings Mean?

If your site isn't crawled and indexed, it won't appear on the SERPs. That is the only thing that is obvious. We have seen a correlation between crawl frequency and page impressions, as well as crawl and page ranking movements.

However, based on our understanding of how Google's algorithm works, the page ranking movement in relation to crawl is most probably due Google's re-calibration of the page's ranking after they've crawled it and re-calculated its 'score'. Since page impressions is directly tied to rankings, that is also directly impacted by that crawl.

Our advice is to check how many of your pages are indexed on your site and whether Google is crawling and locating all of the pages that need to be crawled or is it wasting the crawl budget on URLs that shouldn't be crawled. 

To check if all of the important pages have been indexed, you can check Google Search Console.

You can help to prioritize URL crawl rate by increasing the quantity and quality of internal and external links pointing to it. You can also ensure that you are not wasting crawl budget on low-priority pages by decreasing the internal links pointing to them, adding a nofollow tag to the links pointing to them, or by adding a block crawl instruction on robots.txt file for these URLs. 

These will point Googlebot in the right direction in terms of crawling your web content, which will increase your control over what ultimately appears in the index. 

Related Topics

Ultimate On-Page SEO Tool Showdown: POP vs. Surfer SEO vs. Cognitive SEO

6 Tips Multi-National/Location Businesses Can Effectively Use For Local SEO Efforts

Website Quality Audit Recovers site from Algorithm Penalty with 313% Organic Traffic Increase – SEO Case Study

How to Optimize Content for Users and Google?

Leave a Comment

Your email address will not be published. Required fields are marked *

Let's Talk

We only work with businesses we can actually help through SEO. Contact us for a Free Website Analysis.

To The Top Logo

45 Braidley Road, Bournemouth, BH2 6JY, UK

4, Commerce and Industry Plaza, Mckinley Town Center, Taguig, 1634, Manila

© 2019 ToTheTop.