Ever wondered what would happen if you prevented Google from crawling your website for a few weeks? Technical SEO expert Kristina Azarenko has published the results of such an experiment.
Six surprising things that happened. What happened when Googlebot couldn’t crawl Azarenko’s site from Oct 5 to Nov. 7:
- Favicon was removed from Google Search results.
- Video search results took a big hit and still haven’t recovered post-experiment.
- Positions remained relatively stable, except were slightly more volatile in Canada.
- Traffic only saw only a slight decrease.
- An increase in reported indexed pages in Google Search Console. Why? Pages with noindex meta robots tags ended up being indexed because Google couldn’t crawl the site to see those tags.
- Multiple alerts in GSC (e.g., “Indexed, though blocked by robots.txt”, “Blocked by robots.txt”).
Why we care. Testing is a crucial element of SEO. All changes (intentional or unintentional) can impact your rankings and traffic and bottom line, so it’s good to understand how Google could possibly react. Also, most companies aren’t able to attempt this sort of an experiment, so this is good information to know.
The experiment. You can read all about it in Unexpected Results of My Google Crawling Experiment.
Another similar experiment. Patrick Stox of Ahrefs has also shared results of blocking two high-ranking pages with robots.txt for five months. The impact on ranking was minimal, but the pages lost all their featured snippets.
New on Search Engine Land
About the author
Danny Goodwin has been Managing Editor of Search Engine Land & Search Marketing Expo – SMX since 2022. He joined Search Engine Land in 2022 as Senior Editor. In addition to reporting on the latest search marketing news, he manages Search Engine Land’s SME (Subject Matter Expert) program. He also helps program U.S. SMX events.
Goodwin has been editing and writing about the latest developments and trends in search and digital marketing since 2007. He previously was Executive Editor of Search Engine Journal (from 2017 to 2022), managing editor of Momentology (from 2014-2016) and editor of Search Engine Watch (from 2007 to 2014). He has spoken at many major search conferences and virtual events, and has been sourced for his expertise by a wide range of publications and podcasts.
