Strategic Marketing House has published a detailed guide about the Google-Inspection Tool bots, the new crawlers Google uses in their testing tools starting May 17, 2023 and made the Guide available to the public.
SMH’s Vice President of Digital Marketing Strategy, Carolyn Holzman, and a 14-year veteran in SEO put this guide together based on server log data. Holzman is in a unique position to provide this information about Google’s new crawlers given her on-going indexation research project that she initiated in August of 2021.
The guide goes beyond the current overview available in the Google Developer Documentation on their crawler agents. Based on the data from the test sites in the research, Holzman ascertained through server log analysis, there are four (4) distinctive crawlers responding to indexation requests, Google’s documentation lists two (2) out of the four.
The guide expands on current documentation and highlights the individual crawler agents as they appeared and what specific types of crawls (html and javascript rendering crawls) they perform.
“The beauty of this work is that the server logs revealed everything. And to see them in action meant I didn’t have to go outside of the parameters of my research. I was counting and demarking individual crawling agents anyway and noting the appearance of new crawler agents – I’ve personally cataloged 39 mobile Googlebot and 38 desktop Googlebot chrome crawlers since late 2021 before these new crawlers were launched”, said Holzman.
“The challenge when these new crawlers appeared was how to notate them in the existing data log. They only arrive when indexation is requested via search console or a URL inspection. The only other times these same crawlers are sent which I have verified are requests through the rich snippet tool and the mobile friendly test,” she continued.
The current data log is visible on American Way Media’s Google Index Detector page.
The Google inspectionTool crawlers are noted in the guide and will be a handy reference document for those involved in publishing new and updated content on their sites.
She is currently writing a detailed description of the chrome updates of the crawlers in the run up to the GA4 July 1st deadline when Google Universal Analytics was sun-setted and everyone’s site that are set up in Google Analytics began running on the Google analytics 4.
“That kind of paper is for the super geeks who actually would be fascinated to see the pattern that led up to the GA4 change over on the 31st and how all that lined up with a very high volatility in the SERPS. I’ll definitely make it publicly available.”
Holzman also reports daily findings from her indexation research on the YouTube channel @CrawlorNoCrawl. She is the host of the weekly podcast, Confessions of an SEO, which focuses on the dynamic between SEO and the businesses that they serve. It is now in its 3rd season with new episodes launched on Fridays at 8 AM CST.
Strategic Marketing House is a digital marketing agency that offers a guaranteed ROI program for home services. Other services include marketing communication software and a subscription-based seo service. For more information about this guide or services, call 402-512-4350 or email markethappy (at) strategicmarketinghouse dot com.
Location Info:
Strategic Marketing House
206 Longwood Drive, Papillion, NE 68133-3386
+14025124350