With four algorithm updates under Google’s belt this year and a whole host of theories on search engine optimization (SEO) and AI search to prove, we’ve seen an influx in the number of SEO and AI studies published. Sometimes, however, the results are confidently reported as fact when their significance is actually quite low.
With that in mind, here are some things to look for when scrutinizing an SEO or AI visibility study for accuracy and significance.
Any reputable SEO study must explain its methodology and how it used the data to arrive at its conclusions. The intention and an explanation behind the study should be clearly explained, along with specific data sources and sample data sets. If the information is vague or does not account for any potential variables like seasonality or industry trends, it shouldn’t be taken at face value.
For example, a study publishing before-and-after comparisons of data analyzing a recent Google Core Update shouldn’t just have simple traffic changes with no further explanation. If the methodology contains a variety of industry types with identified variables, it lends significantly more credibility to the study’s findings.
In a perfect world, first-party data like Google Search Console and Google Analytics 4 would be the go-to data source. Obviously, in large-scale studies, this is almost impossible based on the lack of access to those data sets. As a result, major studies often rely on third-party data from SEO and AI tracking platforms like Semrush, Ahrefs, Profound, or others. Although not ideal, the large-scale data offered by these trackers makes measurement between sites and industries possible. And if they’re not as accurate as first-party data, at least all sites are measured similarly in these tools.
A valid study will source these first-party and premium third-party tools for performance correlations. Beware studies that rely on free tools or vaguely explained proprietary tools.
When metrics are vague or even unlisted, that’s an immediate sign that the study is likely disreputable. For example, if a study suggests that certain content types on certain sites saw a 50% lift after an algorithm update, but if it does not explain what metric increased specifically — whether traffic, rankings, revenue, AI Overview ownership, etc. — and during what time frame, then it’s not a study that can be relied upon.
At the same time, vague data sets like an unclear number of websites and industries tested are also something that calls into question the validity of the study. A credible study will list the number of websites, industries, and keywords tested, the keyword types studied, with the comparative metrics clearly identified and specifically categorized.
When reviewing organic search data, just like with any quantifiable data analysis, there must be a large enough data sample to truly determine correlation, and even more so if causation is asserted. If conducting a study about core algorithm update winners and losers, there must be a wide variety of industries and websites tested within those industries. Simply testing the performance of a handful of clients is not a large enough sample size.
While there is no magic minimum number of data sources to examine for credibility, a good start is at least 100 unique domains in at least eight to 10 industry niches, unless the study’s parameters specify a single industry or niche. If presenting findings on the keyword level, at least 100,000 is a decent number to look for when determining a study’s validity.
A very common bias in SEO studies is using limited data encompassing only the top-visited websites worldwide. This does not give any insight into how smaller and more niche sites have been impacted, and it is too broad to confidently state that certain industries were impacted by a certain percentage.
An authoritative source like Search Engine Land or Search Engine Roundtable will commonly write summaries of SEO studies that they have deemed credible and worthy of reporting on their websites. More often than not, these studies are likely to be trustworthy.
Similarly, if a respected SEO influencer or a reputable agency is publishing or sharing the studies, it’s more likely that they have merit. An unknown author or a study on a website that doesn’t specialize in SEO or digital marketing is probably not the best to read into or take with any value.
The date range also must be significant enough to draw any kind of definite conclusion. Simply comparing a week’s worth of data to the week before will not give any useful insights. Period-over-period data can be (but isn’t always) biased. Year-over-year data is best for removing seasonality issues, but month-over-month or period-over-period performance changes could also be monitored if the biases have been carefully considered. Day-over-day analysis is typically not useful.
Algorithm updates are a special circumstance. When Google announces a core algorithm update, it usually takes about two to three weeks to roll out fully. During this rollout period, many websites can experience higher-than-normal ranking fluctuation, even if ultimately not significantly impacted by the update when it’s all said and done.
Although sometimes content types that will be impacted are identifiable prior to the completion of the update, any study published while the update is still in progress should be taken with a huge grain of salt. Ideally, well-crafted studies will be published two to three weeks after the update has finished rolling out to account for aftershocks and thorough data collection.
It’s easy to state that certain content types performed well if one only looks at “winners” and ignores that similar content types on other sites may have seen declines. A thorough study will report on both “winners” and “losers” while being much more specific with its findings.
The best advice is to always be skeptical of any study. Look for reasons the data might be biased or whether the conclusions could be disproven. If you can’t find any and all of the above criteria check out, then you may have a significant study.
Get the latest on the world of digital marketing right to your inbox.
Copyright © JumpFly, Inc., All Rights Reserved
877.882.2850
Privacy Policy | Terms of Service | Top ▲
877.882.2850
Privacy Policy | Terms of Service | Top ▲
Copyright © JumpFly, Inc., All Rights Reserved
Get Our Marketing Insights Right To Your Inbox
Get Our Marketing Insights Right To Your Inbox
Fields containing a star (*) are required
Content from Calendly will be embedded here
Recent Comments