Web Analytics

Google Analytics Changes the Rules of the Game

Posted by Brandt Dainow on August 26th, 2011 at 9:55 am

On August 11 Google changed how Google Analytics measures visits.

The number of visits being reported will significantly increase even though traffic levels remain the same.

This is a profound change which will fundamentally change the picture Google Analytics paints of most sites.  Despite this being one of the most fundamental changes possible in any analytics system, Google have kept the change quiet and made every effort to downplay the impact.

Under the old system, a visit was treated as being over if one of three things happened:

  1. There was a gap of 30 minutes between page requests (as per the standard for a “visit”)
  2. The day ended (a violation of the standard for “visit”, but easier to program)
  3. A visitor closed their browser (also a violation of the standard)

As of August 11, a visit no longer terminates when a browser is closed.  However, a visit is now treated as ending if the source changes.

An example will show what this means:

Bob searches in Google for “buy books online.”  He then clicks an AdWord to Amazon.  After 10 minutes he leaves Amazon and runs another search, this time for “buy reference books online.” He then follows a different AdWord back to Amazon and makes a purchase.

Under the old system this would be treated as one visit. The sale would be attributed to the first keyword and the first AdWord.  The second AdWord and search term would be reported as producing zero visits.

Under the new system this is now treated as two visits. The sale is attributed to the second search and the second AdWord.

Here’s a slightly different example:

Bob is on an Amazon affiliate site.  He clicks a link to Amazon.  After looking at only one page he goes off to a different affiliate site.  After about 10 minutes he finds a link to Amazon in this site and clicks it.  He then makes a purchase.

Under the old system this would be treated as one visit resulting in a conversion.  The sale would be attributed to the first affiliate.  Under the new system this becomes two visits, with the first one resulting in a bounce, and the sale attributed to the second affiliate.

Tech Talk

What Google have done is move from counting visits to counting sessions.  The difference is often obscured by web analytics software, most of which can’t handle both.

A “visit” is defined as a series of page requests to the same website with a gap of no more than 30 minutes between them.  Leaving the site doesn’t end the visit if you return within 30 minutes.

A “session” is an uninterrupted visit to a single site.  When you leave the site the session ends, even if you return within 30 minutes.

Our two examples above constitute one visit and two sessions.

Google say they’re “bringing the definition of session in line with the common definition of a visit” – this is not the case.  What Google Analytics has done is switch from measuring visits to measuring sessions.

Impact

Google says the change is minor and will result in little more than a small increase (around 1%) in the number of visits reported.  The reality is far different.  This will fundamentally change the picture provided of most sites to a major degree.  The pattern of leaving and returning within 30 minutes is fairly common, especially on e-commerce sites (due to comparison shopping), so the only sites which will encounter the minor change predicted by Google are those with extremely low volumes of search traffic which don’t sell online.

I have run comparisons of data before and after this change on a variety of sites.  The degree to which the picture changes depends on the amount of in-out behavior the site gets.

  1. Bounce Rate: Most sites will see a huge change in bounce rate.  Most sites I surveyed saw the bounce rate rise substantially.  One B-2-B e-commerce site saw the bounce rate change from a healthy 36% to a disastrous 52%, while a change from around 30% to around 40% was seen in more than half.  Very few saw no change at all.
  2. Visits: The number of visits will rise for all sites.  My survey couldn’t find any site which saw less than a 20% rise in visits.  In some cases, reported visits doubled.
  3. New vs. Repeat Visits: The percentage of new visits will decline, as many 1-time visitors suddenly become repeat visitors.  New visits typically declined by 10% - 20% in my survey, though one saw new visits decline from 75% of traffic to 50%.
  4. Conversion Rate: Conversion rates must fall under this change – you have the same number of sales, but more visits.  Most conversion rates I surveyed fell by 10% - 30%.
  5. Visit Metrics: Average duration will decline.  In my survey it fell by 15% - 50%.  Pages per visit also declined by 15% - 30%.
  6. AdWord Performance: AdWords will no longer result in 0 visits.  Under the old system if someone clicked two AdWords for the same site within 30 minutes, the visit was attributed to the first ad, and the second treated as producing 0 visits.  Now this will be reported as two visits.
  7. AdWords Data vs. Google Analytics Data: One of the things which causes major dissatisfaction with Google is the difference between traffic you are billed for by AdWords, and what Google Analytics says you actually got.  The difference is typically 10% - 25%.  This will make Google Analytics records closer to AdWords.

Assessment

Firstly, let’s be clear that Google Analytics is not following the definitions for either “session” or “visit.”  Their explanation they are moving to standards compliance is spurious.

It would be better if Google Analytics could handle both sessions and visits, but few web analytics systems can.  If you have to choose, I guess counting sessions tells you more about your visitors than only counting visits.  It would, of course, be better if Google could use the terminology more precisely, but most of the web analytics industry is vague on the difference between sessions and visits.

The serious issue is the manner in which the change has occurred.

Firstly, lack of notice chose to dump the announcement in a blog which few are aware of – after the fact.

Secondly, they implemented the change in the middle of the month.  Most people evaluate their Google Analytics data on a calendar monthly basis.  You can forget August 2011 – by changing things in the middle of the month August’s data is effectively meaningless.

Thirdly, they did not give maintain the old metric and add a new one, they changed an existing one.  This means comparisons between data before and after are impossible.  You won’t be able to assess your performance for 2011; you can’t compare this coming Christmas with last year’s; you won’t be able to compare a period in 2011 with the same period in 2010 in order to eliminate seasonal variations. Effectively, Google Analytics has garbaged most of our historical data.  Most analyses done to compare performance with the past are now impossible.  If you want to do accurate year-on-year comparison work, you’ll need to wait till November 2012 before you can do so.

My assessment is not unique.  Others are also reporting huge changes in the picture they get of their website.  Here are some comments in response to Google’s announcement:

  • “We've experienced a 50% increase in traffic while page views and time on site has dropped through the floor.”
  • “Thank you, from the bottom of my heart, for rat-fu$&ing every metric we use to measure our performance.  Words cannot express how thankful we are for your hard work, dedication, and lack of understanding of your users!”
  • “bounce rate up 50% - time on site down 75%”
  • “I rely on these statistics as they give me a good idea of how my website is performing on a daily basis, and I have a number of benchmark figures that I aim to achieve - these benchmarks are now completely useless.”
  • “This is a jarring shift in metrics without any way to maintain meaningful analysis across the gap”
  • “I am seeing 20% increase in visits. How I am supposed to evaluate these new metrics on steroids vs. my previous metrics?”
  • “our bounce rate increased by 20%! And our pages/visit drop by 20%”
  • “My average time on site has fallen from 7+ minutes to 12 seconds. I think that this update is an example of someone fixing something that wasn't broken. Now analytics is useless.”
  • “Most of my sites' key metrics have been very steady during the past few months. However, in the past day I've seen huge jumps in a number of indicators: Visits: +22%, Avg. Time on Site: -30%, New Visits: -35%, Pages per visit: -23%.  How are we meant to compare our statistics prior to and after this change with such enormous jumps? Sorry, but this all looks very broken...”
  • “Total Visits up 22%, Avg Pageviews down 22%, New Visitors down 62%, Returning Visitors up 121%.  In actuality, there was almost no change in our traffic week over week according to every other analytics platform we use. What can I do with this data? There's no way to compare historical trending now on any of those GA metrics.”
  • “That's a pretty huge change and renders all previous data kind of useless.”
  • “The update makes my data virtually useless. It makes no sense.”
  • “I think I'm reluctantly going to have to abandon GA.”
  • “For merchants or providers whose customers like to shop around before buying, the effects of this change are enormous - eg. 100% increase in traffic reported for the same number of sales/goals = halving of conversion rate.”
  • “Frankly, Google has always failed miserably at customer service. However, Google needs to acknowledge that there is a problem.”
  • “This change is harmful, now it will be even harder to identify the first source that delivered the valuable customer.  Google Analytics team, why don't you give people choice to use the old way that counted visits, a change like that should not happen without any grace period?”
  • “Bounce rate up, time on site down, pages per visit down, avg. time on page down...  How on earth am I supposed to compare the statistics??? Not to mention I've spent hours on figuring out if there is something wrong with my servers or code.  Google, you are changing into second Microsoft - and that is not a compliment.”
  • “This change completely destroys the usefulness of new data relative to historic data. Flying blind for the time being. Argrgrgh!”

Why did Google make this change?

The official reason is Google wanted to be more standards-compliant.  But since they aren't actually any more standards-compliant than before, I don't buy it.  It does make things a little more informative for users, but if supporting users was the reason, I would have expected more concern for historical data and more publicity for the change.  Perhaps I'm cynical, but I think Google does what serves its bottom line, and if that helps users - great, if not - too bad.  I don't think they have contempt for their users, I simply think they're so caught up inside their self-congratulatory corporate culture they simply don't understand them.  Google have a long-term problem with billing disputes over AdWords which costs them tens, if not hundreds, of millions in compensation settlements each year.  There is no denying this change will bring Google Analytics numbers into line with Adwords numbers.  This will improve customer satisfaction and reduce billing disputes, perhaps saving Google many millions of dollars each year.  This is, of course, pure speculation on my part, without foundation or evidence.  I'm probably over-reacting, surely Google wouldn't be so cynical as to mess up the business records of millions of people simply to save themselves a few hundred million dollars?

What Google could/should have done

This is an extremely amateurish way of introducing the change.  It shows a complete lack of thought for the impact on the users.  This indicates the technical management at Google Analytics either don’t care about their users or don’t understand how the product is used (or both).  Either is inexcusable in a modern IT organization that claims elite status for its people.

Firstly, Google Analytics developers should know what the standard is, and comply with it.  The standard for visits and sessions has been unchanged since 1999, longer than most Google Analytics programmers have been working.  Surely the first thing you do as a web analytics programmer is learn the standard?  If this isn’t the case, it can only mean technical managers have never thought about the standard or simply don’t care.  Either is would be considered a sign of second-rate management in most organizations.

Ideally, Google should have implemented sessions as a new metric, alongside visits.  Having designed and built web analytics myself, I can assure you the easiest thing to implement within web analytics software is the addition of a new top-level metric - even if the visit metric is calculated at the data-gathering stage, as I suspect happens in Google Analytics.  I guess it is possible Google couldn’t add a new metric to the system due to additional load, as most of the data is crunched on arrival, but I find it hard to believe Google lack computing or database resources.  If, however, Google are struggling for resources, then they could have warned us this was coming.  It’s unlikely someone just got up one morning and changed the system.  They must have been working on this for some time and could have let us know the instant they first started planning the change.  This would have permitted time for the community to provide feedback which may have helped smooth the process.

Next, if they absolutely had to change the system, they should have implemented the change at the start of a new month, so people’s records could have handled it neatly.  This would have meant each full month of data was measured on a consistent basis.

Feeling bad now?  But wait – there’s more!

When they made the change, they introduced a bug.  I’m not sure that I really understand the details of the bug, as their wording is technically imprecise, but it boils down to – “you can’t trust the data between August 11 and August 18 at all.”

I did hear someone else (not from Google) explain the bug as “in those cases when arriving to the site via a search engine, the GA cookie got incremented improperly and automatically set your first visit as though it were your second visit to the site.”

However, Google now assure us this bug is fixed, and they say there are (probably) no more bugs, but if there are any they’ll fix them if/when they find them, so your data is (probably) trustworthy.

Now - go make some serious business decisions based on all this juicy and reliable(?) data...

The full announcement is here: http://analytics.blogspot.com/2011/08/update-to-sessions-in-google-analytics.html.

3 Responses to “Google Analytics Changes the Rules of the Game”

  1. Jim says:

    Excellent coverage and analysis, Brandt. Another sad little byproduct of the change is that Google gives a boost to sleazy SEO companies who can show naive clients an increase in web traffic.

    • Thom Craver says:

      No, not exactly.

      Two different visits are two different visits. If I follow a link to your site, consciously decide to end my visit, then follow another link (paid or otherwise) back to your site later, it shouldn't matter if it is within 30 minutes or not. It is, in fact a second visit.

      The Sleazy SEOs and people who truly don't understand analytics still measure visits. Visits aren't the end-all be-all. While they are an indicator for certain things, they certainly - by themselves - are not a KPI or anything worth measuring.

  2. Bill says:

    Brandt - this is way and by far the most accurate assessment of this change in reporting that I have seen. Thank you for your efforts. Since I also keep a spreadsheet of major data points, I can retro fit pre Aug. 11 data to post Aug. 11 data. But eventually I'll just roll over and accept the "new" data. The one thing that still makes me nuts is the bounce rate. For us, every thing is the same for anything that is non-paid traffic (i.e., direct, non-paid search, etc.) but for AdWords traffic, the bounce rate jumped from ~28% to ~55%. Ugh. Since every other metric I use to track the business is consistent, I can only conclude that, in fact, numbers do lie.

Leave a comment