Over the last year I've seen a huge rise in mobile visits to the websites I analyse.  At the same time, Google Analytics has reported a massive change in how mobile users surf the web.  A year ago the pattern of mobile traffic was fairly similar to desktop traffic - most people found sites in search engines, some came from other sites, and some came direct.  Some sites saw more people coming from other sites than direct, and some saw more direct traffic than referrals from other sites, but all saw most of their traffic from search engines.  That situation is now very different.  In all cases, at least half, if not more, of the mobile traffic comes direct to the site.  I see this trend in multiple countries, across multiple sectors.  Since none of the sites I track are major, universally-known, brands, I have to wonder how on earth thousands of visitors are able to find these sites without search engines, especially as I know there's no major offline marketing to attract them.  Most are first-time visitors, so they haven't bookmarked the site, which is the way most direct visitors get to a site.  And why is it that a year ago most of these people used search engines, but now they don't?

If GA was telling me these people came from other sites I'd be less worried.  The problem is that attributing a source as "direct" really means - "I asked for a referrer field and got a blank, or none."  - It can just as easily mean the browser refused to provide the referrer, as some are designed to do, or that the GA referrer request wasn't processed by the browser.

So either the entire mobile community has radically changed its web useage in the last year, without anyone noticing or commenting, or Google Analytics can't track mobile referrers as well as it used to (which could be caused by changes in mobile systems, or in GA, or both).

I'm not the only person who's starting to get concerned.  Social media people are starting to complain (http://econsultancy.com/ie/blog/62104-social-media-measurement-is-google-analytics-getting-it-wrong) and I've been working with some US telecoms people who are seeing really wierd stuff in their GA metrics, like mobile patterns completely out of tune with the market (eg: no iOS visitors).

The Google Analytics tracking code requires javascript support, which is not uniform over all mobile browsers, especially Opera, and calls a number of external files, which may not get consistently obtained over mobile connections.   Any of this could be the cause of problems.  As far as I am aware, no one (including Google) have run proper tests for reliability on mobile platforms, and I certainly lack the capabilities to conduct such tests, but I think they're needed.  Mobile is the most important growth area online, and metrics are fundamental to understanding and enhancing mobile activity.  If Google Analytics is inaccurate on referring source, assessment of marketing activity is impossible.  If we are mistakenly believe mobile people are coming direct, instead of via search, it fundamentlly changes our understanding of how to reach mobile users, and sends us up blind alleys.

Testing this can be done via an experiement, by someone able to get a range of mobile phones, access a site with them enough times to generate a valid sample size, then cross-reference what how they accessed the site with what GA reports.  Such an experiment requires serious time and investment.  Alternatively a bunch of us can pool our GA data in an annonymised fashion and subject it to analysis that way.

Have you seen changes in mobile visit sources shifting towards direct and away from search in the last year?  If so, I'd like to know.  If not, that'd be useful information too.  Do you have concerns about whether Google Analytics is accurate on mobile?  I'd love to hear them.

I guess we should have done this work a year ago.  Mobile is a different set of operating systems and browsers from desktops, and it's naive to assume GA code would work exactly the same on all mobile systems just as it does on the limited range of desktop systems.  When this data drives our understanding and decisions, naivity becomes foolishness.

So we need to know - how reliable is Google Analytics on mobiles?

In the next few weeks Google are going to upgrade Google Analytics with what they call “enhanced link attribution.”  This is MASSIVE.

At the moment, if you want to see what people clicked on, Google Analytics can only do this via where the link went to.  If you have two links in the page going to the same place, Google can’t tell which was clicked.  Since most pages have several links to the same thing, this makes understanding how people responded to a page design very limited.

In the next few weeks Google will update the system so that they can track each link on a page separately.  The advantage for design analysis is huge, and it will be an extremely valuable tool when you want to find out how to improve your design.

In order to make this available, you have to add some code to the existing Google Analytics tracking code in the site’s pages.  You have to add:

var pluginUrl =
 '//www.google-analytics.com/plugins/ga/inpage_linkid.js';
 _gaq.push(['_require', 'inpage_linkid', pluginUrl]);

This makes your tracking code look like this:

var _gaq = _gaq || [];
 var pluginUrl =
 '//www.google-analytics.com/plugins/ga/inpage_linkid.js';
 _gaq.push(['_require', 'inpage_linkid', pluginUrl]);
 _gaq.push(['_setAccount', 'UA-XXXXXX-Y']);
 _gaq.push(['_trackPageview']);

I strongly recommend this be done ASAP to all your sites.  You then need to update Google Analytics InPage Analytics to start recording this data.  The data will not be gathered unless the Google Analytics is set for it.   They should roll out the reports some time next month, but you need to have data already there for the reports to have stuff in them.

You can see Google’s announcement at http://support.google.com/analytics/bin/answer.py?hl=en&answer=2558867&topic=2558810&ctx=topic

I'm quick to criticise Google, but this is a VERY exciting move forward for everyone, so points to Google!

The Dangers From Global Web Regulation

Guest post written by Robert Pepper, vice president for global technology policy at Cisco Systems.

A grave threat to the Internet as we know it has emerged as part of well-intentioned, but fatally flawed proposals to update an international telecommunications treaty. If these radical changes are adopted, the results could be devastating. It could break the global Internet into unconnected islands of national or regional networks, extend telecommunications regulations to computing, or lead to onerous government regulation of the Internet.

The story goes back to 1984, when there were approximately eight wired telephones installed for every 100 people on the planet and an independent commission to the United Nations’ International Telecommunications Union concluded that there is “…no good reason why by the early part of the next century virtually the whole of mankind should not be within easy reach of a telephone and of all the benefits this can bring.”

Nearly 30 years later, there are only slightly more than twice the penetration of wired telephone lines, 17 per 100 people.  The astonishing achievement, however, is that while there were practically no mobile or Internet connections in 1984, globally there are now 86 mobile telephones, 33 Internet users and 24 broadband subscriptions (fixed and mobile) per every 100 people.

What may be a surprise is that this huge growth in communications – mobile and Internet – occurred largely outside of traditional telecommunications regulation and out of the focus of a 1988 ITU international treaty, the International Telecommunications Regulations (ITRs).

The fact that the Internet developed outside of traditional telecommunications regulation is no accident. Rather it was the conscious decision of countries that fostered the Internet’s early development.

The result? Huge innovation in technology and the adoption of the Internet by an estimated 2.3 billion people, with a projected reach to half the world’s 7+ billion people in the next five years.  The Internet has transformed the way we work, learn, play and communicate with one another, independent of place and distance. Entire new industries have been created, existing industries are being transformed, governments are providing their citizens better and new services and people are engaging with each other, and their governments, all enabled by an open global Internet.

In short, the Internet has become the key innovation engine of the 21st century economy.

But this growth and innovation is at risk. There are some who want to use the review of the 1988 ITRs in December at the World Conference on International Telecommunications (WCIT) as an excuse to impose legacy telecommunications regulation on the Internet and Internet Protocol networks and services. The ITRs focused on international agreements for wired telephone connections. The Internet was not addressed in the 1988 regulations and, in fact, was subsequently treated as a “Special Arrangement” exempt from the ITRs. This enabled the Internet to develop successful economic and business models based upon commercially negotiated arrangements rather than being micro managed by regulation.

Some proposals for the WCIT would control the routing of Internet traffic in the name of security but would balkanize the Internet and threaten Internet freedom. Other proposals would radically redefine telecommunications to encompass computing. Still others would use economic competition as an excuse to regulate the Internet. These radical proposals threaten not only the Internet as we know it but also innovation and national and global economic development.

We should be extremely wary of any actions that dramatically alter the global Internet landscape and stem the growth, innovation and dynamism of the Internet today. Instead, a treaty that sets out high-level principles for traditional telecommunications such as competition and open and transparent regulatory process, would foster continued innovation in technologies and services that ride on that underlying telecommunications. This is the approach that should be pursued.

This is not to say that the growth of the Internet has not raised important questions and challenges, as well as opportunities, for governments, individuals, traditional companies and their business models, and industry. However, the way we answer these questions and challenges should not be an extension of the ITU’s old rules that were written at a time when most of the world’s telecom companies were subsidiaries of government departments, often the post office. Extending an ancient regulatory model designed for another time to the rapidly evolving Internet would stifle innovation and investment.

While the Internet started in the U.S. and quickly expanded throughout North American and Europe, it now covers the world with explosive growth, especially in emerging economies and geographic centers of innovation.

Changing the regulatory and business model of the Internet now with an outdated legacy telecom model would limit the Internet’s expansion and diminish the potential for further innovation.

This would be an enormous mistake.

These days site reliability has much more impact on search listings in Google than it used to.  My own observations suggest the Penduin update added much more emphasis on assessing site reliability than we realised at the time.  Sites with less than perfect reliability are seeing listings which have remained solid for years suddenly dropping.

If you look at the Google Webmaster Tools, you can see the list of errors it complains about.  These include the HTTP status code 404 (file not found).

So the issue is - when a page is removed from your website, what happens?

Most CMS systems will update site links so the page is no longer linked to.  Google (and Bing) XML sitemaps will probably be updated.  The website sitemap page will probably be updated.  Your site does all this automatically - right?

But this still leaves Google with a listing for that page.  Sooner or later, it will ask for it again.  What happens then?

On many sites, Google will get a 404 error.  This is BAD.  Boo to 404! 

A 404 status code merely says "I can't find it."  It could indicate a minor malfunction, or that the page is just damaged, soon to be restored, or that the page is gone for good.  Google has no way of knowing.  So it has to come back later and try again.  If it keeps getting 404, does this indicate the page is gone for good (and why are you wasting Google's time) or does it indicate you've got a second-rate website which doesn't work properly?  Obviously you don't want Google removing a listing the instant it gets a 404, that's terrible punishment for what may be a temporary glitch.  So poor old Google has to keep hammering the site, asking for the same page, until it finally gives up.

THERE IS A BETTER WAY

This is sloppy server configuration.  We have had a better way of doing things for 20 years.  You need to use the HTTP status code system properly:

1. If the page has moved to a new location, put a 301 (permanent redirect) in place.  This tells Google to replace the old URL with the new one.  You can also do this if the page is gone, but equivalent copy exists somewhere else in the site.
2. If the page is gone and there is no replacement, put a 410 (permanently removed) in place.  This tells Google to remove the URL and never ask for it again.

Google likes both these methods, and will reward your site for it.  Besides, it's just doing things properly, as the web was intended to be done.

Don't move or remove a page from your site without setting the correct status code.  Sites that fail in this respect will see their search listings suffer.  Don't let that be you.  Know your HTTP status code policies, and ensure you use status codes properly.

The full list of HTTP status codes is at http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

Here's a basic video from Google on errors it may report in Webmaster Tools.  http://www.youtube.com/watch?v=tH1gQoCd05g&feature=plcp

Many people have been complaining recently that Google has indexed pages which were explicitly forbidden.  There are two methods for telling a search engine to avoid a page - via a robots.txt file or by a NOINDEX tag in the code.  It seems Google can ignore your guidance and list the pages anyway.

Google have stated:

COMMENT:

At first glance, this may seem logical: why would you put a +1 button on a page if you didn't want people to find it in a search engine?  However, that many people are complaining indicates that there can be cases where you want people to be able to +1 a page, but not see that page listed in Google.  The reasons why you might want this are many - you may want to show a page to some of your +1 friends, but not the general public; there may be several pages with duplicate content and you're trying to control which gets listed in Google; junior content creators may dump the +1 button in without realising the page is restricted.

I recently saw a very serious case like this.  A client had a number of pages in their private investor relations section detailing a forthcoming merger, something they desperately needed to keep quiet until the deal was complete.  Having these pages behind a login, with NOINDEX tags, and with a robots.txt file restricting access, they assumed they would be safe from public scrutiny.  A minor glitch in changing a content template meant that these pages acquired a +1 button, and Google promptly listed the pages.  Because these pages required a login, you couldn't read the details, but the mere fact you could see their titles in Google was enough to alert the markets to the coming merger, with serious consequences.

ANALYSIS:

One of the stated USP's of Google+ was to offer better privacy than Facebook.  Putting a +1 button on a page does not mean I want Google to list it in their search engine, it means I want people to be able to list it in their +1 page.  When Google assume that everything we +1 should be open and public, they take lack of privacy much further than Facebook ever have.  At least when Facebook dig into people's private data, they restrict it to themselves and their advertising clients.  Google just give it everyone.

This is also demonstrates a complete disregard for internet standards.  The internet is composed of thousands of products from thousands of companies.  The only reason the web works is because everyone abides by a common set of core standards.  Once companies start deciding their innovations can ignore these standards, the internet starts to break down.   These standards are not a block on innovation, nor are they restrictive or fixed forever.  If companies want their innovations to have a place in the internet, or if they're unhappy with the existing standards, there are established mechanisms for changing things via bodies such as IETF, OASIS, and W3C.  If Google think +1 should be able to override robots.txt, they should submit a proposal to IETF, and let it run the course.  IETF is possibly the most democratic institution on the planet, and the fairest venue for any new technical proposal.  Any standard from IETF has universal acceptance.

The standards are how we all understand how everything works.  Google's announcement about +1 overriding robots.txt is buried in a minor FAQ.  How is anyone supposed to know?  I am certain most people don't because there is a vast amount of discussion going on at the moment as people try to work out why their restricted content suddenly appeared in Google.  If Google had put this through the standards bodies, we would have all known in advance.

The reality is that if Google had tried to create an internet standard stating +1 buttons override robots.txt, we all know it would have failed.  No matter how important Google thinks it is, +1 is of little import compared to core internet standards.  However, I doubt that was why they didn't try.  I suspect the real reason is they simply didn't think.  To me this suggests Google don't get it - they don't understand what the web is about.

When Google decide that +1 overrides robots.txt, they say "get stuffed to all of you, we're more important than the rest of the world."

When Tim Berners-Lee invented HTML, he chose not to patent it, or sell it to any vendor.  His reason was that only open, non-proprietary systems could create the connected world we now know as the World Wide Web.  If vendors did their own thing, we'd merely end up with a bunch of isolated competing systems which don't interact.  Adherence to the standards of the internet is critical, and no-one, not even The Great Google, has any business breaking those standards.

Get it right Google - restrictions in robots.txt are absolute, and a search engine should know better.

Here's a quick summary of why.  The full announcement is below.

1) Data is transmitted to the USA without proper notification to users. Lack of proper notification is illegal under European and German privacy laws.

2) Data is used to create online profiles and track people for 2 years. Creation of online profiles and cross-site tracking is illegal in Germany and under EU laws unless prior informed consent is given.   Browser settings do not constitute informed consent.

3) Facebook's privacy statement and T&C's are vague, confusing, uninformative, and therefore illegal under German requirements.  This means, even if you read and agreed, you still would not have given informed consent.

Their final comment: "Institutions must be aware that they cannot shift their responsibility for data privacy upon Facebook or the users."

Here is the entire text of the English-language announcement:

P R E S S   R E L E A S E

ULD to website owners:
„Deactivate Facebook web analytics“

The Data Protection Commissioner’s Office (Independent Centre for Privacy Protection - ULD) calls on all institutions in the federal state of Schleswig-Holstein, Germany to shut down their fan pages on Facebook and remove social plug-ins such as the “like”-button from their websites. After a thorough legal and technical analysis ULD comes to the conclusion that such features are in violation of the German Telemedia Act (TMG) and of the Federal Data Protection Act (BDSG), respectively the Data Protection Act of Schleswig-Holstein (LDSG SH). By using the Facebook service traffic and content data are transferred into the USA and a qualified feedback is sent back to the website owner concerning the web page usage, the so called web analytics (Ger.: Reichweitenanalyse). Whoever visits facebook.com or uses a plug-in must expect that he or she will be tracked by the company for two years. Facebook builds a broad individual and for members even a personalised profile. Such a profiling infringes German and European data protection law. There is no sufficient information of users and there is no choice; the wording in the conditions of use and privacy statements of Facebook does not nearly meet the legal requirements relevant for compliance of legal notice, privacy consent and general terms of use.

ULD expects from website owners in Schleswig-Holstein to immediately stop the passing on of user data to Facebook in the USA by deactivating the respective services. If this does not take place by the end of September 2011, ULD will take further steps. After performing the hearing and administrative procedure this can mean a formal complaint according to sect. 42 LDSG SH for public entities, a prohibition order pursuant to sect. 38 par. 5 BDSG as well as a penalty fine for private entities. The maximum fine for violations of the TMG is 50TS Euro.

Commissioner Thilo Weichert, head of ULD: “ULD has pointed out informally for some time that many Facebook offerings are in conflict with the law. This unfortunately has not prevented website owners from using the respective services and the more so as they are easy to install and free of charge. Web analytics is among those services and especially informative for advertising purposes. It is paid with the data of the users. With the help of these data Facebook has gained an estimated market value of more than 50 bn. dollars. Institutions must be aware that they cannot shift their responsibility for data privacy upon the enterprise Facebook which does not have an establishment in Germany and also not upon the users.

Our current call is only the beginning of a continuing privacy impact analysis of Facebook applications. ULD will continue in cooperation with other German data protection authorities. A comprehensive analysis is not to be performed at one go for a small privacy agency such as ULD; moreover is Facebook constantly changing its technical procedures and terms of use. Nobody should claim that there are no alternatives; there are European and other social media available that take the protection of privacy rights of Internet users far more serious. That they also may contain problematic applications must not be a reason to remain idle towards Facebook, but must prompt us as supervisory authorities to pursue these violations. Users can take their part in trying to avoid privacy adverse offerings.”

To Internet users ULD offers the advice to keep their fingers from clicking on social plug-ins such as the “like”-button and not to set up a Facebook account if they wish to avoid a comprehensive profiling by this company. Profiles are personal information; Facebook is requiring its members to register their actual name.

ULD has published its privacy evaluation of website analytics by Facebook in German language on the Internet at

https://www.datenschutzzentrum.de/facebook/

This analysis will be continued, that is extended and specified. Suggestions to ULD are welcome by e-mail to

facebook@datenschutzzentrum.de

For inquiries or in case of general further questions please contact:
Unabhängiges Landeszentrum für Datenschutz Schleswig-Holstein
Holstenstr. 98, 24103 Kiel, Germany
Phone: ++49 (0)431 988-1200, Fax: -1223

Some people reading this may think the German's are being silly. However, I think we need to respect the culture of different countries.  The web brings the cultures of hundreds, if not thousands, of regions and groups into a shared environment.  We cannot assume that any one approach to online privacy is "correct" and that everyone else is "wrong."  In particular, we cannot assume the USA's business-centric attitude that consumers do not have a right to privacy will be respected anywhere else in the world.  Significant portions of the planet believe personal privacy is a fundamental human right.  We must respect the right of others to live in the manner they want, especially if they live in a democracy.  International brands, such as Facebook and Google, have a responsibility to ensure they understand the differing attitudes towards privacy in the countries in which they do business, and work within the boundaries each country demands.  Failure to comply will lead to situations like this, costing the company money and potentially bringing legal penalties to their staff (as happened with Google staff, who were jailed in Germany for privacy violations).  It's just bad business.

The number of visits being reported will significantly increase even though traffic levels remain the same.

This is a profound change which will fundamentally change the picture Google Analytics paints of most sites.  Despite this being one of the most fundamental changes possible in any analytics system, Google have kept the change quiet and made every effort to downplay the impact.

Under the old system, a visit was treated as being over if one of three things happened:

1. There was a gap of 30 minutes between page requests (as per the standard for a “visit”)
2. The day ended (a violation of the standard for “visit”, but easier to program)
3. A visitor closed their browser (also a violation of the standard)

As of August 11, a visit no longer terminates when a browser is closed.  However, a visit is now treated as ending if the source changes.

An example will show what this means:

Bob searches in Google for “buy books online.”  He then clicks an AdWord to Amazon.  After 10 minutes he leaves Amazon and runs another search, this time for “buy reference books online.” He then follows a different AdWord back to Amazon and makes a purchase.

Under the old system this would be treated as one visit. The sale would be attributed to the first keyword and the first AdWord.  The second AdWord and search term would be reported as producing zero visits.

Under the new system this is now treated as two visits. The sale is attributed to the second search and the second AdWord.

Here’s a slightly different example:

Bob is on an Amazon affiliate site.  He clicks a link to Amazon.  After looking at only one page he goes off to a different affiliate site.  After about 10 minutes he finds a link to Amazon in this site and clicks it.  He then makes a purchase.

Under the old system this would be treated as one visit resulting in a conversion.  The sale would be attributed to the first affiliate.  Under the new system this becomes two visits, with the first one resulting in a bounce, and the sale attributed to the second affiliate.

Tech Talk

What Google have done is move from counting visits to counting sessions.  The difference is often obscured by web analytics software, most of which can’t handle both.

A “visit” is defined as a series of page requests to the same website with a gap of no more than 30 minutes between them.  Leaving the site doesn’t end the visit if you return within 30 minutes.

A “session” is an uninterrupted visit to a single site.  When you leave the site the session ends, even if you return within 30 minutes.

Our two examples above constitute one visit and two sessions.

Google say they’re “bringing the definition of session in line with the common definition of a visit” – this is not the case.  What Google Analytics has done is switch from measuring visits to measuring sessions.

Impact

Google says the change is minor and will result in little more than a small increase (around 1%) in the number of visits reported.  The reality is far different.  This will fundamentally change the picture provided of most sites to a major degree.  The pattern of leaving and returning within 30 minutes is fairly common, especially on e-commerce sites (due to comparison shopping), so the only sites which will encounter the minor change predicted by Google are those with extremely low volumes of search traffic which don’t sell online.

I have run comparisons of data before and after this change on a variety of sites.  The degree to which the picture changes depends on the amount of in-out behavior the site gets.

1. Bounce Rate: Most sites will see a huge change in bounce rate.  Most sites I surveyed saw the bounce rate rise substantially.  One B-2-B e-commerce site saw the bounce rate change from a healthy 36% to a disastrous 52%, while a change from around 30% to around 40% was seen in more than half.  Very few saw no change at all.
2. Visits: The number of visits will rise for all sites.  My survey couldn’t find any site which saw less than a 20% rise in visits.  In some cases, reported visits doubled.
3. New vs. Repeat Visits: The percentage of new visits will decline, as many 1-time visitors suddenly become repeat visitors.  New visits typically declined by 10% - 20% in my survey, though one saw new visits decline from 75% of traffic to 50%.
4. Conversion Rate: Conversion rates must fall under this change – you have the same number of sales, but more visits.  Most conversion rates I surveyed fell by 10% - 30%.
5. Visit Metrics: Average duration will decline.  In my survey it fell by 15% - 50%.  Pages per visit also declined by 15% - 30%.
6. AdWord Performance: AdWords will no longer result in 0 visits.  Under the old system if someone clicked two AdWords for the same site within 30 minutes, the visit was attributed to the first ad, and the second treated as producing 0 visits.  Now this will be reported as two visits.
7. AdWords Data vs. Google Analytics Data: One of the things which causes major dissatisfaction with Google is the difference between traffic you are billed for by AdWords, and what Google Analytics says you actually got.  The difference is typically 10% - 25%.  This will make Google Analytics records closer to AdWords.

Assessment

Firstly, let’s be clear that Google Analytics is not following the definitions for either “session” or “visit.”  Their explanation they are moving to standards compliance is spurious.

It would be better if Google Analytics could handle both sessions and visits, but few web analytics systems can.  If you have to choose, I guess counting sessions tells you more about your visitors than only counting visits.  It would, of course, be better if Google could use the terminology more precisely, but most of the web analytics industry is vague on the difference between sessions and visits.

The serious issue is the manner in which the change has occurred.

Firstly, lack of notice chose to dump the announcement in a blog which few are aware of – after the fact.

Secondly, they implemented the change in the middle of the month.  Most people evaluate their Google Analytics data on a calendar monthly basis.  You can forget August 2011 – by changing things in the middle of the month August’s data is effectively meaningless.

Thirdly, they did not give maintain the old metric and add a new one, they changed an existing one.  This means comparisons between data before and after are impossible.  You won’t be able to assess your performance for 2011; you can’t compare this coming Christmas with last year’s; you won’t be able to compare a period in 2011 with the same period in 2010 in order to eliminate seasonal variations. Effectively, Google Analytics has garbaged most of our historical data.  Most analyses done to compare performance with the past are now impossible.  If you want to do accurate year-on-year comparison work, you’ll need to wait till November 2012 before you can do so.

My assessment is not unique.  Others are also reporting huge changes in the picture they get of their website.  Here are some comments in response to Google’s announcement:

• “We've experienced a 50% increase in traffic while page views and time on site has dropped through the floor.”
• “Thank you, from the bottom of my heart, for rat-fu$&ing every metric we use to measure our performance.  Words cannot express how thankful we are for your hard work, dedication, and lack of understanding of your users!”
• “bounce rate up 50% - time on site down 75%”
• “I rely on these statistics as they give me a good idea of how my website is performing on a daily basis, and I have a number of benchmark figures that I aim to achieve - these benchmarks are now completely useless.”
• “This is a jarring shift in metrics without any way to maintain meaningful analysis across the gap”
• “I am seeing 20% increase in visits. How I am supposed to evaluate these new metrics on steroids vs. my previous metrics?”
• “our bounce rate increased by 20%! And our pages/visit drop by 20%”
• “My average time on site has fallen from 7+ minutes to 12 seconds. I think that this update is an example of someone fixing something that wasn't broken. Now analytics is useless.”
• “Most of my sites' key metrics have been very steady during the past few months. However, in the past day I've seen huge jumps in a number of indicators: Visits: +22%, Avg. Time on Site: -30%, New Visits: -35%, Pages per visit: -23%.  How are we meant to compare our statistics prior to and after this change with such enormous jumps? Sorry, but this all looks very broken...”
• “Total Visits up 22%, Avg Pageviews down 22%, New Visitors down 62%, Returning Visitors up 121%.  In actuality, there was almost no change in our traffic week over week according to every other analytics platform we use. What can I do with this data? There's no way to compare historical trending now on any of those GA metrics.”
• “That's a pretty huge change and renders all previous data kind of useless.”
• “The update makes my data virtually useless. It makes no sense.”
• “I think I'm reluctantly going to have to abandon GA.”
• “For merchants or providers whose customers like to shop around before buying, the effects of this change are enormous - eg. 100% increase in traffic reported for the same number of sales/goals = halving of conversion rate.”
• “Frankly, Google has always failed miserably at customer service. However, Google needs to acknowledge that there is a problem.”
• “This change is harmful, now it will be even harder to identify the first source that delivered the valuable customer.  Google Analytics team, why don't you give people choice to use the old way that counted visits, a change like that should not happen without any grace period?”
• “Bounce rate up, time on site down, pages per visit down, avg. time on page down...  How on earth am I supposed to compare the statistics??? Not to mention I've spent hours on figuring out if there is something wrong with my servers or code.  Google, you are changing into second Microsoft - and that is not a compliment.”
• “This change completely destroys the usefulness of new data relative to historic data. Flying blind for the time being. Argrgrgh!”

Why did Google make this change?

The official reason is Google wanted to be more standards-compliant.  But since they aren't actually any more standards-compliant than before, I don't buy it.  It does make things a little more informative for users, but if supporting users was the reason, I would have expected more concern for historical data and more publicity for the change.  Perhaps I'm cynical, but I think Google does what serves its bottom line, and if that helps users - great, if not - too bad.  I don't think they have contempt for their users, I simply think they're so caught up inside their self-congratulatory corporate culture they simply don't understand them.  Google have a long-term problem with billing disputes over AdWords which costs them tens, if not hundreds, of millions in compensation settlements each year.  There is no denying this change will bring Google Analytics numbers into line with Adwords numbers.  This will improve customer satisfaction and reduce billing disputes, perhaps saving Google many millions of dollars each year.  This is, of course, pure speculation on my part, without foundation or evidence.  I'm probably over-reacting, surely Google wouldn't be so cynical as to mess up the business records of millions of people simply to save themselves a few hundred million dollars?

What Google could/should have done

This is an extremely amateurish way of introducing the change.  It shows a complete lack of thought for the impact on the users.  This indicates the technical management at Google Analytics either don’t care about their users or don’t understand how the product is used (or both).  Either is inexcusable in a modern IT organization that claims elite status for its people.

Firstly, Google Analytics developers should know what the standard is, and comply with it.  The standard for visits and sessions has been unchanged since 1999, longer than most Google Analytics programmers have been working.  Surely the first thing you do as a web analytics programmer is learn the standard?  If this isn’t the case, it can only mean technical managers have never thought about the standard or simply don’t care.  Either is would be considered a sign of second-rate management in most organizations.

Ideally, Google should have implemented sessions as a new metric, alongside visits.  Having designed and built web analytics myself, I can assure you the easiest thing to implement within web analytics software is the addition of a new top-level metric - even if the visit metric is calculated at the data-gathering stage, as I suspect happens in Google Analytics.  I guess it is possible Google couldn’t add a new metric to the system due to additional load, as most of the data is crunched on arrival, but I find it hard to believe Google lack computing or database resources.  If, however, Google are struggling for resources, then they could have warned us this was coming.  It’s unlikely someone just got up one morning and changed the system.  They must have been working on this for some time and could have let us know the instant they first started planning the change.  This would have permitted time for the community to provide feedback which may have helped smooth the process.

Next, if they absolutely had to change the system, they should have implemented the change at the start of a new month, so people’s records could have handled it neatly.  This would have meant each full month of data was measured on a consistent basis.

Feeling bad now?  But wait – there’s more!

When they made the change, they introduced a bug.  I’m not sure that I really understand the details of the bug, as their wording is technically imprecise, but it boils down to – “you can’t trust the data between August 11 and August 18 at all.”

I did hear someone else (not from Google) explain the bug as “in those cases when arriving to the site via a search engine, the GA cookie got incremented improperly and automatically set your first visit as though it were your second visit to the site.”

However, Google now assure us this bug is fixed, and they say there are (probably) no more bugs, but if there are any they’ll fix them if/when they find them, so your data is (probably) trustworthy.

Now - go make some serious business decisions based on all this juicy and reliable(?) data...

The full announcement is here: http://analytics.blogspot.com/2011/08/update-to-sessions-in-google-analytics.html.

No need to read more - the error is on the first line - "Dear NULL".  I'm a NULL?  Is that a comment on my importance?  My value as a customer?  If I could walk into an Intel shop would I be greeted with a cheerful "Howdy, Mr Insignificant Nobody"?

Ok - I'm being a little a little over the top, but it's such a glaring fault I had to discuss it.

Let's go through the series of errors and omissions required to achieve this (un)remarkable degree of (mis)targetting:

1) The person who wrote the email software had to consciously insert a command to say "insert First Name here."  Neither they, their supervisor, or any of the testers thought about "what if we don't have a first name."  Gee, why would you?  Every marketing database is absolutely perfect, right?  There's never any need to check for errors, is there?  You never encounter a single item of data missing, do you?  In addition, NULL is a logical data response, not actual data.  You have to write code in a certain way to get NULL treated as data and inserted into output, and that way involves cutting corners (technically this is called "loose data typing".  A good programmer would simply never have got into the mess.  I can guarrantee there was no one on the coding team with more than 3 years experience, including the supervisor.  More experienced programmers wouldn't have made this error.

2)  No one sending out these emails ran a test first.  "Let's just trust the system, and if it costs us sales to be so lazy about it, who cares!"

3)  The email system isn't pulling data from the database properly.  This email was about a subscription service, where they actually DO have my name, so NULL should never have appeared.

I am sure no one reading this would be so unprofessional with their email marketing, but let's review the lessons anyway:

1) Don't use email marketing software you haven't testing first.  Ensure your tests cover every possible form of missing data.  I keep a special database with really terrible records mixed in amongst good ones.  The bad records cover every possible form of mistake - missing data, mismatched data, duplicates, etc.   This is such a common need, you can buy such "testing" databases.

2) Don't assume your marketing database is still working properly, just because it worked last time.  Data always gets corrupted.  Hard disks are metallic moving parts - they wear out.  As they degrade, data gets trashed.  That's why we all spend so much on backups.  Run a test on all email campaigns before sending.

The salutation is the most important line in any email.  It instantly tells someone whether you're sending junk to them or not.  "Dear Sales" or "Dear Info" (or "Dear NULL") instantly say - "mass shotgun email - junk me quick!"  Just a simple "Hi!" if you don't have a name is preferable.

We need to remember that email marketing is a two-step process.  Before we can convince people with our great emails, catchy subject lines, etc, we must first convince them to read the email.  That process begins with the salutation - it's the first impression and sets the tone for the whole encounter.

This is an idea which has been around since the web started to get traffic congestion in 1995.  At present all ISP's treat all site's equally, and bandwidth is allocated according to customer demand.  This is called "network neutrality."  It's a fairly fundamental aspect of ensuring the web is democratic and stops a few large corporations dominating everything.  It means Joe's Fishing Blog gets the same treatment as Microsoft.com, and is a key reason why small sites (like Facebook) can grow so quickly into bigger ones (like Facebook), and why bloggers can be assured anyone who wants to can get to their site.

Some people, including the FCC, are very concerned that this may be the beginning of the end of the open web, but I think history will show this to be a dumb decision for Verizon.  It's just another case of forgetting where the money lies - with the customer (not the provider).  In my experience as an internet investment advisor for venture capital firms, I've seen plenty of dumb ideas get popular with the corporates by focusing too much on what was in it for the corporations and forgetting to ask "what's was in it for the customer?".  People can too easily forget that all online activity, no matter what, starts and ends with the customer/visitor/user.  If people don't use/view/buy it, it doesn't matter how great it is, you're not going to make any money.

Bandwidth is a finite shared resource.  To make one site faster, I have to make other sites slower.  So how does Verizon granting Youtube  a traffic priority help the Verizon subscriber?  With or without priority channels, I can still get Youtube.  Are Verizon going to refuse access to websites which don't pay?  Of course not, most sites won't pay and there are hundreds of millions of them. Verizon couldn't approach every single one even if they wanted to.  Verizon can hardly cherry-pick a few big companies and force them to pay or be blocked.  Can you imagine the conversation in Redmond:

"Well, Mr Balmer, it's like this.  Unless Microsoft give us a million dollars per year, we'll block access to the Microsoft website for Verizon subscribers."

Do you think Steve Balmer would quiver in his boots with fear at the damage he would suffer, or just conclude Verizon were just going to hurt themselves by degrading the service they offer?  Would he reach for his chequebook, or the door?

Do we expect that the speed difference will be so great it will make that much difference?  Unless Verizon is so slow most sites don't work at all, who's going to notice an improvement?

If this will have any effect, it will discourage people from using Verizon.  In order to make it worthwhile to pay for, the performance difference has to be obvious to users.  That means non-paying sites have to be slow, so I get a sense of relief when I hit a priority one.  Why would I sign up with Verizon, where any small site I may be interested in is slower than on other networks?  Will people carefully analyse their experience and conclude Verizon prioritize key sites, or will they make a snap judgement that Verizon has just got slower and move to a competitor.  Do Verizon expect that most people will be aware of this deal?  Can you imagine that down on the farm in Smallville, Martha will say to Jonathan  - "When you've finished fixing the tractor, switch us to Verizon because several of our favourite websites have priority deals to ensure higher bandwidth allocation"?  Server bandwidth allocation is not exactly a prime marketing USP for the general consumer.

Furthermore, how many organisations will pay for priority access?  Smaller companies won't be able to afford it, non-US companies won't have heard of it, and most bigger sites won't bother.  People will continue to visit Facebook and Youtube anyway.  Youtube won't be able to charge higher ad rates just because a few Verizon customers are on a priority channel, the ads will still get delivered to slower networks just as effectively.  So there's no incentive for advertisers to pay higher rates.  If you can't sell the ad space at a higher rate, why would you eat into existing profits and give a chuck of cash to Verizon?  As a business, we only pay a premium price if we can sell at a premium rate.  There's no financial benefit for Youtube, so why bother to give Verizon a penny?

It won't surprise me to see companies like Youtube pay for priority with Verizon - for a year or two.  Many large organisations are stuffed with idiots who've forgotten what the web is really about, and cash-rich dot.com organisations are always blowing huge sums on dumb things.  That doesn't mean this is the way of the future, just a small blind alley which will die quietly.  Net neutrality is an inherent aspect of the web - its not just built into the technology, more importantly it's an inescapable fact of the way people use it.

Hey big corporations! Stop trying to control the web:

"We hold these truths to be self-evident: that all websites are created equal; that they are endowed by their creators with certain unalienable rights; that among these are the right to bandwidth, freedom from censorship, and the right to pursue popularity."

In other words, you should make it plain to everyone at work and home exactly what you think of them, and that everyone at work should know exactly what you're like at home and vice versa.

I think this is a nasty form of totalitarianism which, if allowed to continue as far as he wants, would destroy society itself.  Of course, society won't allow itself to be destroyed. Sooner or later, Zuckerberg will take a big fall.

The definition of totalitarianism is (thanks to Wikipedia):

"a system, usually under the control of a single person or group, recognizes no limits to its authority and strives to regulate every aspect of public and private life wherever feasible.  Totalitarianism is generally characterised by the coincidence of authoritarianism (where ordinary citizens have no significant share in state decision-making) and ideology (a pervasive scheme of values promulgated by institutional means to direct most if not all aspects of public and private life)."

Well, Facebook certainly is authoritarian - look at user dissatisfaction with privacy policies and their lack of influence over how Facebook handles their data.  In fact, if you read Facebook's T&C's, you'll see it's so authoritarian it thinks your data is the property of Facebook.  Do they have a pervasive scheme of values seeking to direct all aspects of life?  If they think privacy should cease to exist - yes.

The idea that we should have the same face in public that we do in private is just stupid.  It's naive, chaotic, and symptomatic of a naive American youth.

Privacy in the workplace is essential: What do you think your workplace would be like if everyone knew exactly what everyone else thinks about them?  Do you want your boss to know everything you think about how they do their job and what their personality is like?  Do you want your co-workers to know what you think of them?  When the next promotion is coming up, do you want everyone else who's interested in getting that position to know you're after it too?  If you review employee performance, do you want all those employees to know everything which is discussed about them?  Do you want your competitors to know what moves you're planning next?

Let's make it more personal:  What about the annoying, but well-meaning, person who has a slight personality flaw?  Every office has one.  Perhaps they talk too long at meetings, or give incredibly boring presentations, or have a irritating habit of laughing too much?  They're a nice person, but they have this little habit which people comment on to each other.  They would be devastated if they realised how people perceive them.  It wouldn't help, and they don't need to know.

There are good arguments in favour of keeping personal and work identities separate and secret from each other.  Imagine if your boss is radical tea party republican, and you're a left-wing democrat.  Do you want them to know?  Would you want your boss to know you spend evenings doing volunteer work providing abortion counselling while your boss spends their weekends protesting outside abortion clinics?   What if your boss spends the evenings holding evangelical prayer meetings while you hang out at the anarchist's collective deriding all religion as primitive superstition?  These are real differences in outlook, but they have no impact on your work performance.  You have a right to keep these things separate.

While many companies encourage after-hours socialising between staff and managers, there are others (very successful companies) which think management and staff should not socialise together because a certain degree of a formal barrier between them makes it easier to control staff.

Mark Zuckerberg can get away with telling everyone in Facebook what he thinks of them - he's the boss.  He's never had the experience of working for an a**hole and having to keep quiet in order to pay the rent.  But if he thinks all his employees and investors tell him exactly what they think of him, he's being naive.

Society depends on secrecy.  Secrecy is what holds us together.  We offer to others the faces which will create the most harmonious relations between us.  We do not tell everyone everything we think, nor should we.  Sometimes it's just a passing emotion, sometimes it would divert us into irrelevant distractions, sometimes it would be unnecessary and just plain hurtful.

European and Asian societies tend to be more private than people in the USA.  Being invited into someone's home in most European and Asian countries is a BIG deal.  It means you have been accepted into an inner circle of trust which few except family members ever penetrate.  You would never invite workmates into your home.  I suspect that having less living space means these societies have needed to create increased psychological space to compensate.

As we age, we learn to create and manipulate different personas for different purposes.  I spent 15 years working in Australia as a salesman.  I learned that if I didn't act the "true-blue aussie" I wasn't going to make sales.  I had to learn to roughen my accent, say "mate" every second word, and pretend to love sport.  I then moved to London.  If I had acted the same way in London, I would not have been taken seriously.  Those same characteristics have different associations in London, they mean different things.  I had to act differently.  Zuckerberg shows a typical youthful inexperience of life by not realising this.

Partial personas, which emphasis limited aspects of our personality, are how we interact.  They're fundamental to society.

The world needs privacy, and it needs people to be able to operate differently in different spheres.  My problem with Facebook is that it makes it hard for me to have multiple personas.  I don't want my work contacts knowing about my private activities.  I want to be judged by them on my work, not what I do on Saturday night.  When Richard Branson started Virgin, he had no money.  His phone number was the telephone box outside his parents house.  Do you think he'd have got anywhere if his customers had known that?  When Saatchi and Saatchi started the two brothers had a total of £1,000 each to start the business.  They spent it all on expensive suits to make themselves look like they were already a success.  These people were lying, hiding the truth from their customers.  Did it do any harm?  No.  We see such things as clever and ambitious.

This, in my view, offers the opportunity for a Facebook killer - the social networking site which assumes that unless you actively publish something to everyone, you want it private, and which makes it easy to present multiple identities and keep them separate.

The web represents a dynamic tension between the ability to manipulate your personal data so as to create multiple partial (or even fictional) identities and a complete lack of privacy.  Zuckerberg represents the extremity of the "open" position, and it needs to be resisted.  If Facebook continues Zuckerberg's drive to a totally open society, we'll learn the hard way why that's not a good idea as people lose jobs because of their private beliefs.  Freedom of conscience will become a thing of the past.  Privacy is fundamental to society.  People don't want to have to reveal everything to everyone.  If Facebook continues like this it won't make us more open, it will just make us better liars.

If you've been working with the Google Analytics API for some time, you may be aware that specifics of some syntax has changed over time.  However, Google kept processing of the old syntax in the system so you didn't necessarily have to change your code.  At some stage during May this changed and some older syntax will not work in certain query combinations.

Until recently, the following combinations would all work fine and produce the same results:

1. source==google&&medium==cpc
2. ga:source==cpc&&ga:medium==cpc
3. source==google;medium==cpc
4. ga:source==google;medium==cpc

However, recently support for && was dropped so you now must use ;.

!Now here's the fun part:

ga: is no longer optional in combined queries, it is required.  So (3) will not work.  However, it is still optional in single parameter queries.  Thus:

source==google will work exactly the same as ga:source==google, but if you combine parameters, ga: becomes required.

This humble post is offered to those of you who, like me, have old code still in use. I hope to save you the three hours of debugging it took me to work out what was wrong with code which had worked perfectly till now.

My thanks to Steve Whittle of Displaysense (www.displaysense.com) who helped me with this when I got stuck in a blind alley.  Sometimes you can get your head too close to the code...

END OF TECHIE STUFF, START OF RANT:

I guess someone in Google Analytics is tinkering with the code for processing multiple parameters, and has removed the old code for handling older syntax.  While I have immense respect for the work Google do in bringing great products to us for nothing, I have to say this is typically shoddy development management.  We now have different components of the same product with competing syntax rules, and no version history or similar documentation to help poor developers like me who write applications which use the API.

I've strongly suspected for many years no one in Google is supervising Google Analytics development properly, that each coder can pretty much do their own thing, because the product is riddled with processing inconsistencies.  This is exactly the sort of instance which proves it.  I'm not blaming the person who made this change, they simply followed the spec as it is now.  I'm blaming their management, who have obviously either failed to document the code and development procedures properly, or don't bother to ensure their coders follow such documentation (if it exists). If you're a technical manager at Google, please go read up on something called version control - you obviously skipped or slept through those lectures in Computer Science 101.

If you want to build applications which people use, and if you want developers to build systems which interact with these applications, you can't make changes at any time and without documentation and prior warning.  You need to make changes by the version.  That means you roll out one version of the product and leave it alone.  You then roll all improvements into the next version.  You warn people in advance the next version is coming at a specified date, and you tell them what the changes will be.  Ideally, you also maintain backward compatibility for a specified period of time or run both versions in parrallel for a while.  This means developers can plan/budget updates to their systems, make those changes in a controlled fashion, and generally walk in parrallel with you.

Google may have just broken a large number of commercial applications which plug into the Google Analytics API.  The first thing anyone knows about the change is when their systems fail or their clients start screaming at them because they can't compile their monthly reports.

If Google were just dumping freeware into the marketplace, this would still be shoddy work, but less of an issue.  However, they encourage commercial development on the Google Analytics API platform, they even promote companies which do so.  People who do this become their partners in a joint business, even if there's no contract in place.  This gives Google a moral obligation to communicate to their partners, and a commercial incentive to do so.  How can you expect people to partner with you if you ignore them and break their systems.  Saying the product is free is no justification for sub-standard work.

In my experience, Google are filled with bright minds doing amazing things.  They're certainly one of the leading developers of web apps today.  But time and time again, they demonstrate they know almost nothing about being part of a development ecosystem - about technical communication or how to support development partners.  There's really no excuse for it, this isn't rocket science.  You only have to look at how other companies which are embedded into development ecosystems work to see what needs to be done.  Can you imagine Oracle changing the syntax in their databases without warning people in advance or documenting the change?  Or Sun making undocumented changes inside Java without warning, or even so much as changing a sub-version number?  Even self-organised open source development runs through version releases, with change documentation.  It looks like everyone can get it right except Google.  What's wrong with the people at Google?  What wierd cult of corporate introspection makes it so hard for Google employees to recognise they exist in a world which is bigger than them, and that they need to interact with?

Corporate arrogance seems to be a feature of dominating IT companies.  IBM were famous in the 1970's and 1980's for pushing people around and poor corporate relations.  They could get away with it because there were few alternatives.  Netscape were the same in the early years of the web, and Facebook look like they may have a similar problem.  On the few occassions I've met with people from IBM/Netscape/Google etc I've found that they seem to think their dominating position is due to the fact they're all special (superior) human beings, and that their future is therefore assured.  In reality they have domination because they had the right products at the right time, and their future is only assured while the market needs remain the same and while there are no alternatives.  There's no such thing as loyalty here.  The instant there's an alternative which offers better relations, people will grab it.  Getting to the top is much, much easier than staying there.

Long term survival in business is not about products, it's about relationships.

Let's be clear - HTML 5 is great and will make many things much easier.  However, HTML 5 is not finished, and is in many respects years from completion.

STAGES IN HTML DEVELOPMENT

From conception to completion, all versions of HTML5 have to go through a distinct set of stages:

1. Working Draft: After first development by a private group of W3C members, the standard becomes a Working Draft.  This means it is open to comment and suggestions for changes from almost everyone.  Most versions of HTML have undergone huge changes at this stage.  Standards can remain at Working Draft stage for literally years, until some form of general consensus is achieved about how it should look.
2. Candidate Recommendation: At this stage, the W3C group creating the standard are happy it does what it should and that there are no serious glitches in it. At this stage the focus is on feedback from developers about how practical - how implementable - the standard is.   At this stage most significant features are locked, but the design of those features can still change due to feedback from implementors.  Minor aspects, such as syntax, may still change.  Even changes in such minor aspects can kill any code developed for earlier versions.  In other words, at Candidate Recommendation stage, it is still not safe to implement.  Candidate Recommendation stage does not usually last as long as Working Draft stage, but can still take a year or so.
3. Proposed Recommendation:  At this stage, the document has been submitted to the W3C Advisory Council for final approval on the basis everyone is happy with it.   It is very rare, but not impossible, to see any significant changes at this stage.  This stage can take a few months to a year.  The primary concern, as far as I can see, is whether the standard will conflict with other W3C standards in development.
4. W3C Recommendation: The standard is now endorsed by the W3C and is safe to use.

CURRENT STATE OF HTML5

W3C started work on HTML5 in June 2004. As of June 2010 it is has reached Working Draft stage.  This means it has been published for peer review.  Comments by almost anyone will be accepted and may be implemented, so the final version may be very different from the current version.  Previous versions of HTML and some XML dialects have, in my experience, emerged very different from their working draft versions.

This means if you work with HTML 5 now, it is almost certain your developments will not work when HTML5 is finished.

How long will HTML5 remain in Working Draft? Well, it's been in Working Draft for over TWO YEARS, since Jan 2008, and it's running late.  It was supposed to be in Working Draft by mid 2007, and reach Candidate Recommendation stage by 2009.  W3C say they now hope to have it to Candidate Recommendation by the end of this year, but they're always over-optimistic.

Ian Hickson (of Google) is editor of the HTML5 specification.  He says he expects HTML5 to reach the Candidate Recommendation stage during 2012.  He has been quoted as saying he does not expect HTML5 to become a W3C Recommendation until 2022, but I haven't been able to confirm that.  Even so, let's be clear - people won't be able to start providing feedback about how to make HTML5 actually work for another two years, so I can't see it being finished before 2014.

Some people have said that parts are "fairly stable" and can be used now, but "fairly" is not "completely" and means any HTML5 development is risky.  Here's the situation as Ian described it;

"People are starting to use parts of HTML5 today. It depends really on what fraction of browsers you want to target. If you only care about Opera, Firefox, Safari, and Chrome, you can use the “canvas” element today without any trouble… but IE doesn’t support it. If you only care about IE, you can use onhashchange… but the other browsers don’t support it. I would give it a few years before the majority of the features are implemented everywhere, but who knows."

(SOURCE: The Chrome Source)

Do you really want to return to the days of having to detect the browser before sending out browser-specific code, all the while knowing that at some unpredictable point in the future you'll probably have to rewrite it all?

HTML5 is a great idea, but it's still just an idea. It's not a standard, it's not finished, and - in my opinion - using it now is just plain stupid.

The essence of Andy's article is "If you could create links or connections between all of human experience living on this earth, you would have an index making Google, Microsoft, Yandex, and Baidu combined look tiny."

This is a well considered article, but it proposes the most nightmarish totalitarian horror show, something the world would never accept.  In order to compile the information described about people, we would all have to accept that we had absolutely no privacy at all.

I don't want Google, or anyone else, to "capture all of the shared human experience" - there are times when I don't everyone to know what I said in an email, or even who I sent emails to.  Most people do not want the world to know everything they read, everything they purchased, every site they visited and what they did there, where they drove to, what they eat, what clothes they bought and where from and what they cost etc etc.  Google has already strayed over the line here and is the subject of criminal investigations in many countries, and it hasn't come anywhere near this level of intrusion.

Andy's article is thoughtful, and (I suspect) represents a view which is widely held - that the more data held about people, the better the world will be.  This is a naive view of people with too much trust in corporations and governments, often typical of (no offence) US citizens.  In Europe we've had a history of watching democracies turn into nasty dictatorships (Hitler was voted into power, as was Mussolini), and we've seen corporations act in similarly nasty ways, for example happily using slave labour provided by the Nazis.  Americans have been blessed by avoiding this, but it does make them naively trusting.

In Andy's world, many corporations and governments would have to exchange data about people, and that data would have to be complete - everything you do from the moment you leave your home would need to be tracked (it's all in a public space) and most of what you do at home would need to be tracked as well.  No one company could do this for the entire globe or cover everything done everywhere by a single person, nor would one company be able to provide all this data to everyone.  So different organisations would have to exchange their data.  We wouldn't know what was held or who was using it for what purpose.  Errors in the data could destroy your life without you even knowing - you've only got to look at what happens to people when their credit history gets mistakes in it to see that.

People in the online information business - digital marketeers, search experts, etc - need to recognise the world wants, and needs, limits on what data can be gathered, how it is stored, who by, and what is done with it.  More data is not always better.  If the data gathering business does not impose it's own limits, it will inevitably clash with limits imposed by social custom, personal preference, and law.

No body wants to live in a goldfish bowl.

Being a survey/panel-based system from volunteers, I'm always slightly dubious about opt-in bias within the Comscore system.  I suspect less tech-savvy people are under-represented in Comscore, nevertheless, their data should be valid for trends, if not for actual numbers.

The key point which should concern marketing people is that Google continues to loose market-share, albeit slowly.  The main winners are Microsoft's Bing and Yahoo.  Google has been loosing about 1% per month, though this drop is slowing. While Microsoft/Yahoo have been picking up some of their increased share from Google, some has also been taken from peripheral players like Ask and AOL.

Looking at market share is deceptive because the absolute number of searches is also increasing, a trend which is likely to continue for some years.  When you look at the absolute number of searches, they show a very different story.  Google searches grew by 2%, while both Microsoft and Yahoo searches grew by 6%.  If they continue at that rate, combined Microsoft/Yahoo searches will exceed Google's by the end of 2011, and Yahoo searches alone will exceed Google in Feb 2013.

While many will consider the idea that Google may not always be market leader incomprehensible, it's worth remembering the succession of search engines which have led, then lost market leadership: Yahoo, Lycos, AltaVista, Hotbot.  Some of these don't even exist anymore, but they were all once many times bigger than Google.

While Google has over 60% marketshare at the moment, Yahoo/Microsoft are 2/3 the size of Google, with around 40% between them.  That's not an insurmountable gap.

Some may wonder why I treat Yahoo/Microsoft as a combined entity instead of two competitors?  Yahoo announced some time ago they would be moving to the Bing search engine from Microsoft.  In the long run I expect Microsoft to absorb Yahoo.  Microsoft's motto has always been "embrace, extend, extinguish."  Once Yahoo are fully committed to Microsoft's Bing, they've lost technical independence.  Microsoft will then control the future of Yahoo, and it's their duty to their shareholders to absorb it once that happens, as they have to many others in the past.

The UK has just rushed through a very comprehensive set of laws about the web called the Digital Economy Bill.  It covers mainly privacy and copyright issues online.  It has received a great deal of resistance from ISP's and other web professionals because they see it as an attempt by old industries, such as newspapers, to restrict what can be done on the web.  For example, during the draft stage there was a clause stating that linking to someone's site without prior permission was violation of copyright on their web address.  I know some of the people involved in drafting the bill and understand that particular clause came directly from a particular newspaper, owned by a certain Rupert Murdoch.  Luckily that clause was dropped.

The current concern headlines are that, under the new act, people can be banned from the web for life on suspicion of copyright infringement, and must prove their innocence.  This is, of course, contrary to the principle of innocent until proven guilty.  How someone without detailed tech knowledge would prove they had not done something is beyond me...

However, the situation is not as bleak as portrayed in the press.  The act allows ISP's to specify T&C's in this area.  They can stipulate:-
"that a right or obligation does not apply in relation to a copyright owner unless the owner has made arrangements with an internet service provider regarding— 
(a) the number of copyright infringement reports that the owner may make to the provider within a particular period; and
(b) payment in advance of a contribution towards meeting costs incurred by the provider."

In other words, if you want to make claims of copyright violation, you may need to register with the ISP in advance, and pay a fee of their choosing in advance.

Furthermore, the act allows the ISP to write these T&C's such that :
  "(a) rights and obligations do not apply in relation to an internet service provider unless the number of copyright infringement reports the provider receives within a particular period reaches a threshold set;
and
  (b) if the threshold is reached, rights or obligations apply with effect from the date when it is reached or from a later time."

In other words, the ISP can set a minimum number of copyright violations which must occur before it will take action, and violations before the day the threshold is reached don't have to be investigated.

One major ISP, Talk Talk, have said they won't comply with the bill.  However, if they simply set a minimum of 1 billion violations before investigation, and a fee of a few hundred million for investigation, I doubt they would get many copyright complaints.  Thus a clever ISP can write T&C's which make any concerns about this bill irrelevant.

Google Analytics have announced they will offer people the ability to opt-out from Google Analytics in the next few weeks. People will take this option via a setting in their browser.

There are more questions than answers about how this will work. In particular, how this will impact Adword (PPC) reporting? We have no way of knowing how many people will opt-out, but it could be the majority in the long term. It depends on how easy it is to use the opt-out and how much publicity it gets.

Google's Announcement in full: Google Announces As an enterprise-class web analytics solution, Google Analytics not only provides site owners with information on their website traffic and marketing effectiveness, it also does so with high regard for protecting user data privacy. Over the past year, we have been exploring ways to offer users more choice on how their data is collected by Google Analytics. We concluded that the best approach would be to develop a global browser based plug-in to allow users to opt out of being tracked by Google Analytics. Our engineers are now hard at work finalizing and testing this opt-out functionality. We look forward to make it globally available to our users in the coming weeks. Posted by Amy Chang, Group Product Manager, Google Analytics

There are a number of good blogs which analyse this issue:

• Andy Beal calls it hypocritical and disasterous.
• Joe Stanhope thinks its a non-issue, but that all analytics vendors will follow.
• Patricio Robles also thinks it's a non-issue.
• Eric Peters suggests that Google are merely doing it merely to meet US government requires so Google Analytics can be used on federal websites.  I suspect he may be correct.

My take is that it is likely that few people will opt-out in the short term.  However, I think it is dangerous to take the issue too lightly.  3rd-party cookie cutting was once a rare phenomena, but anti-virus and security software starting making it easy, or even the default action, on enough browsers for it to noticeably effect stats.

I notice a tone in most articles about this issue of treating people with privacy concerns as either mild nut-jobs or not target market.  I think this is a premature judgement. Prior to 9/11 online privacy was the #1 concern of US citizens in several polls.  Without a compromising attitude in marketing circles, which recognises that rational people may have some legitimate privacy concerns, the privacy field is left wide open to those who would ban all tracking. I think Google's move is part of an inevitable trend as follows:

1) Opt-outs will inevitably become part of every tracking system.  
2) Knowledge of how to opt-out will grow.
3) The desire to opt-out will grow.  If people can't fulfil that desire themselves, software providers will offer it as a service - capitalism always meets a market need
4) Eventually the number of opt-outs will begin to affect stats in a way which impacts on ROI
5) At that stage websites and tracking systems will begin to develop mechanisms for addressing privacy concerns in a more responsive manner
6) We will reach a stage of mutual respect and understanding in which there is general community concensus on what constitutes legitimate tracking and reasonable opt-outs.  In some place this will be enshrined in law.

The speed at which all this happens depends largely on the actions of people like you.  A few good news horror stories (accurate or not) about the dangers of online tracking, followed by a bad response by the marketing community, could accelerate opt-outs dramatically.  Better privacy policies, decent consumer information (rather than merely calling concerned citizens wierdos), and universal adoption of P3P, could dramatically slow things.

I think just ignoring people's concerns is short-sighted.  Online privacy is a growing social and government concern.  When business ignores a growing issue, it usually finds the solution goes against it.

• 78% believe that if they already subscribe to a newspaper, magazine, radio or television service they should be able to use its online content for free.
• 71% say online content of any kind will have to be considerably better than what is currently free before they will pay for it.
• 79% would no longer use a web site that charges them, presuming they can find the same information at no cost.
• People are ambivalent about whether the quality of online content would suffer if people could not charge for it —34% think so, 30% think not; 36% have no idea.
• 62% believe that once they purchase content, it should be theirs to copy or share with whomever they want.

The last point is, I think, the critical one.  People have no real sympathy with copyright law.  They don't see purchasing content as buying access to information.  They see it just the same as buying a product, once they own it, they can do what they like with it.  Since this will inevitably include copying it into locations on the web where others can easily help themselves (and republish) I am forced to conclude you simply cannot restrict content online.  The best you can hope for is to sell it to the few who are willing to pay for getting it first.  The same thing happens with currency exchange rate information.  You can get it for free - 30 minutes late.  If you want it up-to-the moment, you'll have to pay for it (and pay big).  Currency dealers can't operate on a 30-min lag, so they pay.  For the rest of us, 30-min is ok, so we won't.  The trouble is, how much news or written comment is that urgent? In my mind, trying to restrict newspaper or magazine content to subscribers is trying to force the old-world business model onto the web, and it won't fit.  I really don't care if many newspapers go out of business - if they can't adapt they deserve to die - it's called capitalism and that's how it works.  Eventually bright people will discover a new business model which encompasses both online and offline publication.  They'll get first mover advantage, then the rest of the newspaper world will follow...