Comscore Bunfight and Adolescent Industries

Quite a bunfight about Web Metrics this weekend, I read it first in SAI:

Comscore is changing its web ratings methodology to include site-side server tracking. In connection with this change, it is charging all sites who want to be measured with the new methodology a $5,000 "set-up" fee ($10,000 for a year's access to the data).

Comscore's logic here is that the new site-side tracking costs money, so sites should pay. We disagree with this. The site-side tracking is a cost required to improve Comscore's product. The change will likely boost traffic for sites that participate, so Comscore is effectively waving the threat of relatively poor traffic metrics in the face of every company that chooses not to pay. We see that as blackmail.

Yesterday, Jason Calacanis called for an industry-wide boycott of Comscore. Comscore investor Fred Wilson and CMO Linda Abraham defended the company. The argument has gotten heated and personal, and many of the points being made have gotten away from whether the setup fee is fair.

Techcrunch and various others weighed in too. Now some of our clients use ComScore, and there is also Alexa, Hitwise, Google Analytics, various server-side packages, Neilsen etc etc. And they all disagree with each other!

Our view is that no system is "accurate" yet. You have to cross correlate across them all at the moment to get a "fix" on one site vs its competitors, and its relative and not absolute. Two years ago Will McInness kicked off MeasurementCamp in London because so many of us were finding this problem - and in fact one of the things we quickly wondered about was are they even measuring the right things, never mind measuring things right. It is a b it frustrating that 2 years later not much more light has dawned.

But its not surprising. This is an industry in its adolescence, still trying to work out the right way to look at itself. Television went through this decades ago and it took a while for enough empirical evidence to come through as to the best rules of thumb. And liek all adolescents growing up, we expect to see histrionics every so often

As to whether Comscore is right to charge for server-side monitoring, two thoughts:

(i) The market will ultimately decide whether paying $5,000 for a different number to the one you get for free is better.
(ii) If it becomes clear that companies who pay for server side monitoring have higher reported traffic than those that don't, ComScore risks tarnishing its currency, or at least having to report those who are and aren't server side.

With the bigger websites, you can sort of calibrate them against each other anyway, and I would argue thats as good an approach as any right now - ie its pointless trying to talk about 2 decimal place output when the input is a finger in the air.