How accurate is Alexa exactly?

So, we can track our traffic on Alexa and our own Blogsite's analytics...and its a tale of two different Blogs.

According to our own site, our traffic is rising by about 20% per month (excepting April which was flat owing to Easter week).

According to Alexa we have descended from a New Year high to the sloughs of "not in the top 100,000" blogs....these guys have had a similar experience and show similar graphs to ours:

What gives?

While snouting around I found this:

One of the two breakdowns of traffic is the Alexa Reach Rank, which purports to show the number of users per 1 million that access the site in a given period. Alexa provides reach rank data for the day, previous week, and previous three months.

It is the daily Reach Rank that provides an easy method of determining the number of users. In 1908, the physicist Robert Millikan performed a classic experiment to measure the charge of the electron, one of the fundamental physical constants in the universe. He did so by determining at what charges individual oil drops would "float" in an electrical field. Those charges at which particles were observed to float were all multiples of the charge of a single electron. For this insight, Millikan received the Nobel Prize in Physics.

Millikan's methodology is easy to use to tell the size of the Alexa sample on a given day. Just look at the daily reach rank for a large number of sites with ranks below 20,000. Those sites that were not visited in a given day have no Reach Rank for that day. A large number of sites will be visited only once, or twice. Find the reach rank of sites that are visited just once. Then divide one million by that number. The result is the Alexa sample size for a given day.

On a recent day I checked, and found that the Alexa sample size was 180,000. This makes it likely that, if you are a heavy Alexa Toolbar user, and you have a site of your own, your own browsing on your own site and others you pay attention to will bias Alexa's results significantly.

I could provide some personal anecdotal information about this, but I don't want to identify my site here and cause my results to become biased. Instead, I am linking my name to a good page on the Millikan Oil Drop Experiment. My advice is, if you need accurate ranking results, keep the Alexa browser off your own computer. Comments are welcome.

Not the most ringing of endorsements......and that was 2003. Was there anything more recent?

Turns out 4 years later the song is much the same...2007 and its a note from seologs.com

According to Alexa, as of right now, out of all the sites on the web, seologs.com is the 7,306th most popular site. That’s pretty impressive if I do say so myself, but (that’s right, there’s a but, and it’s a really huge one), unfortunately, it’s no where near accurate. Not even close. The reason being is that users who visit this site (those interested in SEO and SEM) are much more likely to have the Alexa tracker installed on their browsers. (Alexa tracking can be installed in Firefox extensions as well as other toolbars).

So just how inflated is that ranking of 7,306 for seologs.com? Well, another site I have access to, but is nowhere close to the topic of SEO (can’t name specifically), has an Alexa rank of about 15,000. That’s a lot lower than seologs right? Well that site gets an average of about 45,000 unique visitors per day (that’s right, per day), so you would expect that seologs.com would get at least that many daily visits right? I wish. Seologs.com gets an average of 1,700 unique visitors per day.

So does this mean that Alexa rank is completely useless? Not really. In fact it can be extremely useful, but (*** another big but) in my experience, only if the 3 month average rank is 100,000 or higher. Anything below that is just too easy to fake with just one person.

Bluddyell...our average is c 900..( we loves ya, mwaaah ) yet our Alexa ranking is in the 700,000's right now - thats some power law !!!!

The jaw droppeth...how the FAQ can anyone take a system this flawed with anything other than an extremely large pinch of salt?

Seems like its OK for big sites as they say, but it is extraordinary that "The Web" is still to an extent reliant on such poor analysis tools...