How accurate is Google AdWords billing?
And how does Google handle possible errors?
According to Google's income statement to investors (on the Google website), Google earn over $10 billion from advertising each year. My estimate is that over 95 percent of this comes from AdWords; other sources claim 98 percent. This means Google's AdWords counting technology was responsible for at least $9.5 billion per year.
Given that such a vast amount of our money goes to Google on the say-so of a single web analytics system, it's extremely important to know how accurate that technology is.
It is well established that it is impossible for any web analytics technology to be 100 percent accurate. Most systems are accurate to within 95 percent. There are many reasons for this, which I've covered in earlier articles (see "Things That Throw Your Stats").
I have asked Google how accurate they consider the system to be. So far they have declined to provide any answer, so they haven't actually claimed 100 percent accuracy. Let's give Google the benefit of the doubt and consider the possibility that AdWords' click count is 99 percent accurate. This would make it the most accurate web analytics technology on earth by a long stretch. It would also mean it had mis-billed over $100 million last year. If it was 95 percent accurate, then it mis-billed $500 million last year.
My personal experience is that the accuracy of the billing system in Google AdWords varies. I run PPC campaigns for clients, and I always check Google's numbers with what our own software, InSite, says. Since we always run a unique tracking code on every Google ad, this is fairly easy. I have disputed numbers with Google several times over the last few years.
I'm not talking about click fraud here. Click fraud disputes do not involve a disagreement about the volume of traffic Google has relayed to your site. Click-fraud involves a disagreement about the validity of that traffic. Both sides agree on the numbers; they just disagree on whether all of it should be paid for.
The situations I'm concerned with are where Google AdWords claim to have sent more visitors to my site than I can verify for myself.
It's unreasonable to expect the numbers to match exactly. No two web analytics systems will ever agree perfectly. However, my experience of other systems suggests the disparity should not be too great. Bing and Yahoo PPC numbers consistently match mine to within two percent. Google AdWords usually match mine to within three percent. I think two percent is about as close as is possible, and three, or even four, percent is acceptable.
However, I do get concerned with the really big disparities. These are rare, but given the amount of money moving through AdWords, even rare disparities add up to a great deal of money. I have personally seen Google AdWords count 25 percent more traffic than I could verify on several occasions. I consistently see Google AdWords billing for 10% - 12.5% more traffic than I can verify in most client sites.
One extreme case of apparant over-charging by Google Adwords:
I recently ran a small test campaign for a new business. In the first two weeks Google AdWords claimed (and billed for) 96 clicks, but we only saw 11. When I queried this Google's AdWord Response Team said they checked and the numbers were correct. They didn't tell me how they'd checked the numbers, but they did point out I had a history of disputing their counts. They said this indicated my software was probably faulty. They invited me to use Google Analytics, which was, in their words, "one of the most powerful web analytics products on the market."
I was not convinced my software was faulty since it has always matched Yahoo and Bing, and usually matches Google AdWords, but if Google wanted to use their own system to check AdWords numbers I was willing to give it a try. I made sure to add all the tracking options Google recommended to ensure the Google Analytics count was as accurate as possible.
Having done everything Google asked of me -- by the book -- I waited a week for the numbers to cook. At the end of that time, Google Analytics told me I had 14 AdWord visits while the AdWords billing system itself claimed 59. My own software, InSite, also said 14 visits, the same count as Google Analytics.
On other metrics, such as traffic from native listings, or other sites, InSite matched Google Analytics to within three percent. I think this validates the accuracy of both InSite and Google Analytics. So the system that Google told me to use was telling me that Google AdWords were over-counting. I now had two different products telling me that Google AdWords was over-charging, and by a significant margin.
This article is not intended to be a rant against Google. I don't want to condemn them just because their AdWords system has made a mistake. I'm not even going to criticize them for insisting I use their software to check. They know nothing about mine, so why should they trust it? What is really important is how Google handles situations when a customer believes they have been over-charged. Given the volume of traffic AdWords handles, it would be incredible if the AdWords click-counting system never made a single mistake. However, given the amount of money at stake, it is very important Google handles situations like this in an appropriate manner.
My next step was to report my findings to Google's AdWord Response Team.
Their answer was straight-forward: "For privacy reasons I do not have access to your Analytics data. I've reviewed your account and found that you accrued charges for legitimate clicks while your campaign was active. Therefore, we are unable to offer a refund for these charges."
This raises a number of questions:
How does Google Adwords calculate billing?
Google's AdWord click-count is based on log analysis. When someone (or something) clicks on a Google Ad, this is recorded in a log file on the Google servers. The logs are analyzed to determine the click-count which will be used as the basis for billing. In Google's view, discrepancies between AdWords' click-count and other web analytics systems don't call the accuracy of AdWords into question because AdWords is counting, while web analytics systems are merely estimating numbers. According to Richard Holden, Google's director of product management,
Google does distinguish between "valid clicks" and "invalid clicks." In Google's terminology, a valid click is one you should pay for, and an invalid click is one you don't have to pay for. According to Holden, the only real issue with AdWord numbers is the validity of clicks and Google's attempts to filter out invalid ones.
Holden stated there are two types of invalid clicks-- "non-nefarious clicks, such as my grandmother double-clicking because she thinks that's the thing to do; or nefarious clicks, such as someone clicking on an ad because they're trying to generate costs for an advertiser."
According to Holden there are two types of queries over AdWord numbers. A customer can question the validity of some of the clicks they are being billed for, or they can question AdWord's click-count because their analytics system is reporting different numbers. In both cases, the case is assigned to an individual member of the AdWords Response Team. If someone has an analytics system which is producing different numbers, the Response Team's first step should be, according to Holden, to work through the different technical reasons why this can happen in an effort to educate the customer. If the customer is still unhappy at the end of that process, the Response Team case worker can ask someone in the Invalid Clicks Team to examine the data and verify accuracy.
The only people at Google who can actually check the data behind AdWord's billing are the members of the Invalid Clicks Team. While communication with customers is mainly handled by members of the AdWords Response Team, they cannot actually verify the data themselves; they must escalate the matter to the Invalid Clicks Team. Richard Holden told me that "this team does proactively look at accounts that we get complaints about. They can look at the logs; they can look at the data very closely." When a Response Team member tells you they've verified the data themselves, Holden states what they mean is that they have managed a process in which the Invalid Clicks Team has verified the data. According to Holden, Response Team members are not anxious to advertise to the world that the Invalid Clicks Team exists.
Holden told me the reason AdWords Response Team members cannot look at the data is privacy-- "We take the privacy of both the advertiser and the consumer very seriously. The Invalid Clicks Team has been screened and is cordoned off, so there is a limited set of people who have access to the data."
During the course of this investigation I've spoken to members of Google's AdWords Response Team, Invalid Clicks Team, and to Google Director of Product Management Richard Holden. What comes across most clearly is an absolute rock-solid certainty that Google AdWords is recording clicks with absolute perfect accuracy. It never misses a click, it never double-counts a click. Ever. It never mis-allocates a click to the wrong advertiser. Every AdWords click is logged with absolute precision. The initial log analysis which counts those clicks is absolutely perfect. Given the number of servers and data centers Google operates on, this also requires that either the logs are merged together, or the counts are, and that this merging is absolutely flawless. The data has to be stored in one or more databases, which means it is being written, stored, queried and retrieved without a single glitch. I would imagine Google has systems in place to ensure data integrity, so these have to work perfectly as well. There is a great deal of effort put into checking whether all the clicks should be paid for, but the assumption is always that the initial log analysis is 100 percent perfect. As a result, there is simply no process inside Google to check whether this is the case.
As a mere advertiser, I am not in a position to judge how accurate AdWord's click-count is. This is not a new problem, nor one which is unique to Google. Accuracy of the information which determines billing for ads has been an issue since the first newspapers took their first ads. The accepted solution is to obtain an independent audit by a suitable organization-- an audit bureau. Once advertising commenced on the web, the audit bureaus moved online. Organizations such as Audit Bureau Circulation Electronic (ABCE) and the Internet Audit Bureau (IAB) exist in almost every country of the world. These bodies assess the accuracy of the systems used to count the numbers which underpin billing for online advertising. Most, but not all, follow standards laid down by the Joint Industry Committee for Web Standards.
Google does not have independent audits of its systems. Holden told me he does not believe this is necessary because, as far as he is aware, none of their major competitors are independently audited. Furthermore he does not believe that appropriate standards exist for such auditing.
Google is never wrong
So we have a situation in which Google is absolutely certain that the initial logging and analysis of AdWords clicks (as yet undefined) is 100 percent accurate all the time. The only basis upon which numbers may be investigated is to determine if the clicks which have been billed should have been billed. Finally, as Google's staff will freely tell you, it is a contractual condition of using AdWords that you accept Google's numbers.
Obviously most of Google's advertisers are happy most of the time; otherwise they would not be spending the sums they do. However, my own experience, and that of others I've spoken to, is that complaining rarely brings happiness. When you do complain Google's customer handling skills are nowhere near as perfect as their technology. According to Holden, the initial response to a query should be working through reasons for discrepancies. In my experience this is rarely the case, and I've spoken to enough people to know my experience is not unique. More frequently the initial response is little more than an assurance that the numbers are accurate, and a blunt reminder that it is a condition of using AdWords that you accept Google's count. It usually requires some pushing to get as far as being given a FAQ indicating why your own numbers could be wrong, and real dedication to get things escalated to the Invalid Clicks Team. Most of the people I have spoken to felt Google was arrogant, and that dealing with Google was frustrating.
It may be that part of the problem is lack of communication. You can't phone Google unless you are spending extremely large amounts of money, so you're limited to email. This is not what most people are used to. In any other area of business, an expenditure of (say) $100,000 a year would rate you at least a telephone contact, if not face-to-face meetings, but not with Google.
In addition, there is a lack of transparency to Google's billing. Google declines to explain how its technology works. According to Richard Holden-- "We believe we have unique intellectual property in how we track and ferret out these problems versus our competitors. This is one reason why we don't talk very specifically about how we actually do it, nor do we talk about specific numbers. The more we publish, the more we give the bad actors the ability to act against us. We do understand this is a double-edged sword because it means less people understand that there is not a wide-scale problem."
What this means is that you can't get a detailed breakdown of how your bill was calculated, nor will you ever have direct access to the data from which that bill was compiled, so you can't check for yourself.
Legal opinion on Google Adwords contract
I asked an international lawyer specializing in media and the internet to look into the legal aspects of the way Google handles AdWords disputes. Rupert Grey, of Swan Turton, is one of the world's leading experts on media law. In his opinion, the U.S. and European contracts are probably legal-- just, but there is real confusion around what you're paying for and how you'll be charged.
It's questionable just how accurate AdWords is.
Google believes the core technology is consistently 100 percent accurate. If you don't accept this, they see their task is to show you the error in your ways, not check the accuracy. This, plus other aspects of Google's behavior, leaves many people feeling frustrated. However, with Google earning over $5 million per hour from AdWords, I don't think we'll be seeing much change in the near future.
Google believe they can never be wrong.
The key factor controlling the process of AdWords billing disputes is Google's unquestioning belief that the AdWords click-tracking and billing system is 100 percent perfect. According to Google your system has to be wrong because AdWords click-tracking is the most perfect computer system ever created by the human race. In this sense it is the absolute pinnacle of technical evolution.
In writing this article I have tried very hard to think of any other technology that even vaguely approaches perfection, but I can't. When you consider the amount of data it processes, you realize Google AdWords' click-tracking must be more accurate than an atomic clock! It defies the laws of physics in its perfection. Of course, you just have to take Google's word for it because it's so wonderful they can't tell anyone else anything about how it works to prove their claims.
Google does have a point; even if your analytics system is correctly implemented, some variation in numbers will occur. But if that variation is very large, you can take it from Google-- your system is faulty. Because Google AdWords is absolutely perfect.
If, by now, you still stubbornly refuse to be re-educated as to Google's Absolute Perfection, Google will happily remind you that if you want to use AdWords, Terms & Conditions stipulate you have to accept their numbers anyway. So tough!