Beware the data-driven organization
Data can help you make some decisions, but most are beyond data’s reach — and these happen to be the most important ones
Data is all the rage, and it’s been like this for a long time. Good organizations are “data-driven.” Objectives are clearly stated, outcomes are measured, and development proceeds through many carefully crafted tests. Users are interviewed at each stage, of course, and product managers are encouraged to listen to actual customers and empathize with them.
Doesn’t that sound good? It should! I agree with most of that! I’ve written about most of it. OKR? I love them! Testing as you go? Sure. Hell, I even use numbers to prioritize hard-to-prioritize things.
If you don’t use all of those methods, you’re betting everything on your gut instinct, and… well, if you’re anything like me, your gut instinct can be very, very wrong. Testing and verifying your progress with data will safeguard you from some of the most significant risks in product development.
But there’s a limit to what data and users can tell you.
I’m not the first one to say that, by the way. Rick Rubin said it in an interview:
“Rick Rubin: The audience comes last.
Anderson Cooper: How can that be?
Rick Rubin: Well, the audience doesn’t know what they want. The audience only knows what’s come before.”
Noel Gallagher had spoken about it in similar, albeit more colorful, terms¹:
“Fuck the customer. The customer doesn’t know what he wants. You fucking give it to him, and he likes it.”
It’s a principle that was perhaps most succinctly put by Henry Ford when he said:
“If I had asked people what they wanted, they would have said faster horses.”
So what gives? Are Rick Rubin, Noel Gallagher, and Henry Ford wrong? Were their perspectives relics of another time? Or is our passion for data misplaced?
No one is wrong, of course! It’s just that not all questions can be solved by data — only a small subset can — and it’s crucial to be able to tell which. Fortunately, it’s pretty easy! Ready for the big insight?
Data can help you choose the best among a finite number of options.
That’s it. There’s no more or less to it. But let’s break it down.
Where data can help
Think of occasions where you have a finite number of discrete options. What comes to mind? I assume the classic examples of A/B testing: Should this button be on the left or right side? Should it be green or blue? Should we show this banner or not?
These are simple things where a few options exist, and you can compare them with one another with limited confounding factors. By all means, use data in those cases. A/B test the various options and go for the one with the higher performance. You can even use data to choose a shade of blue and make $200m doing so²!
The standard A/B test. Image credit: Maxime Lorant — source
That’s all well and good. But do you see from this what you cannot do with data?
Where data fails
You’re a smart cookie, so I bet you could, but let me spell it out for the other kids: Data cannot help in situations where the number of options is infinite.
Let’s use a dumb example. You’ve heard of this Shakespeare person, but you feel that his approach to writing sonnets is a bit antiquated. We’re in the 21st century now, so let’s use science to make better poetry. His sonnet #1 has 105 words. Let’s say we only use the 1'000 most common words in the language. There are 10³¹⁵ sonnets of that length. Do I need to spell out how absurdly large that number is? There are around 10⁸⁰ atoms in the whole universe!
So yeah, things get infinite quickly, and when they do, you can no longer just have different cohorts of users seeing different versions of your product and voting for them, because there aren’t enough eyeballs available. There’s an infinite number of ways to handle your checkout flow. There’s an infinite number of ways to handle your login. And there’s a much bigger infinite number of companies you could build. You cannot A/B test your way out of that.
Data may have helped Google pick the right shade of blue, but it wasn’t data that compelled them to make a search engine rather than an airline. Data didn’t lead Steve Jobs to start selling computers rather than pet rocks. Data will let you optimize, but it will not help you create.
Steve Jobs, Sergey Brin, Larry Page, or any other founder you can think of had a vision. They saw a future that was radically different from their present and that they could work to usher in.
Using data to create
Interestingly enough, I know of at least one high-profile case where a company used data to create rather than optimize. Ironically, it happened at Ford in the 1950s, and I’ll quote from Business Adventures, which tells the story:
“Krafve was not the kind of man to envision his objective in a single revelatory flash; instead, he anatomized the styling of the E-Car into a series of laboriously minute decisions — how to shape the fenders, what pattern to use with the chrome, what kind of door handles to put on, and so on and on. If Michelangelo ever added the number of decisions that went into the execution of, say, his “David,” he kept it to himself, but Krafve, an orderly-minded man in an era of orderly-functioning computers, later calculated that in styling the E-Car he and his associates had to make up their minds on no fewer than four thousand occasions. He reasoned at the time that if they arrived at the right yes-or-no choice on every one of those occasions, they ought, in the end, to come up with a stylistically perfect car — or at least a car that would be unique and at the same time familiar. But Krafve concedes today that he found it difficult thus to bend the creative process to the yoke of system, principally because many of the four thousand decisions he made wouldn’t stay put”
The E-Car in question is the Ford Edsel. Considered the costliest consumer product ever created at the time, it bombed spectacularly, which is probably why you never heard about it. And it’s interesting that they quote Michelangelo in this passage because Michelangelo himself didn’t see his craft in this light. He said:
I created a vision of David in my mind and simply carved away everything that was not David.
When (not) to use data
When you create anything meaningful, you will need to make tactical decisions on plenty of occasions. When Michelangelo had to choose the right chisel for a certain task, that was an easy question with a right and a wrong answer. Data can probably help! But you first need to have a clear picture of where you want to go; for this, you will have to rely on your experience, judgment… and vision!
That can seem overwhelming at times. Having two options and an unambiguous dataset that tells you which is better is much more comfortable. But life isn’t like that, and the problems that cannot be solved with data are the most exciting. Douglas Bowman, the lead designer at Google at the time of the 41-shades-of-blue experiment, says:
“Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that. I’ve grown tired of debating such minuscule design decisions. There are more exciting design problems in this world to tackle.”
Leave those decisions to the growth hackers and other Excel warriors. This will leave you ample time to dedicate to building new and exciting stuff.
¹ I can’t for the life of me find a proper source for this one. The closest I get is this broken link, quoted here. It was apparently sometime around 14–15 April 2012 in Coachella
² It’s beyond the scope of this article, but, while this test and the resulting figure are often quoted, it’s important to note that there are many, many caveats with it, to the point that I wouldn’t consider it in any serious discussion of A/B testing
