Why data is your most powerful tool

24 December 2014 -

Don’t let the numbers confuse you – just let them tell you a story

Will Dean

“Manager wanted. Humans need not apply.”

Is that a Gumtree advert from the near future of your industry?

We stand, apparently, on the brink of a robotic world. As Uber shreds up taxi industries around the world, one imagines it won’t be long before another company – well, Google – launches a wave of driverless taxis, ferrying passengers around London or San Francisco with nary an awkward chat about politics.

And surely the world’s most systematic retailer – Amazon – will embrace robotic shelf pickers at the first chance? Even the most human-seeming professions aren’t immune. A Chicago company called Narrative Science is training computers to write news stories. Their current speciality is conjuring up match reports for junior baseball games from scorecards. This can’t end well…

But surely managing people isn’t a job that can be handed over to our new robot overlords? There’s a key word there, “people”. Yours is an industry based on how humans interact and improve, right? Well, perhaps.

Make no mistake, this is the era of big data. If you work in an industry that can quantify output, and thus, performance, it’s likely that many of your company’s decisions will already be made using data. Who’s making the most sales at an estate agents? Who makes the most efficient use of an expensive drug? Who’s giving us the most value for the salary we pay them?

That’s not new. It’s just that now it’s an easier thing to a) look at and b) to analyse. Even once-subjective industries like my own – journalism – are quantifiable. A news story about corruption in a local police force may have much more merit than a video of a sneezing cat. But you just see which one gets more clicks online.

As the latter example proves, this isn’t necessarily a good thing. We’re a long way from the world in which people got jobs based on the power of their school tie and handshake, but gut instincts still play a key role in how people are managed.

There are many reasons why our intuition is often wrong. But it’s sometimes correct, too. The numbers may point to investing in X, but the better bet may be Y for unquantifiable reasons. Or someone with a first from Cambridge University might be as effective a team member as someone with a 2:1 from Cardiff University.

Malcolm Gladwell’s bestseller, Blink: The Power of Thinking Without Thinking, is perhaps the most famous study of the importance of human instincts and how we judge situations within fractions of a second. Other business scientists have also written about intuition as one of the most powerful tools for managers. A 2007 Massachusetts Institute of Technology (MIT) study, written in the context of the rise of analytics, looked at the relationship between intuition and senior executives. The report suggested that a mixture of experience, networks, emotional intelligence, tolerance and curiosity inform the “gut sense” that can go into making good decisions.

So, with that in mind, it’s unlikely that HAL 9000 is going to be gunning for your job in the next few decades at least. However good automation gets, humans remain key to interpreting data.

But it would be the foolhardy executive who doesn’t at least try to ally data and technology with their own skills and experience.

Duncan J Watts is a researcher at Microsoft and the author of Everything Is Obvious: How Common Sense Fails Us (see review below). Watts believes that business leaders need to combine experience with a scientific approach to succeed in their role. He outlined this for the MIT Technology Review in January. “I believe that leaders across a wide range of contexts could benefit from a scientific mindset toward decision-making,” he says. “A scientific mindset takes as its inspiration the scientific method, which at its core is a recipe for learning about the world in a systematic, replicable way: start with some general questions based on your experience; form a hypothesis that would resolve the puzzle and that also generates a testable prediction; gather data to test your prediction; and finally, evaluate your hypothesis relative to competing hypotheses.”

It’s a simple premise. Treat decision making as a science. It’s one that is already taught to management students, but the day-to-day realities of trying to operate in a scientific way in an office filled with people mean that intuition is often the first tool at hand.

It’s at this intersection between human and computer that technology is beginning to play a larger part.

The giant tech companies, who harvest much of our data, are clearly – even to outsiders – making business decisions based on giant data sets. Be that Google refining search results to a users’ online past or Amazon recommending you a Tom Wolfe book. But what if you’re not in Big Tech? Or your skillset doesn’t include being a quantitative analyst?

Other Silicon Valley firms are working on becoming the company that can help ally human and artificial intelligence. The most notable, Palantir, was founded by PayPal employees who discovered that, while hunting fraud, automated systems could only catch around four-fifths of suspicious transactions. It took humans, allied with the data, to catch more sophisticated criminals.

The technology they’ve designed takes swathes of data from multiple sources, spots patterns and creates visual aides to help humans make decisions. A video on the firm’s website shows how they taught aid organisations, in the wake of Hurricane Sandy, the most effective forms of disaster relief. It’s also rumoured that Palantir’s work, using the same methods for the CIA, led to the capture of Osama bin Laden.

Others are following in Palantir’s wake – OpenGov aims to help local governments plan spending decisions based on computer-aided number crunching and visualisations, while RelateIQ is an app that takes an entire team’s data (emails, calendar appointments) and uses it to increase efficiency and communication, so our estate agent manager, say, would know where a colleague’s sale was up to and could act accordingly.

These three are all American, but they do give you an idea of how technology will soon help most human decision making – everywhere – in the near future.

TECH IT AWAY

Will’s top picks in the big-data field...

Everything is obvious: How Common Sense Fails, by Duncan J Watts, £6.99

An unusual choice for our tech reviews, but this book is an excellent riposte to those who back gut instinct in making decisions. Common sense, Watts explains, is a kind of make-believe backwards, engineered to confirm our biases. The only way to really understand how we behave and make the best decisions is through the scientific study of behaviour that modern technology now allows.

Rating: 4/5

Statistics: Making Sense of Data, Free (www.coursera.org)

This free, 47-week course from the University of Toronto offers an explainer in how to use and analyse data. It’s a commitment, but requires only six to eight hours of study a week and may help to improve your decision-making skills (and studies have found that companies in the top third of their industry that made use of data-driven decision-making were more productive and profitable).

Rating 5/5

Palantir Gotham (www.palantir.com)

Palantir’s software is a substantial business investment (think upwards of £500,000), but as a market leader in providing visual representation of information for humans to compute and act upon, it is steadily finding a market in aiding companies in the financial, pharmaceutical and legal industries.

Rating 5/5