While reading Jared Diamond’s excellent book on the rise and subsequent global dominance of Eurasian societies Guns, Germs and Steel, I was stopped in my tracks by his chapter on the evolution of technology entitled Necessity’s Mother. Diamond briskly demolishes the commonly-held view that necessity is the mother of invention. In fact he argues that many inventions were developed by curious-minded people or those with a fascination for certain aspects of the technology in question. Rather than satisfying an existing demand, the inventors were the early adopters and consumers were often slow to catch up, only starting to demand the product once it had been in available for a considerable time. We are biased towards focusing on invention-as-problem-solving and so we ignore the fact most people didn’t feel they needed the early forms of phonographs, automobiles, typewriters, cameras or televisions. In fact, I recall similar bafflement and outright hostility to some technologies that have been developed during my own lifetime such as the internet, cellular phones and email.
This phenomenon is of particular interest to me because I work in predictive analytics – a ‘technology’ whose time has finally come, at least if the daily deluge of articles, ads, tweets, conference invites and announcements I receive is to be believed. Stories about predictive analytics are making their way into the mainstream media. We are told a particular police department has started using this new approach to target criminal activity more effectively; that an insurer saved millions of dollars by predicting which claims were fraudulent; that a utility company halved the number of power outages by modelling the hourly risk of asset failure. Faced with growing numbers of stories like this, one might be forgiven for thinking that predictive analytics is technology that has now crossed into the realm of necessity.
But here’s the thing. The same sorts of pronouncements were being made ten, fifteen and even twenty years ago. ‘Oh but come on Mr Q’, I hear you say, ‘this time things are really taking off – we’re finally turning the corner’. Oh yeah? Well maybe you’re right or maybe this ‘corner’ is actually the curvature of the earth. The cold reality is that when it comes to advanced predictive analytics, most data-rich organisations are still a lot more like the Flintstones than they are the Jetsons.
I’ll admit that a growing number of organisations, both commercial and not-for-profit, have adopted more sophisticated analytical approaches in last few years, but these are still the exception rather than the rule. To wit, how come predictive policing isn’t used by all forces? Why do most utilities still rely on calendar-based maintenance programs to keep assets functioning? How come my bank, a major customer of a leading analytics vendor’s software, has offered me the same product every time I’ve logged on to check my account for the last ten years? Is it some experiment in attrition marketing? The paucity of sophisticated analytics in most organisations might lead you to think that they prefer to regard the world as a random mess. Why is that? Let’s examine the possible barriers to the adoption of data-driven technologies that encourage more a forward-looking and rational approach to decision making.
- It’s really expensive – this one falls at the first hurdle because in fact the predictive analytics market is very mature and completely wide open. You can pay through the nose for a fancy decision tree or you can download an entire data mining package for free. Also, compared to the costs of a lot of other IT implementations, the cost of predictive analytics is peanuts. Just look at how much money was being spent during the days of the ‘rush to warehouse’, the early dotcoms, ‘CRM fever’ or even now on ‘No SQL’ solutions and Hadoop-like platforms.
- ‘We don’t have the right data for that sort of thing’ – a common enough belief that usually dwindles to a painfully thin excuse on closer examination. If advanced analytics relied on oodles of clean, consolidated data from multiple sources stored in pristine warehouses you wouldn’t even be reading this.
- ‘We don’t know how to use this stuff’ – fair enough, a lack of experience and knowledge of successful predictive analytics applications is obviously a legitimate barrier although hardly an insurmountable one. Sadly, a quick glance at the curricula of the some of the most sought-after MBA courses available globally reveals a depressing lack of modules with an analytics focus.
- We don’t need it – yet’ – finally, we arrive at a simple and important truth. However much predictive analytics might benefit an organisation, most can function reasonably well without it. At least until their competitors start driving optimised decisions based on deep analytics.
This leaves me thinking: why have any organisations made a significant investment in predictive analytics? In my experience the answer is either a) because due to competitive pressures they had no other choice (e.g. the telco sector) or b) because they happened to employ individuals who had enough vision, clout and chutzpah to see the enormous value in being innovative with data. So if invention is the mother of the necessity, what is the mother of enterprise? I wonder if it’s courage.
