Organisations hold information, lots of it. Often it’s all over the place and sometimes its not acknowledged as being useful but there is always lots of data from customer interaction and transaction information to production line or stocking information. The trick is learning how to use past data to do better in the future. Organisations are a lot bigger than they once were with more and more employees who have a shorter and shorter tenure. These changes have significant implications for how analytics is done. Where data volumes are small and employees are knowledgeable it’s often possible to get a lot of insight from your data just by eyeballing it. However as data volumes grow that’s no longer practical.
Spotting fraud on tax returns
It’s a situation that I encountered a while back when I did some work at the Inland Revenue. They wanted to spot patterns in their data to help identify fraud on tax returns. They had a small team of inspectors who had mostly been there for 30 or 40 years, all of whom had been working on their own particular niche of tax problems for almost their entire career. Over this time they had developed an eye for patterns, or what the Unit Director referred to as ‘the tax inspector’s nose’. When the amount of paperwork was small and the Revenue had experts in the field who were extremely experienced in that business, they were able to detect outliers and unusual patterns by eye. They knew what to expect, they knew what normal was, they learned about oddities which skewed results and because of this they were able to predict ‘by eye’ which tax returns were likely to be fraudulent with a high degree of success.
However, the number of tax returns being submitted was growing rapidly, new staff (who did not yet have the tax inspector’s nose) were being hired into the department and the old hands who did have the nose were retiring. No patterns were being detected anymore. More fraud was slipping through with no safety net to stop it. The Revenue recognised that they needed some kind of automated solution that would enable them to apply a ‘virtual’ tax inspector’s nose to the vast volumes of data they were collecting.
How can you analyse growing data volumes?
It’s a problem common for many organisations. When data volumes are small, experienced staff can get a fairly good understanding of the situation by eye. Someone who really knows the business can spot changes and unusual patterns in the data using the equivalent of the tax inspector’s nose. However data volumes are growing rapidly across the board and this way of detecting changes and patterns is no longer feasible for many organisations. In a big data world individuals cannot possibly look at all the data, or review transactions by eye. Rapidly lowering technology costs mean most organisations now have more data than they know what to do with. In principle more data is a good thing, as long as it is still being analysed. But who is looking for the patterns, the outliers, the opportunities or threats in all that data once the process becomes automated?
The first step moving from looking over the data by eye towards automated data analysis is often to move towards a system whereby the business intelligence team review business performance at the end of the month or quarter. This is fine as far as it goes, but the intelligence gained from this kind of exercise often comes too late to do anything genuinely useful with it, like making an offer to an individual customer, or put a halt on a transaction. And it’s not always easy to spot a nuanced pattern from within a ‘slice and dice’ business intelligence report. Many organisations that say they’re doing predictive analytics aren’t really. They’re slicing and dicing data retrospectively – in the process they’re certainly learning more about what has happened in their business, but they’re not learning much about what is going to happen. To really get value from large datasets you need to move beyond business intelligence reporting towards true predictive analytics.
Predictive analytics supplements the tax inspector’s nose
In reality only a small proportion of organisations are really truly adopting predictive analytics. More and more are beginning on the journey but it’s still a huge differentiating opportunity for businesses looking to make a clever, cost effective change. Organisations who understand the value of their data are now implementing predictive analytics. This generally means investing in a software ‘workbench’ which enables an analytics to get a complete view of the organisation’s data. Pattern detection techniques can be used to look for a great range of patterns, more nuanced patterns and more complex patterns than could generally be spotted by the tax inspector’s nose. Cases (such as tax returns, or credit card applications or whatever it is that the organisation is interested in) can either be processed in bulk and examined for patterns, or processed one-by-one and examined for anomalies. The advantage of this approach is that new anomalies can be detected as new techniques for tax avoidance or fraud are used – something that the tax inspector’s nose would not always have picked up. In examples like these organisations can use predictive analytics to enhance the knowledge and understanding in the heads of their experienced employees and can very quickly put this boosted insight into use across the organisation.
