Big data puts pressure on business intelligence

Because of the way they are deployed, less than 10% of employees regularly use BI tools. Forrester Research puts the fraction at between 5% and 9% and of that user base, anecdotes abound of daily reports taking several days just to look through.

BI is plagued with two different kinds of backlogs: firstly, the almost unmentionable problem of implementation and integration delays and secondly, the human backlog of painstakingly searching through reports looking for insight and trends.

Too much data

"BI works exceptionally well at turning database rows into useful insight," says Anton van Heerden, GM at Altech ISIS. "But the explosion of data and the limited deployment among employees has put pressure on those users who have to try and make sense of the reports."

He does not go far as to say that big data has damaged BI beyond repair but he does think that companies need a new approach when thinking of mining big data.

"Today's BI deployments need a human, preferably with some kind of domain expertise, looking at the reports and seeing what to make of them. However, as data scales up, simply throwing hardware and faster networks at the problem will not solve the underlying issue: there is too much data. Even automation and reliance on industry-specific solutions can only go so far."

Companies should be looking to formally trained statistical analysis of the output from automated intelligent systems built into the back-end infrastructure.

Getting the right results is expensive

"However, such analysts with the proper qualifications are hard to come by - and expensive. In addition, no one quite knows yet where to pay the complexity costs: do you make sure all your sources of information are clean first? Do you do it at the extract-transform-load (ETL) stage or do you leave it unstructured and introduce complexity in the back-end to make sense of it? Different industries will have different answers to these questions."

The challenge needs an entirely new approach and it must be company-wide and driven from the top down.

"At the company, we have found that executive buy-in of a big data strategy is essential. Changing company culture - which is what it comes down to - is useless to attempt without the backing of at least one C-level executive. Secondly, companies need to see how their data is siloed. If there is too much concentration of ownership, then it needs to be flattened as much as possible. Not everyone can - or should - have a statistics degree in the company, but everyone should be able to back up their day-to-day decisions with data; that means promoting a data culture where everyone has access to what they need and can trust the results. Finally, everyone should be encouraged to look at outside data for insight: public records, market research or even sensor information."