On Intel’s website you’ll find a 2012 video that, despite its animated illustrations and cheery score, should scare the living data out of you. It’s called Big Data 101: How Big Data Makes Big Impacts, and contains the following slice of information.

‘Data has been getting bigger for a while now. From the dawn of time to less than a decade ago, mankind generated about 5 exabytes of data. In 2012, global data will grow to 2.7 zettabytes – that’s 500 times more data than all data ever generated prior to 2003. And it’s going to grow three times bigger than that by 2015!’ (‘Exa’ is one followed by 18 zeros while ‘zetta’ has 21).

Data, as the video shows, is getting bigger. Companies across SA and the world are scrambling to get their minds around big data and how to use it.

‘Big data is an all-encompassing term that refers to very large data sets that cannot be processed in acceptable time frames using traditional database technology,’ says Paul Morgan, technical director of business intelligence firm Decision Inc. ‘There is no industry standard as to what encompasses “very large” data sets, and the measure of what constitutes “big” is largely subjective.’

What we do know is that big data is big business. One estimate from global market research company MarketsandMarkets says that the global big data industry was worth $14.87 billion (R158 billion) in 2013, and will grow to over $46.34 billion (R493 billion) by 2018.

Organisations need to start thinking of big data as a business requirement, says Vaughn Naidoo, head of product portfolio, architecture and innovation at Telkom Business’s data centre operations. ‘Companies are faced with growing volumes and increasing complexity of structured and unstructured data sources. Big data is an enabler for business analytics and intelligence. For a business to remain competitive it will not have a choice as to whether or not to participate.’

Linux Warehouse MD Jan-Jan van der Vyver says that big data promises increased profitability for SA companies as businesses will be able to gain information about existing and potential clients. ‘The rainbow nation offers unique challenges and opportunities for better market segmentation and the identification, targeting and value extraction from clients. It has even been shown possible to track, and act upon, client sentiment or their major life changes,’ he says.

But can all that data really be used? Or is big data – like netbooks and Y2K – just big hype?

‘There is a certain amount of hype. Data in vast quantities has been around for many years but the technology to handle this volume adequately, especially in real time, has been expensive and difficult to implement,’ says Morgan.

However, he says: ‘The volume, velocity and variety of data is exploding in this digital age. Maturing technologies and new entrants to the market have made it easier and more cost-effective to access and use this data.’

‘The new generation of big data-centric analytics tools will turn information into more actionable insights for users’

SUREN GOVENDER, ACCENTURE ANALYTICS MD, ACCENTURE DIGITAL

Martin Naude, chief technology officer of software solutions company Entelect says this is already becoming a reality. ‘Many organisations are battling to find new ways to monitor and analyse all of this data on a day-to-day basis using traditional or manual methods. The sheer volume of data created means that it’s virtually impossible to check everything. Most organisations use sample testing. This means that if, hypothetically, they check, say, 30 transactions out of 10 000, and if they are all correct, they assume everything else is correct. I’m sure you can see the potential problem with this,’ he says.

While traditional business-intelligence tools are still useful for generating static reports of data at rest, Suren Govender, managing director of Accenture Digital’s Analytics division, believes that they fall short in the age of big data when data discovery, real-time analysis and visualisation capabilities are important.

Govender says that when the new generation of big data-centric analytics tools are embedded in business intelligence, it can provide early alerts based on patterns and turn information into more actionable insights for users. He calls this ‘issues-driven business intelligence’.

However, understanding that businesses can collect mounds and mounds of data is just the beginning. As with so many ‘next big things’, big data is all about balancing quantity with quality.

Paul Mullon, founder of information management consultancy COR Concepts, says that because storage is perceived as cheap, it’s too easy to gather lots of data. ‘We’re also getting that data from multiple sources, possibly in different areas of the business, so we need data quality controls to make sure that it has got integrity, authenticity and reliability so that we can trust it and [ensure] that it’s accurate.’

Data capturing is therefore becoming even more important, as is data storage. According to analyst firm Gartner about 80% of data within a business is ‘unstructured’, which means it’s not always easy to access. The secret to using big data effectively lies in what Mullon calls ‘information governance’.

‘It’s about identifying all the information. Then it’s about asking the questions. How are we using it? Which business processes is it going to be used in? Where does it need to be stored? In what format do we need to store it? Who’s the information owner or custodian? When does it get destroyed? Who’s responsible for that? How do we keep metadata of the fact that we once had the data, but we’ve now destroyed it as part of a formal business process?’

Mark Hiller, general manager of Lexmark SA, makes a similar point. He says there is still an enormous amount of enterprise information contained in text documents and presentations, audio, graphics, email, video, web pages and other office programmes. ‘Classifying data as unstructured doesn’t mean that it lacks any structure. Rather it means that it is not part of a database or exists in the enterprise relational data model.

‘Even worse, much of our enterprise process exists as unstructured data in the heads of workers and lacks any systematic approach for capture, management, communication, measurement and improvement. When work activities are unstructured, the day-to-day behaviour of workers lacks efficiency and cohesiveness.

‘For many companies, the knee-jerk reaction to this combination is to try and digitise everything in sight and store it for safekeeping. But this does little to address the issue of unstructured data. In fact, it can make the situation worse. Unless this information is properly catalogued and tagged, all this added content has to be manually trawled through every time someone wants to find something.’

As with so many ‘next big things’, big data is all about balancing quantity with quality

Solving the issue of enterprise architecture is one thing. As the Protection of Personal Information (POPI) Act becomes law, more and more businesses are going to find that big data – if inadequately managed – might cause great legal headaches.

‘POPI requires that you gather data for a purpose, that you only use it for that purpose, and that you then get rid of it when that purpose is complete,’ says Mullon, highlighting the issues around storing customer data that contains personal information.

‘We can’t just keep everything forever any more. It gets complicated when we use big data from an analytics point of view, when we’re extracting it and we want to get real business insight from it.

‘Businesses will have to sanitise the information in such a way that it cannot refer back to an individual. We can pick up trends and things like that, but we mustn’t be able to point back to a specific individual. That’s where it’s going to be tricky,’ he says.

According to Morgan, another issue regarding big data – especially in SA – is the lack of technical skills combined and business knowledge.

‘We don’t have enough appropriately skilled resources to deal adequately with more traditional IT areas, such as web development and business intelligence. Are we going to miss the benefits of big data because our country simply doesn’t have the capacity to take advantage of it?’

To answer that, it must first be determined which business will really be affected by big data.

Meryl Malcomess, marketing director of SYSPRO Africa, offers a word of caution. ‘Most medium-sized businesses don’t have a new big data problem. They simply have a data problem. Big data, as a principle and problem, is only really faced by organisations that truly create enormous amounts of data. These organisations include banks, government institutions, large retailers, online consumer facing businesses and the like.

‘They have massive amounts of transactions and system interactions taking place, both in volume and pace, and need to be able to quickly process the data and extract value.

‘That said, there are no differences in the solutions for small or big data – small, medium and large organisations can all benefit from big data solutions, even if applied to smaller data sets,’ she says.

As Malcomess points out, SA business are no different to their global counterparts.

But, she says: ‘How they deal with data problems – whether they are big or not – will have a material affect on how they compete both locally as well as globally.’

By Will Sinclair
Images: Andreas Eiselen/HSMimages