What is "Big Data"

"Big Data" - a computing revolution

Big Data – the next big thing!

You can't help but get caught up in all the excitement surrounding Big Data. After all, it's the next big thing – the revolution that will change business forever. So what is Big Data and is it all it's cracked up to be?

Data so big and/or rich that conventional tools and techniques used to perform analysis are rendered useless.

We now get data from an incredible number of different sources and it's relentless; website analytics, social media followers/likes/favourites, browsing habits, customer relationship management (CRM), sales pipelines, purchase histories, sensors, email, phones, etc. Big Data is all about trying to find relationships and correlations in all this disparate data, so we can obtain a better understanding of our company and our customers.

It's not about the quantity of the data, but the quality of the data that counts.

A common misconception is Big Data is all about gathering and storing huge amounts of data. In reality, Big Data is the revolution of specialised tools and techniques for shifting through large amounts of data and uncovering previously unseen insights.

What people don't tell you about Big Data

Big data is like looking into a wishing well, only to find none of the coins you have are big enough!

Much of the talk about Big Data revolves around the type of data you should collect and how to collect it, e.g. Google Analytics and purchase histories. Far less is written on what to actually do with the data once you have it. That's because with Big Data it is actually performing the analysis that is the difficult part. It's not like you can load the data into an Excel spreadsheet and start crunching some numbers – for one thing, Excel 2013 has a row limit of 1,048,576... hardly big data size!

An addition, because the analysis of Big Data involves unstructured data, the statistical relationships that are uncovered cannot be easily verified and as such you need to use extreme care with any results and conclusions.

Processing power!

Many of the tools and techniques used in Big Data have been around for a long time. One of the reasons many people don't know about these is due to a totally different problem – processing power.

Training systems to find hidden patterns in data takes a long time and demands a lot of resources!

Back in the early 2000's, I set up a number of experiments to teach a computer the underlying syntax of the English language. By today's standards, the data sets were tiny (some 3000 sentences), but these took nearly 2 weeks for the computer to learn! Things have come a long way since then (better algorithms and a huge growth in processing power). Even so, only the biggest companies in the world have the computing resources available to really take advantage of Big Data.

Where does that leave us?

The reality of Big Data is still beyond most companies, but not for long. In an amazing show of openness, Google, Microsoft and now Amazon, have all made available tools that can be used to analyse disparate data sets. We are on the verge of a computing revolution that will drastically change the way in which companies learn and interact with their customers. Exciting times!

ARTICLE TAGS: BIG DATA | ARTIFICIAL INTELLIGENCE | KOOLTH | GOOGLE | MICROSOFT | AMAZON