The Big Data ‘Problem’

It is said that the term ‘Big Data’ comes from what a decade or so ago was ‘The Big Data Problem’ (big going with problem not data)!

So what was the Big Problem with data and how has it morphed into the Big Data Opportunity?

As humans, we quickly figured out that we needed a way to express ourselves, with images, writings, sculptures and for many centuries this has worked just fine.

But the 20^th century saw innovations that changed the way we communicate forever, leading to today’s digital world where devices communicate information ‘digitally’, allowing us to upload, download, store and share text, audio, video, images and even money.

We are now creating more data in every two days than was created from the dawn of civilisation all the way up to 2003 (a figure in the area of five exabytes or 5 trillion megabytes). In fact, 90 percent of the digital data that exists worldwide today was created within the last two years; and 2.5 quintillion bytes of extra data are created each and every day.

And not only do we create new data, but we also started digitising what existed in physical format: books, paintings, vital records, research papers, music, photos and so on.

Great no? Well, not exactly. All this digital information, this data, had to be stored somewhere.

Traditional servers and processors were not built to handle this rising storm of data that was coming in, in varied formats and in increasing numbers and speeds.

To give you an example, in 1956, IBM introduced the first hard disk drive, the RAMAC 350. It could store up to 5MB of data, or the equivalent of an MP3 song, at a cost of about $1,000 per megabyte. This made data storage an expensive endeavour, afforded only by governments and large corporations, creating a big problem with data.

Over the last six decades data storage has changed dramatically and storage technology has started catching up. Today, a 3.5-in. desktop drive can store 4-5 Terabytes of capacity or about three-quarters of a million MP3 songs, at a cost of $0.000021 per megabyte!

This solved the data problem, at least as far as storage is concerned, but not the Big Data problem!

Why? Because if your data can be stored and processed on a single machine, then your data is not Big enough!

Which brings us to the 3 Vs of Big Data: Volume, Velocity and Variety. To qualify for ‘Big Data’, typically you need to have a high volume of data that comes in at different speeds (that is velocity) and in different formats (variety) –so much so that it is impossible to be stored and processed by a single machine. 

To understand what this data is, use it and extract value from it, you will need to invest in Big Data technologies. These will allow you to Capture, Store, Analyse, Search, Share, Transfer, Visualise, Query, Update and Secure.

To give you an example:

Say you have a small startup that makes specialised swimming fins with activity trackers - the Fitfins! At the beginning, you have a handful of customers that shop online from your website and 1 employee. This data you can store and process easily with today’s available off-the-shelf technology and applications on a single machine –a PC for example.

But with swimming becoming the latest fitness must, Fitfins become the new craze and orders start piling up from all over the world. You hire more employees, you expand your facilities, you move to a more robust eCommerce platform, you partner with marketplaces like Amazon and eBay to sell Fitfins on their sites, you provide advice and trends on your social pages, you sponsor swimming competitions etc.

Now data starts coming in from many different sources, internal (finance, HR, production, warehouse) and external (eCommerce, marketplaces, social media, supply chain partners). You know that there is important information to be retrieved from these data; information about customers, about the product, about the market, about trends, about channel performance; but you don’t know what this information is exactly or how to go about finding it.

This is where big data comes into play.

• First you need a place where all this data can be stored: this is your cluster of servers where all your data are stored.
• Then you need a tool that will allow you to manage this data –search, build applications, share, analyse, visualise: this is your Big Data platform.
  
  
• Finally, you need someone who knows what they are doing and can help you build your Big Data infrastructure and, most importantly, start deriving value from your data: this is your Big Data team of experts, who may be in-house or outsourced.

With your Big Data ecosystem now in place, you can actually find hidden nuggets of valuable insight about your business, like for example that most product returns come from people who wore Fitfins in cold waters.

Final note

So, if you think there is hidden value in your data but you struggle bringing it to light, then it’s time for Big Data. Our advice to reap the Big Data opportunity, is to start small, look for expert advice and support and in no time, you will have a whole new business case for your business.

If you would like to find out more about how Big Data could help you make the most out of your current infrastructure while enabling you to open your digital horizons, do give us a call at +44 (0)203 475 7980 or email us at Salesforce@coforge.com

Other useful links:

Big Data trends that will drive 2018

What is Big Data

Understanding the 3 Vs of Big Data - Volume, Velocity and Variety