The Big Data Movement
Big data. Sounds mysterious, maybe even mythical. But these two words are changing industries everywhere. A recent explosion in data mining and data processing has brought on the golden era of all things data. It’s time we collectively got up to speed on what this phenomenon is and how its power is affecting our lives.
Analytics have been a part of businesses for decades. Industries have been combing through any available data for hidden secrets or trends. These analytics could center around anything from customer purchasing data to machine operability and maintenance on a manufacturing floor. Everything generates data, but the key is how you utilize it. Big data is a culmination of these efforts. We now live in an age where mining data is no longer impossible and machines are more useful than ever. This has brought a renewed focus around what big data is capable of achieving.
So just what is Big Data?
Big data is essentially a massive accumulation of statistics from day to day interactions within a business, product or service. These data are collected by industries and then analyzed for trends that provide useful revelations. The hardest part is extracting the so called “useful” revelations because often they come from correlating data that may not be obviously related.
Jerry, the mouse
Let’s use an example to explain further. Say you have a mouse problem in your garage like I recently experienced. We will name him Jerry. You’ve seen Jerry sprinting around but haven’t been able to catch him. But then Jerry goes away for days. Wammo! Problem solved… Until you see him again. How did he get back in? Meanwhile, your garage door panel no longer works, you haven’t slept in days dreaming about this mouse, and the world feels like it’s going to end!
What data do we have? Jerry is getting in somehow, but you’ve kept your garage shut for days. He’s obviously finding a way in and out but you’ve found no spots after clearing your garage. No matter where you’ve set the trap, he’s avoided it. And that garage door panel is still broke!
Enter big data analysis. You, being the super computer that you are, connect the dots. A day later while fixing you garage panel, you noticed the power seemed to be dead. But after changing the battery the problem persisted. So then you looked at the wires and.. aha! The wire is in half. But how? That’s right, ole Jerry chewed through it. And lo and behold right next to the wire is a tiny gap in your garage wall. You set the trap, and a day later, you bid farewell to Jerry.
This is a much simplified version of big data usage at play but hopefully drives home the point. Multiple data can be connected, even if you believe them to be unrelated. The big data phenomenon is placing these seemingly obscure puzzle pieces together to tell a story. Take those few data points and trends from the Jerry example and explode that by millions. Then connect those dots to solve your mouse problem. Now we are talking big data.
If you’re still confused by my wild analogies, maybe this handy video will help you understand big data on some basic level:
Big Data: Why now?
It is a fair question. Why is big data suddenly gaining such momentum? After all, data gathering as a practice has been around forever. Businesses have been formulating new metrics each year to measure growth and success. So what gives?
My processor is bigger than yours
One of the more obvious shifts in data collection is around computers. I remember back in the early 2000s when a dual processor and x32 bit computing was all the rage. Even for those companies ahead of the curve (quad-core hardware, etc), the software was still catching up to the hardware.
As developers figured out how to take advantage of the additional horsepower, big data became more useful. Multiple processors, increased RAM, and x64 bit processing were just some of the contributors that unlocked the potential of big data. Suddenly, processing 1,000s to 100,000s of data points could happen in a flash. With organized data in hand, the analytics folks could now focus more on meaningful conclusions from the treasure trove of information.
As we marched through the 2000s, cloud computing then suddenly became all the rage. Over night, you were no longer beholden to what physical disk space was available to your computer. Along with expanded storage came the explosion of the virtual desktop. Those incremental shifts in processors and RAM just mentioned? Go exponential. Companies like VMware and Citrix created software to easily daisy chain super computers together. With one 7ft by 2ft rack, you now had processing power beyond anything we’d ever seen before. Big data was the biggest benefactor because now everyone could afford to collect massive amounts of data to improve their business.
The other key factor in the rise of big data has been our old friend father time. However, not like time mentioned above where technology could mature. Time in the sense of pure data volume generation.
In order for big data to work, you need two things. One, as the name suggests, tons of data. This doesn’t happen overnight. Even with hundreds of thousands of data points pouring into storage, big data can always use more. Two, and most importantly, big data is impactful because it provides a base to uncover trends. Data in bulk is not that impressive, but draw a trend line through it, and suddenly you have people’s attention.
Those trend lines are enabled by time. A quick example of father time at work. You shoot free throws at a basketball hoop in your driveway. Because you’re a boss, you sink 7 out of 10 free throws. A decent 70% shooter you are, only missing three. But it’s a nice day, so you keep on shooting. An hour later, you’ve made 97 out of a 100. You’ve still only missed three but now you are a 97% shooter! Time, consistency and sample size allowed this to happen and that’s what makes a useful trend. We tend to believe you are closer to a 97% shooter than a 70% shooter. This is the power of big data and what it can point out.
The Future of Big Data
So where does all this big data talk lead us? Honestly, you are feeling the impacts in your everyday life. I could list thousands of examples as to where this is being deployed today but I’m guessing you have a hunch so I’ll just mention a few.
Some uses of big data are already mature. Google and Facebook have paved the way in the social scene, with targeted advertising. The reason most ads are so seamless today is due to big data. Then retail titans like Amazon and Walmart have employed these techniques to predict what you need next. Those models have only become stronger with more data, improving convenience for the consumer each day.
Where big data is continuing to mature is in automation. The voice assistants in your cell phones or smart devices, like Google Assistant and Alexa, are all about big data. They are combining statistics around your purchase habits and those like you to simplify your life. As more people adopt such technology, it will continue to solidify it’s usefulness.
Self-driving cars are another great example of big data maturation. While the trial period has been somewhat dangerous for this technology, the collosal amount of data gathered each run will continue to improve the safety around the technology. Combine that maturing code with the billions of data around people’s driving habits, and it’s clear you will have a path to success (though not full-proof). Should be interesting to see where that technology ends up.
Lastly, big data around production equipment and automation is another key development. Robots scare alot of people but it’s true they have the potential to be programmed for high efficiency. We are not there yet, but big data will continue to drive that bus.
The future for big data is bright. The ability to generate useful data has only begun to crescendo. As machines get “smarter” they will implement this data in real time to become more efficient. In an optimistic world, this should make life easier for the masses with better, more useful products and services. It remains to be seen what impact big data and automation will have on society in the long run. Only father time will tell.
Like this column? Please follow Jarkable on the social media platform of your choosing for updates to all the latest postings!