Big data seems to have been everywhere this year, every media outlet, pundit and business seems to have latched on to the concept, or at least the term, many without really comprehending what “big data” means, how it is collected and how it could be used. In all of the discussions however, the individuals who are either the source, the users or potentially the unanticipated collateral damage of attempts by businesses to adopt or use “big data” have either been forgotten or ignored.

So, in this post, I though I would spend a little time considering some of the things big data is, what it isn’t and how it could be used. I will also spend some time considering the real value of the personnel who may be impacted by the use of big data and, why we still need to value them for their contributions as humans, instead of casting them aside in our pursuit of the fashionable, shiny new concept.

What Is Big Data?

I will qualify this section of the article by stating that I am not a data specialist or a computer engineer, but, when dealing with such broad terms as big data it is not necessary to be too much of a specialist, specialists will develop the tools and techniques to use the data but we all need to have a working knowledge of what it is. In essence, what we are now referring to as big data is the vast pool of electronic data that is generated globally, generated from pretty much every source imaginable. This includes data created from sources as diverse as commercial transaction records, computer monitoring of jet engines, personal mobile phone usage, our travel habits and the data produced by industrial plant everywhere.

In the past, we lacked the computing power to both store and manipulate much of this data, resulting in the great majority of it being discarded, but, as processing and storage capacity and capability has increased, we are finding we are now in a position to try to distill some of this pool of data into useful and meaningful information.

Not All Data Is Valuable

One thing to note in the whole debate is that the vast majority of the data generated and potentially stored is essentially noise, having little to no real present or even future value. The process is the electronic data equivalent of distilling a pool of sea water to extract the salt, the water represents the useless noise component of the data and the salt the potentially useful data, though even the raw salt is just that, raw, and still needs further processing before it has any real value.

Once the data has been extracted from the noise, it must be manipulated and interpreted to produce useful information that can be acted on, whether that is to recommend an engine for maintenance or to send an email to an internet shopper suggesting something they may be interested in based on previous searches. This data interpretation will eventually be fully automated, but to identify the potential uses, develop the algorithms, and build a business around this data requires a lot of work by humans, and highly skilled humans at that.

Where data analytics will eventually lead the human race is the subject of a lot of debate. There are many uses for well considered opportunities that will bring enormous benefits in health management, traffic management, control of process plant, maintenance planning of critical equipment, etc. But equally, there are a great many ‘grey’ uses such as personnel profiling based on personal and family health records, travel, friend networks, and so forth that may be undertaken, but which may change society for the worse. As time passes both the good and the bad will appear and decisions will need to be made over what, as a society, we want and what we don’t.

The Role Of The Little People

So, in all of this rush buy businesses to jump on board the big data bandwagon, just where does it leave the little people.