IT / Technology
Leadership and “Big Data”
There’s been a lot of discussion about “big data” in the media recently but what is all the fuss about and isn’t this just the latest information technology and data challenge that the IT folks have to deal with?
Before we look at Big Data, let’s look at what small data is. Small data is the amount of information you can conveniently store and process on a single computer, whether this is a tablet, laptop or PC at one end of a continuum, or a server blade or full server at the other end. Most data kept in organizations is small data, mainly being database information of all kinds, or data kept in documents and spreadsheets etc. And even in the public domain, data kept in computers and servers on things like energy use by household , the times of local buses or trains, or government spending are all examples of small data too.
A lot of people argue quite compellingly that we have yet to come to grips with even our small data challenges in the workplace and elsewhere in life. However, this does not mean that we can stop the onset of Big Data challenges at the same time, so let’s look at the new challenge of Big Data.
What is “Big Data”?
Big data by comparison to what we have described above as small data is a collection of data sets so large and complex that it becomes much more difficult to process using straightforward database management tools or traditional data processing applications. As it says on Wikipedia, “the challenges of Big Data include capture, curation, storage, search, sharing, transfer, analysis, and visualization. Put another way, Big Data is characterized by its volume, velocity, variety and value.”
Examples of this high volume, velocity, variety and value data is now too numerous to mention but to offer some ideas, here are just a few examples:
- Credit/debit card swipes
- Web site clicks
- Dialed phone numbers
- Vehicle transponder data
- Facebook/LinkedIn posts
- Closed-circuit TV footage
- YouTube video posts
- Downloads of all kinds
Every one of these across the globe runs into many millions a day at the raw data level and this does not include all the metadata that often goes with it. For example, each phone call transaction might include not only a phone number string, but the caller ID, the number and ID of the number called and geographical location of both parties.
So how large is this big data?
Over 2,000 Petabytes (which is 1024 terabytes) or 2.2 million terabytes of information is created every day in the world. And for those who don’t know, a terabyte is 1024 Gigabytes and a Gigabyte is 1024 Megabytes. In layman’s terms, if your average MP3 is 3 megabytes, 1 terabyte would hold 333,333 songs. And 50 Petabytes would hold 25 trillion pages of printed text-something that would take 25 billion old style floppy disks to hold. In addition, 90% of the data in the world was created in the last 2 years-but data is not information! The challenge is therefore to derive meaning from as much of this Big Data sets as we can.
So you get the idea, the amount of data generated is huge (even though we should also recognize that individuals are still pretty amazing data processors too). The average human brain can process one petabyte per second (although storing it for recall is still something of a challenge!).
Big Data and Leadership
Given the above, Big Data is fast accumulating in the world whether we like it or not and some of it is very useful and interesting from a business perspective. This is because some of this data is already in the public domain and can be drawn upon in different ways. In other words we can take small company specific data and relate it to public big data in order to create a richer picture and one which may change our decisions about who we provide service to, why, how, where and when.
Naturally, as with most management responsibilities, making effective use of data is more art than science. This starts with someone very high up in the organization (and often a general manager or CEO) being an advocate for taking control of and using both small and big data in the future. Organizations should then be seeking leaders, at all levels, who can understand the complexities of the business, recognize opportunities for better use of data, and make the business case for assembling the necessary data resources. Put another ways, leaders become serious “knowledge managers” who make sure that internal (mainly small) and external (often big) are used in intelligent ways to drive better outcomes.
Big data is already here in the world and merely adds to the challenges we already have in an online knowledge economy. In reality, much of the current debate about big data is still very simplistic being more concerned with how much to spend on storage or advanced analytics technology, rather than to start asking questions about what data exists and how it can be best drawn upon (and especially as it grows and more of it becomes available for use).
Perhaps the greatest challenge today is that maybe less than 0.1% (and it may be much lower than this) of the data we have available to us is being tapped intelligently and turned into useful information. Only visionary leaders will change this situation and taking big and small data seriously is the first step. Then and only then can we properly capture, curate, store, search, share, transfer, analyze, and visualize what all this data means and use it in ways that are both responsible and growth-centered.