Got a Big Data problem?

CIO #1: Do you think you have a Big Data problem?

CIO #2: Yes I do.

CIO #1: When did your Big Data problem start?

CIO #2: I would say we’ve had a big data problem since the mid-1990’s. 

CIO #1: Huh?  What do you mean?  Sources for the Big Data problem like social media and smart mobile devices have not even existed that long, and I didn’t even think your data volumes had reached that level.

CIO #2:Well, here is the way I think about it. Our business is rarely discussed in social media, our most critical data generating and consuming apps are internal-facing, and none of those apps even support mobile interfaces…but…I 100% guarantee you I have a BIG problem with DATA.  We are creating and storing more and more data every year. We’re seeing it used, and sometimes abused, all over the business by people I didn’t know needed the data.  We have a huge investment in our data warehouse. We have a couple reporting tools. Dashboards are being built and re-built almost every period.  Data is being transformed and moved around to the point it has lost value in some cases.  And, with everything we’ve done so far, our lines of business are constantly complaining.  We simply cannot process the data and provide reports fast enough or in a way that is useful. 

CIO #1: Uhhhhhh….

CIO #2: Sorry to go on and on, but you hit a nerve.  I’ve been battling this one for a long time – longer than many self-proclaimed Big Data experts have even been in game.  It’s gotten better over the years, but I still cannot get ahead of this very big data problem.   

CIO #1: Well……I guess you have a point when you put it like THAT.      

There are so many definitions, interpretations, and claims about Big Data right now, it’s nearly impossible to keep up.  But how much does the exactness of this definition really matter?  If the big data problem is defined by the limitations of traditional data management technologies, how and when do organizations reach those limits?  Does CIO #2 have to wait to fix his “big data problem” until the limits have been reached?

What matters is that organizations identify they have a big data problem, or rather a big problem with data.  It matters that organizations know their definition of their problem is what matters.  No matter how fast, how big, or how varied an organization’s data universe is, it’s all relative – to their history, to their resources, and to what questions they are trying to answer. 

What matters more is that both IT and the ‘Business’ recognize they have an opportunity to do more with their data.  Both groups should work together to address the problem.  IT cannot do it alone under the promise of the newest technologies.  Business users should not attempt a quick fix in spite of IT.

And what matters most is the impact of fixing your big problem with data.  So, before you start to fix it, determine the impact you need.  And although the problem may be big, start small.  Start with a departmental problem felt most by teams that deal directly with your customers, because if you don't, your competition will. 

One last thought.  We can talk more about it next time.  This blog post started with a question.  Ask yourself that same question.  Did you answer no?  If so, that may be your problem.

About the Author: Darin Bartik