In the past year, my Big Data journey has evolved into a successful, revenue-generating, innovation-enabling solution. How did I get here and what recommendations can I offer?
First, properly set expectations for Big Data. One executive asserted, “If you get enough data together in one place, it automatically generates answers to questions you didn’t know to ask.” He believed that a primordial soup of data would simply spring to life. As you invest, emphasize that the goal is to answer questions that had previously been too complex or expensive to answer.
Second, avoid Data Scientists initially. They will be useful later, but not at the outset. There are two types of Data Scientists. The first know how to set up a huge Hadoop or Elasticsearch cluster or a big NoSQL database. The latter know statistics, the ‘R’ programming language, and graph theory. In both cases, they’re a solution in search of a problem.
Third, listen to parts of the company that lack a voice. Big Data reduces the cost or complexity of solving problems. Look for areas where the business has been unwilling or unable to invest. Our journey to success began when a support engineer observed that we could use Big Data to predict within 90 days that a Data Domain would run out of capacity. He was tired of taking support calls about “failed backups” because the backup teams were not trained to monitor storage capacities. He knew EMC could do better, and Big Data allowed us to do it without a huge investment.
Once you have properly set expectations, avoided the pitfalls of gratuitous investment, and have found a critical but underappreciated problem to solve, there are three success factors:
1. Be Open – Too often organizations will create a Big Data Lake, but prevent people from accessing the data. Innovation comes from bringing creative people and data together. Governance is important, but don’t let IT lock everyone out.
2. Revenue vs. Optimization – Many people want to optimize a process (e.g. fewer support calls or faster bug triage), but optimization is difficult to quantify and even harder to justify investment. Instead, focus on ways that Big Data can augment your revenue. At first, we futilely tried to get funding by demonstrating “reduced support case load.” Interest and funding expanded when we tracked the revenue generated by selling additional Data Domain storage and systems to customers who were about to run out of capacity.
3. Generalist vs. Specialist – At the beginning, you don’t need a hyper-optimized Big Data infrastructure. You need somebody who understands the business problem, what data they need, how to access the data, and how to deploy basic Big Data tools. In short, you need a problem-solving generalist who can learn quickly. As the solution expands, hire specialists to optimize each part of the process. At the beginning, though, generalists win.
As with most business/technology transformation, the challenge with Big Data is not one of technology. To succeed with Big Data, manage business expectations, avoid technology hype, and embrace revenue-generating ideas from underfunded areas. If you keep your Big Data Lake open and accessible, you’ll unlock the innovative passion of parts of the company that have been desperate to do more.
Often, starting small – especially when it comes to Big Data – can have the greatest payoff.
For more insight into my big data journey and lessons learned, check out The Business Impact of Big Data podcast series.
