Stocking the Data Lake with Smart Data: IT-Business Partnership is Key

The data lake is proving to be a crucial tool as EMC IT strives to partner more closely with the business clients it serves to help them get the most out of enterprise Big Data. For example, EMC IT is offering a smart data base that lets business users across the company leverage a uniform customer profile for more efficient and effective sales analytics.

Created in collaboration with EMC Global Services, the CAP (Customer Account Profile) is based on information collected and aggregated from multiple sources to provide a holistic customer view—a single version of the truth, if you will, about our customers.

CAP is managed by IT and is one of the enterprise data sets made available via the data lake to business clients seeking to analyze customer trends, opportunities and insights.

The CAP tables actually stemmed from a longstanding effort by the services team to meld customer data together to create a single customer profile. Because there was no easy way to blend customer data on things like bookings, revenues, install base, service request and more, the team used to struggle to piece together such information using shadow IT.

However, with the introduction of Pivotal and Greenplum solutions and IT’s focus on working more closely with the business, IT was able to work with Global Services to not only bring together this customer data into a single data base, but to also seamlessly provide the complex logic and technologies to convert the raw data into a more consumable format for data users.

data lake
Click to Enlarge

When we began creating our business data lake two years ago, access to this customer information became even easier as we included CAP among the data sets to which business groups can subscribe. What we found was that many business groups across EMC who use the data lake were eager to have an aggregated summary of customers.

By providing this general description, IT enables business users to get the same view of customers without having to re-invent the wheel each time they want customer data. They no longer have to collect their own information and redo all the complex logic to make it usable. Users can take a subset of CAP data and put it in their own space and work with it supporting their specific analytics.

IT maintains the CAP tables, updating them with the help of Global Services and adding new features. We have built in a lot of formulas that make CAP a really “smart” table that works both for technically skilled users and those who are less technical.  You really don’t need to know all the SQL sets or technical skills to consume these tables. Someone can get in there and do some great stuff with this generalized data.

To understand how much easier CAP makes analytics, consider an example from Frank Coleman, Senior Director of Operations for EMC Global Services. If the Customer Service Operation Team asked him to supply a few years’ worth of data to research an approach to improving service efficiency, Frank says, it previously would have taken his team a few weeks to a month to pull it together. But with CAP, it can now be done within a day.

In fact, with CAP, GS can often pursue and review the answers to client’s analytics questions in real time.

We currently have 23 different users with work spaces that are accessing the CAP tables and expect usage to continue to grow.

The value of our CAP collaboration goes beyond this one data set. It showcases the journey to an analytics-driven company and the need for IT and the business to partner very closely to define the levels of data consumption. In this case, IT worked with the customer to create some complex logic and then flatten out that data for easier consumption. (Read Frank’s blog, 4 Key Steps on Your Journey to the Data Lake)

As data lakes evolve, IT departments need to partner a lot closer with their business clients and work to understand what they are seeking to do and how they want to consume data. IT then needs to deliver it in a consistent way.

About the Author: Mark Duncan