Master data and Big Data sit on opposite ends of the data spectrum. They look different, are managed differently, and ultimately serve different purposes. However, with the proper enterprise data practices in place, these two seemingly unrelated data sets working in tandem can become greater than the sum of their parts.
Master data is slow changing data. Master Data refers to attributes such as name, address, phone number, emails, contacts of your customers or attributes and features of a product. Master data can also be used to slice and dice transactional data, in order to better understand a company’s business operations and opportunities. Master data is typically small – the largest online retailers may have a master customer list that is a few millions rows of data, but for the most part, master data is much smaller in scale. Master data is also significantly cleansed and is scrubbed periodically to ensure its accuracy. On the opposite end of the spectrum is Big Data. Known for its massive volume, variety and velocity, big data is generally acquired from external sources with little or no room for cleansing or scrubbing it.
Master data and big data do share one important similarity – They can both serve as great assets for those organizations that pay close attention to them. More and more companies are expanding their horizons by exploring the vast world of unstructured data from external sources, such as social media, mobile, chats and other online interactions. As a result, there is a growing challenge within these organizations to monetize the benefits, and thoroughly understand what the data is telling them beyond insights at an aggregate level.
Social media analytics are one of the primary drivers of big data. While there are numerous algorithms to analyze the data at the aggregate level (e.g. sentiment analysis), it is still challenging to analyze the intent of individuals and connect the dots to what the company already knows about those individuals. Are some of them existing customers? Is a sudden spike in interest coming from a prospect about which the company is already aware? Is there a potential for up-sale? Is there additional risk within a segment of consumers based on who is talking, where, and why? Are certain products being discussed more than others? These are some of the questions that aggregate analysis will not be able to answer well.
Enter master data.
Organizations with robust Master Data Management (MDM) practices paired with trusted data quality processes and a set of reliable data governance policies, are more likely to recognize how master data feeds big data. If the goal of a big data analysis is to discover potential “new customers”, then it makes sense to be able to run the list generated from the big data analysis against those already available in the master data repository to determine the true “new customers”.
This relationship between big data and master data is symbiotic. Big data can “unlock” master data entities from various new sources, such as social media or text (contract PDFs) to further create a 360-degree view of the entity, often informing their relationships and hierarchies.
In reality, big data and master data couldn’t be more different from each other. There are certainly challenges in how these two unique forces can come together. Master data has traditionally been collected from reliable sources and consumed by structural data environments, such as EDW and data marts. The unstructured nature of big data, coupled with a need for a different approach to processing and storage, has created numerous challenges in how big data and master data interact. For example, master data must be augmented to ensure the disparate pieces of information that are gleaned from big data analysis can be associated with an existing customer or product. If that association cannot be built, the result will be untrustworthy and dilute any long-term value realization from big data efforts.
If there is one key lesson here, it is that the attention on big data should bring about a renewed vigor and focus on the need to strengthen the organization’s existing enterprise data practices, especially those concerning its MDM practices. In summary, a big data approach can benefit from using master data as a starting point of analysis, especially into customer and product data, whereas master data can be enriched, enhanced and better understood for more targeted campaigns and promotions to help monetize the benefits their collaboration can reveal.