Harmonizing Your Data for Analytics Readiness

Consumer goods companies are spending millions of dollars to access data from multiple sources. But unless these data sets are harmonized, their true power can’t be realized.

• Inconsistent values for the same metric from different sources, such as internal shipment data versus syndicated market data.
• Teams using different data sources to address the same problem, leading to conflicting insights and an inability to compare notes.
• Misaligned reporting periods between data sources, like that of your marketing department versus retailer transactions, which makes it impossible to measure the effectiveness of activities like promotions.
• Paying data providers to clean data sets to make them useable on top of the license fee — and even then, they aren’t living up to their potential because they’re not aligned with other data sources, making it difficult to justify the spend.
• No central process for data governance, which means that rebrands, retailer consolidations or changes to reporting processes render all analysis useless until a comprehensive remapping across all data sets has been undertaken.

Having different data sets pointing in different directions can be worse than not having data at all. At best, incompatible data sets represent a waste of money; at worst, using them for business-critical analytics could lead to ill-guided strategies and material loss to the company.

Good data can transform a business, however. So your focus should be on turning these disparate sources into a coherent whole that lets you access the relevant data at the right levels. Data harmonization does exactly that, and typically at a fraction of the cost of purchasing the data sets in the first place.

What is data harmonization?
Harmonization is a continuous process that aligns all available data sources — market data, shipment data or customer data — across all key metrics.

By synchronizing data points across products, channels, time periods and geographies, otherwise disparate data sets can easily converse with each other. And that brings the full range of business data to bear, making the entire organization "analytics-ready." This will:

• Increase the accuracy of reporting, which makes building comprehensive models for engaging your customers a certainty.
• Allow more accurate assessment of total addressable markets, market share and market forecasts.
• Make exception reporting possible by having relevant data points address key questions.
• Streamline the sales and operations planning processes by having a  "single source of truth."

Harmonization vs. transformation
Combining data sources in a single warehouse is not the same as harmonization. There are many tools in the market that claim to transform and integrate different sources, but few are capable of truly harmonizing disparate data sets to the level required to build powerful analytics.

Sometimes, integration is really all that you need. If the objective is to check the data for completeness, validate for accuracy or simply add to your existing data warehouse, then you may not need a complete harmonization. Setting up a simple "extract, transform, load" (ETL) process or commissioning a data service provider to clean your data may suffice.

However, if your objective is to make the organization "analytics-ready," then you should embark on the harmonization journey. This should include:

• Comprehensive validation steps to ensure complete, consistent and accurate data points to enable metric-to-metric comparison across sources.
• Creation of attributes — additional fields or derived metrics — using mapping files or business rules to enhance the analytics-readiness.
• A scalable process that can handle additional metrics and domain nuances.

Harmonization also simplifies data governance — a process overlooked at most companies. Updates to one data set will automatically be reflected in all others. This can prevent companies from making multi-million dollar errors. Imagine if a retailer consolidation was reported in your internal systems, but not in the syndicated market data; the insights derived and, therefore, the recommendations for your customers could be disastrous.

The Trouble with Data Sets
While you should expect data providers to share data sets that are ready to use, that unfortunately is not always the case. Even data from top global providers can have problems such as:

• Different structures for different markets or categories.
• Additional costs for cleaning data or providing a streamlined feed.
• An inability to access data from other providers due to competitive practices.
• A lack of true harmonization capability or the patience needed to meet specific client demands.

This means that, even after spending millions of dollars to access data from multiple sources and even more on having it all streamlined, your organization could still be left with data sets that can’t talk to each other.

Taking the plunge
Choosing the right partner to guide you through harmonization is vital. But before you can choose that partner, setting a clear objective is integral to ensuring that the process does not end up becoming a simple data integration.

A strong vision, and identified use cases, will help you choose the relevant fields and optimize your effort and resources toward enabling the analytics solutions you need. This requires a commitment from any organization embarking on the harmonization journey to deliver the right infrastructure, set up robust governance processes, and ensure that key personnel are engaged in the project.

For consumer goods companies, the rewards are considerable. You will finally be able to unlock the enormous value in datasets currently sitting idle or  — even worse — generating misleading insights. By turning the flood of data that's available into a coherent, cohesive whole, data harmonization gives businesses a full-spectrum view of their organization and the markets in which they operate.

Santosh Kumar is Principal – CPG and Retail, at Fractal Analytics, Jersey City, NJ. He can be reached at [email protected]