Is Big Data Only Where You Think It Is?

Shopper insights.
Loyalty data.
Social commerce.
E-couponing.
Unstructured customer feedback.
The common point between these four disciplines – and many others? These are all variants of “Big Data.”
 

Recently, a CPG social forum asked the question “Is ‘Big Data’ the most dominant theme this year in our industry?” If you look at the amount of coverage Big Data is getting, it well may be. However, when browsing through the literature on the topic, I realized that people may be forgetting about the most obvious Big Data opportunity, one that has been maturing at a dramatic pace over the last five years and which is now starting to fulfill its promises.
 

The most immediate Big Data opportunity in CPG may actually be downstream data: many of our largest customers sell into more than 50,000 stores in North America alone, and the same number of stores in Europe. That means that they can get over 5 billion data elements DAILY with information on what is happening in their retailer partners’ stores. With two years of history, levering over a trillion data points of downstream data certainly qualifies as Big Data – and strains the traditional processing methods of databases.
 

As long as weekly retail data was analyzed primarily for reporting, a large traditional database with a business intelligence tool was most likely all that was needed. Some companies outsourced this function while some built internal databases to address this challenge. Most organizations were happy with a weekly lag for their data (a great deal better than the four-to-five-week lag from syndicated data providers). However, this situation dramatically changed over the last few years. 
 

With the industry renewing its focus on out-of-stocks, weekly lagged data no longer worked, nor did the traditional reports: looking at products not selling was generating massive lists of store-item combinations that did not reflect out-of-stocks but only low velocity items. At the same time the economic environment drove many organizations towards a new set of best practices. Why settle for a slow ramp-up for new products when we can respond next day to stores not scanning or not carrying inventory? Why set standard inventory levels and planogram depth if it is possible to customize it to the actual selling conditions at each store?
 

As a result, the supporting infrastructure has transformed. Most of the retail databases established for weekly reporting purposes no longer can handle the near-real time processing needed to identify and rapidly react to out-of-stocks, replenishment issues or sudden changes in consumer demands. The traditional no-scan approach to out-of-stocks (no sales for x days) is replaced by heuristics and advanced statistics that take into account each item’s sales pattern at each store.
 

Many of the industry leaders have come to realize that with the increase in retail data availability, the issue has shifted from “ownership and control of data” (i.e., whether you collect it or not) towards integrating it into their business processes for forecasting, promotions execution, merchandiser management, etc. in a seamless, replicable way. They have defined and are deploying a corporate strategy to build this infrastructure today, across their entire organization: functions and processes, retailers and continents.
 

Moreover, operational conditions need to be tied back to the other initiatives traditionally linked to Big Data. Why offer a customized promotion to your most loyal customers if the item is out-of-stock in the store they generally shop? What is the return of your marketing investments at the most local level - across a neighborhood? If you are experiencing high inventory levels for a specific item in a specific store, why not use a targeted alert when customers actually enter that store without requiring store personnel to apply a discount sticker? In the end, Big Data is successful when it links together all sources of intelligence in a holistic framework: market, shopper, competition – but also operations, by leveraging downstream data showing the actual sales occurring now in the stores.
 

Are you thinking of retailer downstream data within the context of Big Data? As usual, I’d love your feedback: you can reach me at [email protected].


_______________________________________________________________________________