It is coming. The evidence is piling up like a giant snow drift from a blizzard on out doorsteps. 
It comes in many forms. Close your eyes and think about the changing nature of data  in your supply chain over the last two years. Data volumes are exploding, data velocity is increasing and data types are proliferating.  Most companies that I am working with are struggling.  The question is how to develop a road map to best use the many types of data the right way. The list is long and includes:  transaction data, sensor transmission, social text proliferation, downstream channel data, distributor network sales, warranty information, customer contracts, product IDs for serialization, geo-location and map data. Yes, data is exploding both in type and volume. 
It will continue to grow, but more importantly, it allows us to to define new capabilities.   This growth will be exponential.  Today, it is on the doorstep of our supply chain, early adopters are experimenting and will use it to power supply chain innovation; and within five years, I believe that the holistic use of this data will be mainstream.  

 The Drivers

 A major force is convergence:  unleashing the power of mobile, geolocation, digital and social data together.  Innovators—Amazon, Ebay and Yahoo—are busy marrying structured and un-structured data to harness new opportunities in their big-data supply chains. We are also seeing work in this area by Endeca and Microsoft. It is an exciting opportunity even though it is not main stream yet for manufacturers.
 The second opportunity is building TRUE customer-centric supply chains. Demand-driven is not sufficient. We must do more than sense and shape demand.  Instead, they need to be market driven connecting customer’s customer to supplier’s supplier horizontally.   Supply chain leaders have always wanted to better serve their customers, but there was no way to listen and learn.  Today, new technologies make this possible.  They are called sentiment analysis. This shift will change processes from the outside-in (market-driven response) versus the traditional inside-out response to power test and learn environments. It will be the merging of structured and unstructured data.  For example, in consumer products, Point of Sale (POS) data may tell us what is selling when and where, but unstructured data will give insight into the elusive why.  This more complete data set allows the set-up of test and learn environment powered by technologies like Applied Predictive Analytics (APT).
 The third driver is global infrastructure.  The average manufacturing client has three ERP instances (one for each geography), and each has over a terabyte of data.  The good news is that hardware is cheaper, the bad news is that these systems have become so integral to the business environments, that they can no longer afford the outages associated with maintenance upgrades and system upgrades. We are seeing big data supply chain techniques applied to increase up time and more seamless integrate the data instances.

What is a Big Data Supply Chain?

 I am excited about this topic.  So excited, that I have written too much for a blog post and too little for report, so I am going to share insights on the topic of Big Data Supply Chains in a series of small blog posts over the next five days.  In this series of blog posts, I familiarize the reader with the challenges and barriers.  In the subsequent posts in this series, I attempt to boost line of business leader’s vocabulary by defining new terms to know, share insights on emerging technologiesand close by giving the reader a call to action.  I hope that you enjoy the series.   
I call it Big Data Supply Chain 101 for Business Leaders or Big Data Supply Chains for Dummies.
Let’s start with a definition.  What do I mean by Big Data Supply Chains?  They are value networks that extend from the customer’s customer to the supplier’s supplier that sense, shape and respond by listening, testing and learning with minimal latency. 

• What it looks like:  It combines structured and unstructured data to sense, listen, test and learn to shape the intelligent response  horizontally and cross-functionally.  The processes are outside-in not inside out. They are bi-directional from buy to sell-side markets connecting the customers’s customer to the supplier’s suppliers.  Because the data volumes are so immense –with high velocities and variabilities– the creation of big data supply chains will require new techniques for the capture, storage, search, visualization and sharing of data. It is the world of terabytes, exabytes and petabytes.
• The building blocks:  A pre-requisite is excellence in today’s business analytics:   reporting, scorecards/dashboards, optimization, and intelligent rules. It will challenge today’s world of supply chain applications.  I believe that it will transform the Advanced Planning and Scheduling (APS) market and will redefine Customer Relationship Management (CRM) and Supplier Relationship Management (SRM) applications.  Hence the title of this blog post.  It will transform today’s R&P applications–APS, CRM, SRM, ERP–but, only for those that mind their Ps and Qs of supply chain understanding to harness the power of big data supply chains.

 The Challenges.

 It is a major shift.  One that will take time.  Let’s start by examining the challenges: 
Issue #1. Business leaders step up.  While the vision of the company is dependent on the business leader, they are struggling to keep up with the myriad of technology changes. In my work, I find few business leaders understand the principles of supply chain analytical design, and I also find few IT leaders that understand the “Art of the Possible”or the business impact, of these new approaches on their business. It is a conundrum. …one that requires inspiration, perspiration and innovation. The lack of business leadership will be gating factor between leaders and laggards.
 Issue #2. Will someone with a title “business”, please stand up! I sit and chuckle at supply chain  conferences when the presenter says, “IT should work with the business.” Why do I laugh?  They don’t exist. There is no one person that has the title “business” on their resume.  If only it was that easy. Organizations are hamstrung by the lack of capability and accountability in business leaders to capitalize on this change.  As a result, IT is forced to deal with business users in many different forms –marketing, sales, supply chain, research/development, procurement, etc–each with their own world view of how new technologies can improve their functional response; yet they lack the understanding of how to build the horizontal processes for the company.  As a result, many of the efforts in supply chain will continue to be project-based not program-based. This splintered approach has resulted and will continue to result in many disparate applications that fail to meet the criteria of good analytical design for the supply chain: isolated projects with no master plan.  It is time for the line of business leaders across the corporation to stand-up and help form a guiding coalition to realize the potential of harnessing data for insights.  Big data supply chains is too large of an opportunity to move forward as a project-based approach.  I say ENOUGH!
 Issue #3. Supply chain needs to be the business. Unfortunately, the terms business analytics and supply chain are politically charged words. This potential of this opportunity is too LARGE to get hung up in politics.   Most organizations today are focused on improving the efficiency of vertical silos and the supply chain organization does not have permission or authority to extend into the buy and sell-side areas of the supply chain. The politically charged nature of these terms is a barrier to gaining opportunity from these forms of disruptive technologies. This is why an influential business leader needs to oversee/govern analytics (to manage cross functional issues/needs) 
Issue #4. A different destination. To invest in big data supply chains, we often have to take a 180 degree turn from the generally accepted programs of ERP and APS. We have to open ourselves up for a different approach, and work in co-development with new vendors with a focus on innovation. It is an unproven road with dead-ends, uncertainty, and lots of opportunity.
 It also challenges the metrics for project success.  Yes, it is the kind of project that the Chief Financial Officer (CFO) hates. New investments in new technology with an undefined ROI….  (Conservative companies want tried and true projects with a conservative ROI.) It is a stark departure from the known world where organizations are on a fixed march to implement systems from the land of the Rs & Ps: ERP, MRP, DRP, APS, CRM, SRM. These projects are targeted, the resources are in place, and the project plans are neatly filed.  The barrier is why change?
Issue #5. A different vision. Supply chain investments historically have focused on improving efficiency.  Supply chains respond. It is seldom an intelligent response.  These new approaches, allow the supply to learn and predict. This machine to machine learning is a radical shift for supply chain leaders. We can learn the impacts in the financial and insurance industries where technology enabled a continuous learning environment, allowing the organization, to listen, learn and then drive an intelligent response.  Based on many- to-many rules mapping (versus traditional one to one fixed mapping), the new approach allows the learning to be around the clock and across geographies. The larger challenge will then become change management.  Organizationally, we don’t know how to listen and learn.  And, within organizations, not all people want to be measured or even share their data.

 Dusting off the Stoop to see the Facts.

 So, how do you justify the change?  Let’s start by facing the facts. 
There is a great opportunity. Organizations have an explosion of data volume and types that is just beginning to be used. And, in many cases, they are hidden in silos –marketing and sales– largely unknown by IT.   This data combined with new horizontal processes built from the outside-in can sense product recalls, identify market opportunities and improve new product launch by 60%.  It is often hard for supply chain leaders to understand the power of quicker data to drive better and more-timely decisions until it happens.  Why?  For most companies, the unfortunate reality is that enterprise architechitures are a mess, with most companies using Excel and Access as primary planning tools. There is a lack of trust that their IT group can ever build anything to fix the problem.
Business are trying to manage frayed ends. Customer Relationship Management (CRM) and Supplier Relationship Management (SRM) will be little help in seizing the opportunity for big data supply chains.   There is just no GOOD place to put the data in these systems. And, the goal is different.  (Each of these applications focuses on enterprise efficiency, and is not an appropriate adaptors to connect supply chain to supply chain to create value networks.)  As a result, companies that force this external data into CRM and SRM will find that they are navigating a dead-end path. 
What is needed is a shift in focus. As supply chain teams mature, the focus shifts from vertical process excellence, to cross-functional and horizontal processes to deliver a supply chain strategy. These horizontal processes gain more value from external data sources. Examples include Sales and Operation Planning (S&OP), Revenue Management, Supplier Development, Demand Sensing and shaping, etc. The combination of structured and unstructured data gives a more holistic view to make these horizontal processes more effective.
Forward thinkers are in search of a data model. Enterprise Resource Planning (ERP) is insufficient as an enterprise data model and many EDWs don’t address the requirements to use and drive insights from unstructured data. As a result, companies are working with IBM, Informatica, and Teradata to build a data model for the enterprise and design the right product, customer and supplier metadata elements.
The promise of integration is unfilled. Enormous integration challenges still abound in companies with structured data, and it is even greater with unstructured data.  This is true even in companies that have standardized on a common ERP system. The proliferation of data sources from outside the enterprise external to ERP will make this issue worse before it gets better.  The emerging technologies associated with Big Data Supply Chains offers hope.
Corporate systems are a mess. While the goal was to standardize on ERP and eliminate disparate systems, this was largely a pipe dream. It has not happen. For example, in the consumer products industry, the 2008 study by GMA IT committee supports that the average CPG Company greater than 1 Billion in revenue has more than 640 systems. The pendulum has also started to swing back to best of breed applications as business leaders become more influential in purchase decisions. There is more focus on what meets business needs best vs. what fits best with the existing technology stack.
Computing power is here. We are not using it. Computing power has grown 20X since the market inflection to client server in the early 1990s. The evolution of “R and P” technologies has largely ignored the possibility of what can be done with parallel processing and in-memory capabilities.  These topics are Greek to most business leaders but they need to understand the implications even if they have no desire to understand the technology.
 Time to start.  Calling all visionaries. Most companies lack inspiration. Manufacturing companies are not pushing the envelope of 1990’s BI capabilities, much less the concepts for big data supply chains. As a result, we will be forced to learn new concepts from ecommerce, financial, hospitality and insurance industry players.  Manufacturers, across all industries, are very early in the thinking of what can be possible.  The pace of adoption and gaining competitive advantage will be gated by the lack of business visionaries and their ability to pull their noses off the grindstone to envision how this can benefit the business. In a down economy even visionaries have a tendency to circle the wagons and focus on execution. One way around this is to cull out small battles that can be won quickly that can be used to “market” the vision. Many smaller BI vendors have used a similar strategy to great success during economic turmoil.
The cost is going down. It is no longer just for the few. The beauty of the new techniques is that the cost of computing is going down, and the techniques are no longer just for the VERY large corporations. Cloud, mobile, in-memory, and vertical or functionally targeted applications and services are making the barrier to entry much lower than ever before.

Guiding Principles:

 In tomorrow’s blog, I share definitions for the supply chain leader’s big data supply chain vocabulary.  Words to know to understand what is happening.  But, before we jump into the fray, I want to be sure that we are grounded in guiding principles to take this work on deliberate, systemic path versus a splintered projects that lead to no where.

• Build cross-functional teams to unleash the power. All too often, the groups that know the most about disruptive technologies are adjunct to eCommerce teams. Make this mainstream and challenge the group to think about what new processes could be if they build horizontal processes with minimal latency from the outside-in.
• There is no data mart cheap enough. This was a quote that I heard recently from eBay. I think that it is very true. Our foray into data marts is largely driven by a project approach versus a deliberate, and conscious choice on building effective value networks.
• Focus on meta-data design and master data will be easier. Try new master data techniques.  While I hear many business users fret about master data issues, many times the issue is lack of attention to metadata design (especially customer, product and supplier data). I also see innovators attempting new techniques to solve the master data issues: bypassing traditional techniques by indexing their data for rapid assembly. This gives flexibility to embrace the differences between master data registry and master data reference.
• Never let anything come between the user and their data. Empower the business user by focusing on self-service. To maximize the use of data for insights, empower the business users to directly use the data and even manage it (mapping, extracting etc.) themselves. Who knows the data better than the user, right?  Focus on training and design to enable self-service by the line of business user.
• Write once and read many times. In big data supply chains, focus on one system of record. Everyone has the moments when they show up at a business meeting only to argue about “whose report has the right data”.  Solve this problem by writing once and using many times. 
• Success. While many companies feel that success happens when a project is installed on time and on budget, I feel that success happens when line of business managers USE the systems. For most companies today, that use Excel and Access, after pouring millions of dollars into “R & P” systems, this should be a lesson learned and a mistake that should be avoided.  I feel that adoption, usage, and user satisfaction are better success criteria.

 So, even though in subsequent days, we are going to explore new techniques and value propositions for BIG DATA SUPPLY CHAINS, don’t forget these principles.  I give you a challenge: use the opportunity in front of us to harness the benefit from big data supply chains and new forms of predictive analytics to redefine your systems to better align with the generally accepted BI principles listed above. While you may throw away your Rs and Ps, don’t forget your Ps and Qs.
So as you read this series, do you think that BIG DATA SUPPLY CHAINS make APS, CRM and SRM obsolete or do you think that we can apply the principles and techniques of BIG DATA SUPPLY CHAINS to redefine these applications to take the next ascent of the climb on the journey of supply chain excellence? 
 For more on big data supply chains, you may want to reference these prior blog posts: 
SAP HANA: http://www.supplychainshaman.com/new-technologies/raving-ranting-and-craving/ 
The Challenges in Consumer Products: http://www.supplychainshaman.com/new-technologies/march-of-the-penguins/ 
An Introduction to Big Data Supply Chains: http://www.supplychainshaman.com/altimeter-group/user-in-the-era-big-data-supply-chains/ 
Supply Chain Listening: http://www.supplychainshaman.com/new-technologies/can-you-listen-2/ 
Use of Social in Supply Chains: http://www.supplychainshaman.com/new-technologies/more-than-just-a-f-word/ 
Future of Supply Chain Technologies: http://www.supplychainshaman.com/page/4/?s=listen+and+learn 
Listen and Learn: http://www.supplychainshaman.com/supply-chain-excellence/put-your-money-where-your-mouth-is/#more-202