Agile Integration: Part One--How Data Integration Challenges Slow Down Implementation of Supply Chain Planning and Optimization Systems
on Feb 23, 2017
Cleaning up the data and creating workable integrations to ERP and other source systems is almost always the number one obstacle to rapid implementation of supply chain planning and optimization systems. Here we look at the challenges involved, and start to explore a more agile approach to integration.
Full Article Below -
( This article is excerpted from the complimentary report Agile Integration: Accelerating Time-to-Value in Supply Chain Implementations,
available for download here. )
No matter how powerful the algorithms and intelligence are in a supply chain planning and optimization system, the results can only be as good as the data fed into those engines. The vast majority of the data consumed by these systems resides outside that specific system; mostly in the firm’s ERP systems, but also in various other execution systems such as WMS, TMS, procurement systems, as well as other supply chain planning systems (large companies often have more than one supply chain planning system, for different purposes, divisions, or regions). Increasingly, that data is coming as well from Internet-of-Things (IoT) and sources outside of the enterprise, such as weather, traffic, market data, trading partners’ systems, social media, and other information. Thus, having agile, rapid-implementation integration tools is critical to taking advantage of rich, diverse sources of data in order to unlock and realize the full potential of supply chain planning and optimization systems.
Agile Integration: Key to Unlocking the Power of Supply Chain Systems
Single ERP Vendor Approach Does Not Solve Supply Chain Integration
Companies that have committed to a strategy of using a single ERP vendor’s suite for all their applications—including using the ERP vendor’s supply chain planning and optimization modules—often do so on the hope that they will get deep, out-of-the-box integration between all of those components. They believe or hope that the ERP vendor’s supply chain planning module is tightly pre-integrated with the core ERP system, including all the relevant master data, and the other components from the ERP vendor, such as WMS, TMS, and procurement. Unfortunately, that is often not the case. Such a level of pre-integration often doesn’t exist and/or the company is running multiple ERP instances, often with customizations per instance.
Lack of Pre-built ERP-Supply Chain Integration; Different Architectures and Platforms
In too many cases, an ERP vendor’s supply chain suite was developed completely independently and on a different architecture1 than their core ERP system. With two development groups operating independently, the integration between the two systems is in many cases incomplete, requiring a lot of additional effort (and consultants) to make it all work.2
Multiple ERP Instances
In the 1980s and ‘90s, a big part of ERP’s appeal was the promise that one system could ‘do it all.’ That myth has since been largely busted; the vast majority of large enterprises have many differentERP systems,3 scores of best-of-breed systems, and hundreds of minor systems. The multiplicity of systems is driven by different business units’ desires for independence and systems that meet their specific needs, as well as a steady flow of mergers and acquisitions bringing in new business units that already have different systems.
Faster, Easier Integration Tips the Scales to Best-of-Breed Solutions
It has become steadily less costly to integrate best-of-breed systems. In the ‘80s and ‘90s, data was generally passed between systems using either ‘swivel chair integration’4 or hand-crafted integration code, making for very expensive and brittle integrations. Integration platforms have come a really long way since then, steadily lowering the barrier to a multiple best-of-breed suite strategy.
Making Mega-Mergers Work
A good example of how a supply chain solution provider can help with integration is Logility’s role in the merger of two very large F500 companies. They have taken on the responsibility of integrating supply chain planning and optimization into the ERP and other systems of the newly combined entity and doing it in a timely manner. This allows the line of business people to focus on the business integration and doing revenue-producing work to sustain them through the multi-year process of integrating the two firms. It also frees up internal IT resources to focus on more fundamental architectural and policy integration, instead of spending their time on supply chain application integration.
Not All Best of Breed Integration is Equal
However, not all best-of-breed vendors have the same level of integration capabilities. This is critical when assessing a solution provider, because rapid integration capabilities are a foundation for business agility. Too many best-of-breed providers still use outdated integration approaches, with lots of project-specific work required, resulting in complex, lengthy, risky, and costly integration projects. In contrast, some best-of-breed solution providers have become ‘best-of-integration providers’ as well, bringing together a set of purpose-built technologies, templates, and methodologies that dramatically shrink complexity, cost, duration, and the amount of in-house resources required for integration projects.
Agility and speed in integration of best-of-breed systems is highly valuable in keeping a business competitive over time; whether it involves integrating a new business from M&A activities, or quickly adopting new systems to gain new competitive capabilities, or upgrading existing systems regularly without disruption5 to keep up with the latest and greatest technologies. Thus ‘time-to-integration’ should be a key criterion when selecting a supply chain solution provider. Each month shaved off of implementation is one month more of value and competitive advantage realized from the system.
Challenges for Supply Chain Integration
Integration, Still the Long Pole in Supply Chain System Implementations
Despite the tremendous progress in integration tools, integrating data across multiple systems is still a difficult job and is often the critical path of an implementation project. Anything that shrinks time-to-integrate is likely to shrink overall project duration and time-to-value proportionately. That is part of why rapid integration is so valuable. Here are some of the challenges that need to be overcome to realize rapid integration:
Lack of internal resources—Integration projects often relying on internal resources (IT and end user), which are in short supply, thus extending the project. Anything that can lighten the internal project team’s load can help shorten implementation times.
ERP’s supply chain modules not pre-integrated—As mentioned, even the ERP vendor’s own supply chain module often requires a tremendous amount of work to integrate.
Multiple ERP systems or instances—The integration challenge is compounded when there are many ERP instances and/or a mix of ERP solutions from different vendors. When different instances have been customized, then usually each integration needs to be customized too.
Inadequate testing—Cutting corners on testing to meet deadlines can potentially create disastrous false starts, turning what could have been a killer, winning project into a pariah that no one wants to be associated with.
Poor sustainability/adaptability—Integration is not a ‘one and done’ effort. It must be made sustainable as the systems and data around it continually evolve over time. If it requires a significant new integration project to adapt each time changes are needed, then those integrations are delayed and the supply chain system starts losing value and usefulness, sometimes quickly and dramatically. When the solution integration was done largely using internal resources, with a custom design, it is far too common that the solution integration becomes unsupportable once the key resources leave or change jobs. As the integration becomes unsupportable, it will often take the entire solution down with it.
Global Enterprises’ Integration Challenges
Global businesses face several additional challenges when it comes to integration of their supply chain systems:
Short or non-existent daily time window for moving data—Depending on the approach taken, the integration system may need to wait until all data across the globe has been finalized for the day. That can create a very narrow window of time to extract the data for a firm with global operations.
Huge data volumes—Supply chain planning and optimization can involve huge amounts of source data. For example, for store/SKU inventory levels, a chain with 3,000 stores and 25,000 SKUs per store will have 75 million store/SKU inventory records to be updated daily. The inclusion of IoT data makes this data explosion even bigger, as enterprises track potentially billions of individual events (over time) across the enterprise.
Lack of data quality and completeness—Poor quality or incomplete data leads to poor quality outputs and thereby lack of confidence in the answers the planning systems are giving. Optimization engines are only as good as the data they receive. Once experienced people start seeing bad answers, they no longer trust the results, leading to resistance, rebellion, and workarounds, which prevents the value of the investment from being realized. Cleaning up existing data sources is usually the biggest part of an integration project. And for sustainability, the system needs the ability to continually validate that clean, complete, properly formed, within-range data is flowing into the supply chain systems, so that any problems are detected and corrected early. Ideally, this validation is done at the source, so that bad data is corrected before even entering the source system. Nevertheless, a second line of validation defense should be done as the supply chain system ingests the data.
Incomplete supply chain model/representation—While validation of individual data elements is a challenge, it is often even harder to validate that the supply chain model described by the sum of the individual data elements is meaningful. If a supply chain contains products that go through ten different stages, but there is no linkage in the systems between, say, stage six and stage seven, then it will be impossible for an optimization engine to generate a meaningful solution. A purpose-built integration solution must include technology that allows those types of conditions—the gaps and inconsistencies in the end-to-end supply chain model6—to be identified and resolved.
Figure 1 illustrates these challenges and the timeframes associated with a typical approach to integration, contrasted with the accelerated timeframe that is possible with an agile approach.
Figure 1 – Traditional Approach vs. Agile Approach to Integration—Compressing Time-to-Value
In part two of this series, we will discuss what to look for in an integration platform that has been purpose-built for supply chain planning and optimization. We also describe a potential roadmap to agile, incremental integration and value realization.
1 Sometimes a different architecture was selected for a good reason, for example to provide an in-memory architecture to enable more rapid supply chain planning and optimization. That decision may have been necessary because the original core ERP was not built on an in-memory architecture. -- Return to article text above
2 Some ERP vendors are making big investments to move their different systems onto common platforms going forward, but in many cases, they still have a long way to go. In any case, if you are still using their legacy ERP platform, their new platform may not be of much help in getting your supply chain solution up and running now. -- Return to article text above
3 Even when a company has standardized on SAP or Oracle or another platform, they often have many instances of that ERP system (up to 30 or more for a large corporation). -- Return to article text above 4 ‘Swivel-chair integration’ refers to having an employee read information from one screen and then swivel to another screen to key it into the other system. In practice, they were often reading it from a printout. In the ‘90s, enterprise application integration platforms were nonexistent or in their early infancy. -- Return to article text above 5 This requires a sustainable approach to integration, i.e. one that provides rapid integration of new systems and simplified accommodation of changes to any of the systems already integrated. -- Return to article text above 6 The end-to-end model of the supply chain is embodied by the data representing all the various process steps, across all the various systems involved in executing an end-to-end supply chain process. -- Return to article text above
To view other articles from this issue of the brief, click here.