Achieving multivendor interoperability with open systems

• By Bill Lydon
• Cover Story

Summary

ExxonMobil collaboration helps create a converged IT/OT industrial control system based on open standards

By Bill Lydon

A big step toward achieving multivendor interoperability with open systems was demonstrated by CPLANE.ai and collaborator ExxonMobil this year: It proved that deploying and integrating general computing and Internet of Things (IoT) technology using open standards as part of the industrial control system (ICS) provides great benefits. This foundational work will allow deep integration of intelligent processing, driven by machine learning, to achieve more efficient, adaptable, and reliable next-generation systems.

The pilot project used open computer industry standards to demonstrate the automation of provisioning, initiation, and life-cycle management of open-architecture, multivendor industrial control systems. The pilot leveraged a number of standards, including the Open Process Automation™ Standard (O-PAS), OPC-UA, DMTF Redfish, IEC 61499, and OASIS TOSCA (Topology and Orchestration Specification for Cloud Applications).

This pilot proved using orchestration to deploy and integrate general computing, IoT, and process automation, using open standards as part of the ICS, optimizes and simplifies the management of multivendor systems while improving security and reliability.

Pilot overview

The pilot consisted of a heterogeneous mix of information and operational technologies (IT/OT) simulating a chemical processing plant. The system started with industrial IT compute devices (replacing legacy distributed control systems and programmable logic controllers) connected to an Ethernet network. The compute devices were also connected to specific industrial I/O networks (based upon the design of the plant). The CPLANE.ai industrial orchestrator was used to fully install and configure all process control software and control logic to bring the simulated plant to an operational state. This took approximately 10 minutes, compared to an estimated 50–100 person-hours by conventional methods.

Key findings of the project were:

• Multivendor, open process automation can be integrated into a holistic system using open system orchestration technology.
• Open standards, such as the O-PAS, make interoperability significantly easier to manage and more reliable to implement.
• An integrated, hybrid architecture of IT/OT digital assets can be managed in one framework.
• System orchestration is critical for accelerating innovation and adopting converged IT/OT systems, particularly in an open, multivendor, and interoperable control system.

Don Bartusiak, chief engineer for process control at ExxonMobil Research and Engineering said, “CPLANE.ai exceeded our expectations on what is possible in demonstrating the management of a converged IT/OT industrial control system. System orchestration is growing in visibility and importance within the Open Process Automation Forum, and many of the findings of this pilot will help us shape the evolution of our standards.”

Goals of the orchestration pilot

This pilot was conceived and executed in the context of rapid changes in industrial automation. Large industrial manufacturers like Merck, DuPont, Shell, BASF, Georgia Pacific, and ExxonMobil are making new demands on traditional process automation vendors for solutions that unify information technologies and operational technologies into a single system of management and control. They want these converged IT/OT systems to be open, interoperable, and inherently secure. Global standards bodies, such as the Open Process Automation Forum, OPC Foundation, and NAMUR, are developing the standards for these open systems.

The specific purposes and goals of the pilot were to:

• reduce the complexity and effort of deploying an ICS by orders of magnitude compared to conventional methods
• perform a full ICS deployment with little to no IT expertise
• conform as much as possible to open standards and practices as defined by the O-PAS
• demonstrate the automated provisioning and initiation of a multivendor, converged IT/OT control system from a pre-software-installation state to a fully operational and ready state—the “startup phase”
• demonstrate a significant reduction in the complexity and effort required to perform this “startup phase” of both IT and OT systems compared to existing, manual software installation processes
• perform this “startup phase” with little or no IT expertise from the perspective of the system operator
• prove that an orchestration platform can provide hands-off deployment of a converged IT/OT system that conforms to the open, multivendor specifications outlined in the O-PAS
• prove that DMTF Redfish can reliably provide physical system metadata to facilitate correct deployment of OT software applications
• specifically demonstrate the use of cloud technologies and standards like OASIS TOSCA for orchestration 

Life-cycle system management

Managing an automation system over its life cycle is a significant investment that includes adding field controllers and I/O points, updating and upgrading software and firmware, adding features, and reconfiguring control. Today users are faced with using separate tools from each distributed control system (DCS) and programable logic controller (PLC) vendor they have in their operations, which requires unique support, training, and software maintenance agreements. For greater efficiency and responsiveness, the computer, communications, and IT world have been using open orchestration tools and techniques so a single operator, or even no operator, can manage huge and complex digital infrastructures.

Industrial orchestration manages all compute elements, software stacks, control applications, networks, and containers as a single, integrated system. As next-generation industrial control systems transition to a rapidly maturing and increasingly complex digital technology stack, system orchestration customized for industrial systems is important for creating a system from open components from multiple vendors.

Steve Bitar, an automation leader at ExxonMobil Research and Engineering, said “this was an ambitious project from the beginning. And the results have really given us a vision for how complex open industrial control systems can be managed and automated.”

In this pilot, the CPLANE.ai orchestration platform was deployed to solve the complex digital life-cycle problems of managing large edge computing, distributed clouds, and Industrial IoT environments. It was customized to meet the needs of industrial systems. CPLANE.ai has been actively involved in the Open Process Automation Forum over the past three years and is an important contributor to the Open Process Automation Standard.

Pilot plant description

The pilot plant for the demonstration simulated a chemical mixing and heating process involving several plant assets: a reactor batch processor, a heat exchanger, product storage tanks, and a water chiller. The pilot infrastructure consisted of 14 individual compute devices that represented 13 distributed control nodes (DCNs) and a single advanced computing platform (ACP). The DCNs ran the control loops for the industrial process while the ACP hosted the human-machine interface (HMI) application and the IEC 61499 engineering design tool. The compute devices were a combination of different microprocessors (Intel X86 and ARM), with different configurations of RAM and storage, from different manufacturers. These devices were also divided between two locations, with approximately half in New York and the other half in California, to represent a truly distributed topology.

The compute devices were connected on the northbound side to an Ethernet network, while the southbound side was connected to a (simulated) Fieldbus network with simulated sensors and actuators. The two operational sites (Calif. and N.Y.) were connected through a VPN connection via a third site, the CPLANE.ai datacenter in California. The entire digital infrastructure was managed from a single location in California.

The pilot demonstration

The goal of the pilot was to use automation (via the CPLANE.ai industrial orchestrator) to correctly install all the software necessary to deploy the control system of a simulated chemical plant. The CPLANE.ai orchestrator should complete this “startup phase” operation in approximately 10 minutes. Manual installation of a similar system often takes a team of two or three engineers several days or a week.

1. The initial state of the control system:

• Sensors and actuators (simulated) were connected via an I/O bus or directly to the DCNs.
• IEC 61499 function blocks were already created and compiled for the process in the engineering design tool.
• All DCN and ACP devices were connected to an Ethernet network for both the control and data planes.
• All DCN and ACP devices were powered on and preloaded with Linux and DMTF Redfish client (simulated in this demonstration with CPLANE.ai client software).
• The CPLANE.ai orchestrator and the process automation system were running on the same control network on a dedicated X86-based device running Linux.
• The IEC 61499 engineering design tool was connected to the CPLANE.ai orchestrator.

2. The demonstration began with the HMI running, but showing that it was not receiving sensor data, nor did it have knowledge of the digital infrastructure underlying the control system.

3. In the first step of the three-step deployment process, the operator initiated discovery of the digital infrastructure with a single mouse click in the CPLANE.ai orchestrator. The orchestrator then polled all the digital devices on the control network, registering each device into the orchestrator’s information model database. Additional details of each digital device were ingested into the orchestrator’s information model using the DMTF Redfish protocol and other standard IT discovery protocols. This discover process takes approximately 90 seconds.

4. The orchestrator then built a digital topology model of the physical infrastructure including in-depth data describing the physical infrastructure and its state. This topology model will be used to make intelligent decisions on exactly where each software element of the control system will be installed. Now, the orchestrator effectively has a “digital twin” of the control system’s digital infrastructure in its memory.

5. The orchestrator could then display all of the digital infrastructure of the control system including rich detail for every connected device. Examples of details in the information model are:

• microprocessor type and manufacturer
• device manufacturer and model number
• available device RAM and storage
• utilized and available CPUs
• device IP address
• network zone
• OS type and version

6. The second step of the three-step deployment process was to “program” the CPLANE.ai orchestrator with all of the requirements for the control system’s “digital life cycle.” This “programming” step was performed by uploading one or more OASIS TOSCA documents that contained all the system requirements in a structured, reusable data format (YML). Examples of the requirements in an OASIS TOSCA2 document are:

• device requirements for software applications (e.g., microprocessor type, available RAM, OS type and version)
• policies that govern system life cycle (e.g., software applications that are prerequisites for other software applications, high-availability requirements, security zone requirements, affiliations with specific I/O locations)

7. The orchestrator was now programmed with the “system engineering knowledge” in its memory. Using sophisticated algorithms, it could make intelligent decisions about which application should be installed on which device. And, equally as important, the orchestrator understood exactly which steps must be taken in the correct order to install all of the software for the entire system in an automated and deterministic manner.

8. A converged IT/OT system like a modern, open process automation system can only be effectively and efficiently managed within a holistic framework. The orchestration platform merged all the requirements, policies, procedures, workflows, and state management into a single fabric. This single management fabric could now act in multiple dimensions (detailed below) with a deep understanding of dependencies and interdependencies of the entire digital infrastructure including:

• supervisory applications (HMI, historian)
• process control applications
• compute layer (OS, Docker, virtual machines)
• network security
• networking

9. The final step was to activate the orchestrator to deploy the ICS software. The operator simply pushed the “deploy” button, and the provisioning and installation operations commenced.

10. The orchestrator, in parallel, performed the following actions on all of the DCNs and ACP nodes of the system:

• examined the current state of each device as well as the entire system
• calculated the final desired state of each device and the whole system
• evaluated the policy constraints (the “rules”) of each device (networking, security, prerequisites, etc.)
• calculated all the steps necessary to deploy the entire system without violating any of the constraints. This calculation then programed a workflow engine to execute each step both in sequence and in parallel where permitted.

The workflow engine (internal to the orchestrator) took actions such as:

• called premade scripts (like Ansible and Python) to direct installation routines
• dynamically created custom scripts where needed to instantiate a change in state of a device
• cleaned up devices to allow for new software installations.

Simultaneously, the orchestrator monitored the digital infrastructure for events that signaled successful actions taken or unexpected errors that required the orchestrator to take action or perhaps notify the operator.

11. Some of the actions performed on the system by the orchestrator include the deployment of:

• the OPC-UA client and server instance on 12 DCNs
• the OPC-UA discovery services (LDS) on a single DCN
• Docker containers on 12 DCNs that will host the engineering run time
• the engineering run time on 12 DCNs (in their respective Docker containers).

Also, the orchestrator directed the IEC 61499 engineering design tool to install function blocks on specific DCNs once the engineering run-time environment was ready.

12. This deployment process took approximately five minutes to perform. The end result was a fully deployed control system.

13. Switching back to the HMI, the first thing that was noticeable was that there were values in the sensor displays showing that OPC-UA had been installed correctly, was connected to the right I/O, and was successfully sending data to the HMI.

14. Using the HMI, the (simulated) chemical process was started. Each DCN successfully executed its function block code to make the chemical process operate as expected.

15. This whole “startup phase” operation took approximately 10 minutes from the initial state as described above to a fully operational state where the HMI could initiate control functions.

Engineering principles applied to pilot demonstration

System startup (the “startup phase”) is not the end goal of using system orchestration. The primary goal is to maintain continuous plant operations during change, failure, or upgrade as specified by digital life-cycle policies. The “startup phase,” demonstrated in this pilot, lays the foundation for future phases to demonstrate continuous plant operations.

Automated management of complex digital systems has been done before. Converged IT/OT control systems must follow the success of other industries’ successful management of large, complex systems. Orchestration has been widely adopted to manage the life cycle of global telecommunications networks, automate the world’s largest cloud data centers, and to accelerate the deployment and maintenance of software applications for every large business.

Do not just “make it work.” To avoid costly dead ends, choose a standards-based framework that is designed for industrial automation, rather than the currently “hot” technology.

Key lessons from the pilot

1. “Systemness” for open systems is achievable using system orchestration. Multivendor, open process automation can be integrated into a holistic system using the technologies and techniques of orchestration.
2. Open standards make interoperability significantly easier to manage and more reliable to implement. These open standards will continue to evolve as technologies evolve, which will help to future proof your investment.
3. Integration with current OT solutions requires cooperation and collaboration from vendors but is achievable with spectacular results. Shift thinking from IT and OT to an integrated hybrid architecture of IT/OT digital assets managed in one cohesive framework.
4. System orchestration is critical for accelerating innovation and adoption of converged IT/OT systems.

CPLANE.ai is now conducting demonstrations of its digital infrastructure test bed with partners and members of OPAF. In the future, the company will apply the lessons from this pilot project to other OPAF-related test beds and pilots globally. Certainly, this pilot project illustrated a way to leverage open standards for life-cycle management of industrial control systems.

For a whitepaper describing the methods and results of the pilot, see www.cplane.ai/industrial_orchestration/whitepaper.

 

 

• Facebook
• Twitter
• Linked In