reference architectures for manufacturing - Literature Library

defiantneedlessNetworking and Communications

Oct 23, 2013 (4 years and 8 months ago)


Industry adoption of EtherNet/IP
for control and information resulted in the wide
deployment of standard Ethernet in manufacturing.This deployment acts as the
technology enabler for the convergence of manufacturing and enterprise networks.By
gaining timely access to production key performance indicators (KPIs) at the right
levels,manufacturers benefit fromnetwork convergence.Information convergence
between manufacturing and business systems also enables greater business agility and
opportunities for innovation.
This technology and network convergence creates an unclear demarcation line for
network ownership.Groups that traditionally had limited interaction within
manufacturers now collaborate.To support this network convergence,controls
engineers and Information Technology (IT) professionals experience both
organizational and cultural convergence as well as share best practices.The
emergence of manufacturing IT,distinct fromenterprise IT,takes this collaboration
to a new level.
To support and accelerate this network convergence,Rockwell Automation and Cisco
collaborated to develop Reference Architectures for Manufacturing.These resources
provide users with the foundation for success to deploy the latest technology by
addressing topics relevant to both engineering and ITprofessionals.Reference
Architectures for Manufacturing provides education,design guidance,
recommendations and best practices to help establish a robust and secure network
infrastructure that facilitates manufacturing and enterprise network convergence.
This whitepaper outlines the recommendations and best practices described with the
Reference Architectures for Manufacturing.At the end of this whitepaper is a listing
of additional reference material.This listing includes resources not specifically
described within this whitepaper.For additional information on Reference
Architectures for Manufacturing,see notes 1 and 2 within the listing on the last page
of this whitepaper.
Control and Information Convergence
Convergence is not a new concept.For example,companies often undergo
convergence through expansion,mergers and acquisitions.Enterprise-wide systems
unite disparate business systems into a common enterprise resource planning (ERP)
system.Finally,users converge voice,video,and data into a common information
In the manufacturing industry,islands of automation for production and control
systems have increasingly converged into an integrated plant-wide control and
information platform.Users also unite disparate batch,continuous process,discrete,
safety,motion,and drive control industrial network technologies into a
multidisciplined industrial network by utilizing EtherNet/IP,a standard Ethernet
Wide deployment of EtherNet/IP in manufacturing triggered migration fromthe
traditional 3-tier network model to a converged Ethernet model,as shown in Figure
1.Convergence has not flattened the network model.Segmentation of functions,
geographic areas,and security for domains of trust requires a multi-tier model.
he traditional 3-tier network model evolved during the early days of Ethernet.
Characteristics such as collision domains,half-duplex and 10Mbps limited Ethernet
usage in production control applications.Proprietary,vendor-specific industrial
networks proliferated early on,until organizations like ODVAbegan promoting a
Common Industrial Protocol (CIP
By dividing a network by function and geographic area into smaller local area
networks (LAN),the 3-tier network model provides natural segmentation.This
lessens the impact of traffic management and security.By connecting devices such as
drives and robots with a controller,a device-level network controls,configures,and
collects data fromthese intelligent devices.Adevice-level network in one area does
not typically interact with other device-level networks.By acting as a backbone for
device-level networks,control networks interlock controllers and provide
connectivity to supervisory computers.Agateway maps information fromthe
manufacturing systems to the enterprise systems.The manual,store-and-forward
mapping mechanismrequired significant implementation and support efforts.
Figure 1
The naturally information-enabled,converged Ethernet model eliminates the need for
dedicated gateways.Although the technology has converged,the model has not
flattened.Data access fromanywhere at anytime presents a new challenge.
Manufacturers must protect their assets fromboth internal and external threats
(people with good intentions that make mistakes and those wishing to inflict harm)
because users typically know how to plug into Ethernet.No longer isolated in the
manufacturing realm,industrial networks make manufacturing computing and
controller assets susceptible to the same security vulnerabilities as their enterprise
Plant-wide networking with Ethernet technology requires planning and structure.
Establishing smaller LANs,to shape and manage network traffic as well as creating
domains of trust that limit access to authorized personnel requires a multi-tier,
segmented methodology.
Traditional 3-Tier
Manufacturing Network Model
Converged Ethernet
Manufacturing Network Model
Built on Industry Standards and Methodology
esigning and deploying a robust and secure network infrastructure requires a well-
planned roadmap.The manufacturing process dictates usage of equipment such as
sensors and actuators as well as their geographic deployment.By consulting
operations,users can determine information flow requirements.Users should also
identify what production information the business systemneeds.For example,a
business systemmay require KPIs or regulatory compliance data.Finally,the
roadmap should address standards implementation for common terminology,
methodology,and best practices.
Reference Architectures for Manufacturing are built on technology and
manufacturing standards common between ITand manufacturing.These include
technology standards such as IEEE’s 802.3 standard,unmodified Ethernet,Internet
Engineer Task Force (IETF) Internet Protocol (IP),and ODVA’s CIP.Additionally,
Reference Architectures for Manufacturing uses manufacturing standards to establish
a Manufacturing Framework as shown in Figure 2.This framework establishes a
foundation for network segmentation for traffic management and policy
enforcement,such as security,remote access,and Quality of Service (QoS).The
framework uses standards such as the ISA-95 Enterprise-Control SystemIntegration,
ISA-99’s Manufacturing and Control Systems Security,and the Purdue Reference
Model for Control Hierarchy.
Figure 2
Manufacturing Framework
Rockwell Automation and Cisco share a common technology viewby supporting the
facilitation and acceleration of network convergence as well as the promotion of
standard,unmodified Ethernet.In addition to jointly serving as principle members of
ODVA,the companies individually participate in standard organizations like ISA.
For additional information about ODVA,see note 4.
Throughout the Reference Architectures for Manufacturing,terminology refers to
“layers,” “levels,” and “zones.” The Open Systems Interconnection (OSI) seven-layer
reference model defines layers – e.g.layer 1 for Physical,layer 2 for Data Link,layer 3
or Network.Layer 2 devices forward data and provide network services based on
Data Link layer characteristics such as Media Access Control (MAC).Layer 3
devices forward data and provide network services based on IP.For additional
information on the OSI network model,see note 5.
Figure 2 depicts levels and zones of the Manufacturing Framework.Both ISA-95 and
the Purdue Reference Model for Control Hierarchy segment industrial control
devices into hierarchical “levels” of operations within a manufacturing facility.Using
“levels” as common terminology breaks down and determines plant-wide information
flow.For enhanced security and traffic management,ISA-99 segments levels into
“zones.” Zones establish domains of trust for security access and smaller LANs to
shape and manage network traffic.For additional information about ISA,see note 7.
The Manufacturing Framework groups levels into the following zones for specific
Enterprise Zone:
Levels 4 and 5 handle ITnetworks,business
applications/servers (,enterprise resource planning – ERP) as well as
Demilitarized Zone (DMZ)
– This buffer zone provides a barrier between the
Manufacturing and Enterprise Zones,but allows for data and services to be shared
securely.All network traffic fromeither side of the DMZ terminates in the
DMZ.No traffic traverses the DMZ.That is,no traffic directly travels between
the Enterprise and Manufacturing Zones.
Manufacturing Zone:
Level 3 addresses plant-wide applications (e.g.historian,
asset management,manufacturing execution systems - MES),consisting of
multiple Cell/Area Zones.
Cell/Area Zone:
Levels 0,1 and 2 manage industrial control devices
(e.g.controllers,drives,I/Oand HMI) and multidisciplined control applications
(,batch,continuous process and discrete).
Shaping and Managing Network Traffic
Developing a robust and secure network infrastructure requires protecting the
integrity,availability and confidentiality of control and information data.Users
should address the following when developing a network:
• Is the network infrastructure resilient enough to ensure data availability?
• How consistent is the data?Is it reliable?
• How is data used?Is it secure frommanipulation?
Reference Architectures for Manufacturing provides recommendations,design
guidance,best practices,methodology (Figure 3) and documented configuration
settings.This helps establish a robust and secure network infrastructure for control
and information data availability,integrity,and confidentiality.Built on industry
standards and a future-ready network foundation,Reference Architectures for
Manufacturing addresses today’s applications like safety through CIP Safety
tomorrow’s applications like motion through CIP Motion
,time synchronization
through IEEE 1588 precision time protocol (PTP) with CIP Sync
ncorporation of voice over IP (VoIP) and video on demand (VOD).
ITprofessionals frequently use reference architectures as a common concept and tool
within the enterprise.Fromretail companies to data centers,Cisco develops reference
architectures for a variety of industries and applications.Reference Architectures for
Manufacturing,as shown in Figure 3,incorporates the Rockwell Automation
Integrated Architecture
and Cisco Ethernet-to-the-Factory,a Cisco Validated
Design.For additional information on the Integrated Architecture,see notes 1 and 8.
Figure 3
Reference Architectures for Manufacturing
To align with the Manufacturing Framework shown in Figure 2,Reference
Architectures for Manufacturing utilizes the Campus Network Reference Model.
Common with enterprise networks,this multi-tier model naturally segments traffic
into three main tiers:core,distribution and access.
Layer 2 access switches aggregate control devices within the Cell/Area Zones.
Additionally,layer 2 provides network services such as switching,resiliency via spanning
tree protocol (STP),Quality of Service (QoS),virtual local area network (VLAN) and
security.Multilayer (layers 2 and 3) distribution switches reside in the Manufacturing
Zone (level 3),brings together access switches fromthe Cell/Area Zones and provide
network services.Services include layer 2 and 3 switching,routing,load balancing,
resiliency via Hot Standby Routing Protocol (HSRP),QoS and security.Finally,the
core switch aggregates distribution switches and provides high speed switching.Like
Reference Architectures for Manufacturing,ITprofessionals frequently use
core/distribution/access as a common concept and tool within the enterprise.
Designing a resilient network infrastructure with lowlatency and jitter increases the
vailability and integrity of control and information data.Latency,or delay,represents the
time elapsed fromwhen one device transmits data until another device receives it.Jitter
epresents the variation of delay.Converging multidiscipline control and information
traffic into a common industrial network requires reducing latency and jitter.
To reduce network latency and jitter,Reference Architectures for Manufacturing
recommends segmenting and prioritizing network traffic.Segmentation reduces the
impact of broadcast and multicast traffic.
Reducing network latency and jitter starts with the Cell/Area Zone.When designing
the Cell/Area Zone,users should create smaller layer 2 Cell/Area Zone network
segments organized by function or geographic area.Restrict data flow out of the
Cell/Area Zone unless plant-wide operations explicitly require it.Each Cell/Area
Zone should be implemented with a dedicated VLANand IP subnet.VLANs
segment network traffic and help restrict broadcast and multicast traffic as well as
simplify security policy management.As a best practice,use the layer 3 distribution
switches to route information between Cell/Area Zone VLANs and plant-wide
operations in the Manufacturing Zone.Avoiding large layer 2 networks helps simplify
network management.For additional information on VLANs,see notes 1,2,and 5.
Network topology choice impacts the availability and integrity of control and
information data.Figure 3 depicts the bus/star,ring and redundant star topologies
described in Reference Architectures for Manufacturing.Since applications drive
topology choice,users should address key considerations.These include application
performance requirements,network latency and jitter tolerance,downtime and mean-
time-to-repair (MTTR) tolerance as well as future upgrade and expansion
requirements.Fromright to left,Figure 3 depicts increases to network resiliency,
modularity,flexibility and implementation complexity.As a best practice,implement
a resilient topology such as the recommended redundant star topology.Redundant
star provides natural segmentation,shapes traffic to help reduce latency and jitter by
improving data integrity as well as offers the resiliency required for greater data
availability,which helps reduce downtime.Modularity of the redundant star also
increases scalability and flexibility for network expansion and upgrades.
Not all network traffic is created equal,nor should users treat it equally.To minimize
application latency and jitter,control data should have priority within the Cell/Area
Zone.Quality of Service (QoS) gives preferential treatment to some network traffic
at the expense of others.Control data is more sensitive to latency and jitter than
information data.To minimize latency and jitter,users should apply QoS to control
data within the Cell/Area Zone.Before implementing QoS within the
Manufacturing Zone,use a multidiscipline teamof operations,engineering,ITand
safety professionals to establish a QoS policy for the Manufacturing Zone.This
policy should support the needs of operations,including what to apply QoS to and
when.Additionally,the multidiscipline teamshould understand that this policy may
differ fromthe enterprise QoS policy.Enterprise QoS policies commonly give
priority to VoIP.
Although not specifically addressed within Reference Architectures for Manufacturing,
developing a robust network infrastructure requires proper design and implementation
of an industrial Physical layer.Physical media,layer 1,within the Cell/Area Zone is
subjected to environmental and noise conditions not found in the enterprise.These
conditions can impact availability and reliability of data,introducing latency and jitter.
For additional information on physical media planning and installation,see note 6.
Recommendations and best practices for the Cell/Area Zone include:
• Shape and manage traffic by implementing smaller Cell/Area Zones with a
separate VLANand IP subnet per Cell/Area Zone.
• Use managed layer 2 access switches to segment traffic with VLANs,prioritize
traffic with QoS,implement security policies with port security and access control
lists (ACL),and provide diagnostics.
• Utilize a redundant star topology for greater network resiliency and modularity,
along with rapid spanning tree protocol (RSTP) to manage loops.Implement
multiple spanning tree (MST802.1s) version of RSTP (802.1w) to support usage
of multiple VLANs.For additional details on MSTand RSTP,see note 2.
• Lower network latency and jitter by using Gigabit Ethernet ports for trunks and
uplinks,VLANs to reduce broadcast traffic,Internet Group Management
Protocol (IGMP) to reduce multicast traffic,QoS to prioritize traffic and
redundant star topology for natural segmentation.
• For additional information on these best practices,see note 2.
The Manufacturing Zone contains all systems,devices and controllers critical to
controlling and monitoring plant-wide operations.This zone includes Site
Manufacturing Operations and Control functions (level 3) as well as multiple
Cell/Area Zones.To preserve smooth plant-wide operations and functioning of the
systems and network,this zone requires clear isolation and protection fromthe
Enterprise Zone via the Demilitarized Zone (DMZ).All manufacturing assets
required for the operation of the Manufacturing Zone should remain there.Assets
include Rockwell Automation FactoryTalk® Integrated Performance and Production
Suite as well as other applications and services,such as Active Directory,DNS,and
Level 3,Site Manufacturing Operation and Control,has a dedicated network
segment within the Manufacturing Zone and contains the FactoryTalk servers.Users
should assign this network segment with its own IP subnet and VLAN.The
FactoryTalk servers connect to a dedicated multilayer access switch,which aggregates
into the layer 3 distribution switches.The distribution switches act as the network
segment’s default gateway.To provide redundant default gateways to the Cell/Area
Zones,distribution switches should use Hot Standby Routing Protocol (HSRP) or
Gateway Load Balancing Protocol (GLBP).Distribution switches will route all
traffic to and fromthe level 3 network segment.
Recommendations and best practices for the Manufacturing Zone include:
• Keep FactoryTalk within the Manufacturing Zone.For additional information,
see note 10.
• Keep replicated services such as DNS,Active Directory and DHCP within the
Manufacturing Zone.
• Implement a level 3 (Site Manufacturing Operations and Control) network
segment with its own IP subnet and VLAN.
• Use layer 3 distribution switches to route between Cell/Area Zone VLANs and
the level 3 network segment VLAN.
• Use HSRP or GLBP on the distribution switches to provide redundant default
ateways to the Cell/Area Zones.

For additional information on these best practices,see note 2.
Securing Manufacturing Assets
The recommended “defense-in-depth” approach,depicted in Figure 4,helps to
address internal and external security threats as well as helps provide confidentiality
for control and information data.By utilizing multiple layers of defense (physical and
electronic) at different levels within manufacturing,this approach addresses disparate
types of threats.No single technology or methodology fully secures industrial
networks.Acomprehensive security model should be designed and implemented as a
natural extension to the manufacturing process.Security should not be implemented
as an afterthought or bolt-on component.
For the purpose of this whitepaper,“defense-in-depth” layers for securing
manufacturing assets include:
• Physical Security:This limits physical access to authorized personnel for areas,
control panels,devices,cabling,the control rooms and other locations as well as
escorts and tracks visitors.
• Network Security:This contains the infrastructure framework,such as firewalls
with intrusion detection and intrusion prevention systems (IDS/IPS).
• Computer Hardening:This includes patch management and antivirus software
as well as removal of unused applications,protocols and services.
• Application Security:This contains authentication,authorization and audit
• Device Hardening:This handles change management and restrictive access.
• For additional information on “defense-in depth”,see notes 11 and 13.
Figure 4
Defense-in-Depth - Multiple Layers
The recommended Manufacturing Network Security Framework,utilizing “defense-
n-depth” is depicted in Figure 5 and includes:
• Manufacturing Security Policy:This security policy roadmap identifies
vulnerability mitigation.Amultidiscipline teamof operations,engineering,IT
and safety should develop this manufacturing security policy.
• Demilitarized Zone (DMZ):This buffer zone provides a barrier between the
Manufacturing and Enterprise Zones,while allowing users to securely share data
and services.All network traffic fromeither side of the DMZ terminates in the
DMZ.No traffic traverses the DMZ,which means that traffic does not directly
travel between the enterprise and manufacturing zones.
• Defending the manufacturing edge:Users should deploy stateful packet
inspection (SPI) firewalls (barriers) with intrusion detection/prevention systems
(IDS/IPS) around and within the industrial network.
• Protecting the Interior:Users should implement access control lists (ACLs) and
port security on network infrastructure devices such as switches and routers.
• Endpoint Hardening:This restricts access,prevents “walk up,plug in” access and
uses change management to track access and changes.
• Domains of Trust:Users should segment the network into smaller areas based on
function or access requirements.
• Physical Security:This restricts physical access to manufacturing assets and
network infrastructure devices.
• Security,Management,Analysis and Response System:This monitors,identifies,
isolates and counters network security threats.
• Remote Access Policy – For employee and partner remote access,implement
policies,procedures and infrastructure.For additional information on remote
access,see note 12.
Recommendations and best practices for securing manufacturing assets include:
• Deploy holistic security based on “defense-in-depth.”
• Conduct a security risk assessment,see note 15 for additional information.
• Develop a manufacturing security policy that support manufacturing operation
requirements based on enterprise security policy best practices.
• Implement a manufacturing network security framework to establish domains of
trust and appropriately apply security policies.
• Establish a DMZ between the Enterprise and Manufacturing Zones.
• Prevent traffic fromtraversing the DMZ.
• Use application mirroring within the DMZ to converge Manufacturing and
Enterprise Zone information,noted in next section.
• Harden computers and controllers,see note 13.

Utilize industry standards such as ISA-99.
• Leverage Rockwell Automation Network and Security Services,see note 15.
• For additional information,see note 13.
Figure 5
Manufacturing Network Security Framework
Information Convergence via the DMZ
Information convergence has helped provide manufacturers with greater business
agility and opportunities for innovation.With these opportunities,come challenges.
Manufacturing computing and controller assets have become susceptible to the same
security vulnerabilities as their enterprise counterparts.Protecting manufacturing
assets requires a “defense-in-depth” security approach.For additional details,see
notes 11 and 13.The best practices described within Reference Architectures for
Manufacturing,utilizing “defense-in-depth”,help to provide a robust and secure
network infrastructure facilitating information convergence between manufacturing
and business systems.
The first best practice calls for establishing a DMZ between the Enterprise Zone and
the Manufacturing Zone.As noted earlier,the DMZ is a buffer zone providing a
barrier between the Manufacturing and Enterprise Zones,but allows for data and
services to be shared securely.All network traffic fromeither side of the DMZ
terminates in the DMZ.No traffic traverses the DMZ.That is,no traffic directly
travels between the Enterprise and Manufacturing Zones.Finally,users should
ontain all manufacturing assets,such as FactoryTalk,required for manufacturing
operations within the Manufacturing Zone.
To maintain these best practices while allowing information convergence between the
Enterprise and Manufacturing Zones,Manufacturing Zone applications should
replicate data to an application mirror within the DMZ.Users should then replicate
the data fromthis application mirror to an application within the Enterprise Zone.
This can be either unidirectional or bidirectional.
Figure 6
FactoryTalk Transaction Manager and MSSQL Server
An example of data mirroring is shown in Figure 6.FactoryTalk applications that
utilize Microsoft SQL (MSSQL) server,for example can maintain the best practices
and methodology noted above.For additional information on FactoryTalk,see notes
9,10,and 11.Figure 6 also demonstrates that FactoryTalk Transaction Manager
provides two-way data exchange between tags,such as Logix Controller or
FactoryTalk View,and applications like an MSSQL server.These tags may contain
KPIs or other important data that needs to integrated into an enterprise application.
Since traffic cannot traverse the DMZ,an MSSQL server in the Manufacturing Zone
cannot directly transfer data to and froman MSSQL server in the Enterprise Zone.
This means that all traffic between the two zones must be initiated or terminated in
the DMZ.
Users should implement the methodology shown in Figure 6 to enable information
onvergence while maintaining DMZ best practices.The FactoryTalk Transaction
Manger with MSSQL server solution involves:
• The FactoryTalk Transaction Manager server (level 3) is configured to read/write
its SQL data to and froman MSSQL server (data mirror) located in the DMZ.
• The MSSQL server data mirror in the DMZ then replicates the data to and from
the Enterprise Zone MSSQL server.
• Business systems within the Enterprise Zone only access the enterprise MSSQL
The convergence of manufacturing and enterprise networks increases access to
manufacturing data,which assists manufacturers in making better business decisions.
This business agility provides a competitive edge for manufacturers that embrace
With these opportunities,come challenges.Network convergence exposes
manufacturing assets to security threats traditionally found in the enterprise.Users
also face an unclear demarcation of network ownership and cultural difference
between deploying enterprise and manufacturing assets.Implementing best practices
fromboth engineering and ITalong with the recommendations described in
Reference Architectures for Manufacturing will help users establish the secure and
robust network infrastructure needed to facilitate manufacturing and enterprise
network convergence
Additional Reference Material
1) Reference Architectures for Manufacturing Website
2) Design and Implementation Guide (DIG)1.2
3) Ethernet Design Considerations for Control SystemNetworks – ENET-SO001
5) Network Infrastructure for EtherNet/IP:Introduction and Considerations
6) EtherNet/IP Media Planning and Installation Manual
7) ISA-99,Industrial Automation and Control SystemSecurity
8) Rockwell Automation Integrated Architecture
9) FactoryTalk Website
10) FactoryTalk Positioning within Reference Architectures for Manufacturing Whitepaper
11) FactoryTalk Security Quick Start Guide
12) Remote Access Whitepaper
13) Securing Manufacturing Computing and Controller Assets Whitepaper
14) Rockwell Automation Knowledgebase -
15) Rockwell Automation Network and Security Services
Publication ENET-WP004A-EN-E-November 2008 Copyright 2008 Rockwell Automation,Inc.Printed in USA
EtherNet/IP,CIP,CIP Safety,CIP Motion and CIP Sync are trademarks of ODVA.FactoryTalk is a registered trademark of Rockwell Automation,Inc.
Integrated Architecture is a trademark of Rockwell Automation,Inc.