DOC - ILC Document Server

chinchillatidyNetworking and Communications

Oct 26, 2013 (3 years and 11 months ago)

113 views


Control System Architecture Model for ILC Reference Design

ILC Tech Note

March 19, 2007

Authors: Cla
ude Saunders, John Carwardine, Ned Arnold, Margaret Votava, Vince
Pavlicek, Ray Larsen, Bob Downing, Steve Wolbers, Bakul Banerjee, Schinichiro
Michizono, Stefan Simrock, Jim Patrick, Sharon Lackey, Paul Kasley, Paul Joireman,
K
ay Rehlich, Kazuro Furokawa, Erik Go
ttschalk, Frank Lenkszus, Ron

Rechenmacher


Abstract

The
ILC control system architecture is evolving as requirements are gathered and the
research and development effort gains steam. We present the
general purpose contro
l
system
architecture as of the time of writing the ILC Reference Design Report.

We do not
cover specialized subsystems such as timing, fast feedback, and machine protection.

The
rationale for establishing
architecture

at this stage is threefold: we requir
e a model from
which to make a cost estimate; we require a model to validat
e that we are meeting the
control system requirem
ents; and
we must communicate our vision to the technical
system groups early and often.

The

architecture described here

is not full
y
specified;

nonetheless

a number of distinct design choices are made
.

Future requirements gathering,
research, and development will complete the picture.


1

Introduction

The architectural model

of the ILC control system is presented here from both functiona
l
and physical perspectives. This mod
el has served to produce a cost

estimate, as well as to
document our coverage of the control system requirements to date.
We also connect the
functional and physical models by showing where various functions are (or cou
ld be)
physically realized.

While specific technologies such as Ethernet are
mentioned
, these
are ultimately intended to be representative technologies, and may well change by the
time the actual project construction phase begins.


2

Functional Model

The fu
nctional perspective combines software functions and network functions into a 3
-
tier model.
The 3
-
ti
er approach (or design pattern) is common

in modern software
architecture. This approach provides many benefits including separation of concerns, re
-
use, lo
ad management, and change management.
Older and smaller scale control systems
consist of only 2 tiers; the client or application tier, and the technical systems tier. The 3
-
tier model introduces an intervening services/mid
dleware tier where many essenti
al
shared functions reside [1].


Figure 1

below
details

the internal structure of the 3 tiers, as well as the networking
functions required within and between the tiers. The remainder of this section is de
voted
to elaborating on this figure.



Control System Architecture Model for ILC Reference Design

ILC Tech Note



FIGURE 1
:
Con
trol System Functional Model


2.1

Software Functions

2.1.1

Client Tier

The client tier consists of applications with which people directly interact (graphical user
interfaces or GUI). Applications will range from engineering
-
oriented control consoles to
high
-
level p
hysics control appli
cations to system
management applications. Engineering
-
oriented consoles are focused on the direct operation of the underlying accelerator
equipment in the front
-
end tier. High
-
level physics applications will utilize a blend of
services

from the service tier that in turn combine data from the front
-
end tier and
supporting data from the relational database.


2.1.2

Services Tier

From

the perspective of a

user of the client tier, the serv
ices tier is largely invisible.
The
goal of the services ti
er is to manage the execution of logic in the problem domain, and
leave the problems of user interaction and graphical presentation of data and status to the
client tier. The services tier provides services that coordinate many activities while

Control System Architecture Model for ILC Reference Design

ILC Tech Note

providing a

set of well
-
defined non
-
graphical interfaces. Device abstractions such as
magnets and BPMs that incorporate engineering, physics, and control models are
represented in this tier. This makes it possible to relate high
-
level machine parameters
with low
-
leve
l equipment settings in a standard and centralized way. This centralization
of control provides many benefits in terms of coordination, security,
automation,
optimization, and conflict avoidance. For example, a parameter save/restore service can
prevent tw
o client applications from simultaneously attempting to restore a common
subset of operational parameters.


Some candidate services are:




Device Abstraction


access to devices that combine front
-
end channels and
physics parameters.



Model Interaction


acc
ess to engineering, physics, and control models of the
system.



Feedback


configuration of slow and fast feedback loops.



Archiving


high
-
performance background archiving of data along with access to
past data.



Save/Restore


provides central control over
operational parameters.



Logging


central logging of status from software components.



Alarms


coordinates alarms from front
-
end tier and other sources.



Deployment


provides explicit management of software/firmware deployment.



Automation


provides state
machines for repeatable, complex procedures.


The use of a relational database has become commonplace in control systems.
Traditionally considered to be a single point of failure, today’s high
-
performance, highly
managed, redundant databases can be an inte
gral part of a control system. This applies
during machine operations as well. Just as with a large
-
scale consumer web site, the
underlying database can be separated into one or more tightly controlled OLTP (OnLine
Transaction Processing) and OLAP (OnLine
Analytical Processing) databases. Each is
tuned to a particular purpose. There is ample evidence of the use of relational databases
in high availability scenarios.

2.1.3

Front
-
end Tier

The front
-
end tier is where the technical equipment controls reside, often r
eferred to as
Input Output Controllers (IOCs).
This tier provides access to the field I/O, including field
buses of various sorts, as well as
discrete analog and binary I/O

[2]
. This tier is
configured and managed by the services tier, but can run autonomo
usly. The primary
abstraction in this tier is a channel, or process variable, roughly equivalent to a single I/O
point. Some control systems also provide the ability to group channels at this level into
abstract devices that generally represent a piece of
equipment such as a whole motor or
beam position monitor.



Control System Architecture Model for ILC Reference Design

ILC Tech Note

2.2

Network Functions

The functionality and performance requirements within and between the three tiers varies
considerably. We show in Figure
1

a set of network functions.
Each network function
will h
ave a physical realization, and will carry one or more protocols.

Below we begin to
characterize the network functions and types of protocols we expect the physical network
to carry.

We mention specific protocols in some cases, but these examples are simpl
y
meant to clarify, and don’t represent an actual choice yet.

2.2.1

Deployment and Management Network

The deployment and management network function provides out
-
of
-
band monitoring and
management of the many controls resources, both hardware and software compone
nts.
We classify the protocols here as DMP (Deployment and Management Protocol).
Typically found here are management protocols like SNMP (Simple Network
Management Protocol) and RMCP (Remote Management and Control Protocol). FTP,
TFTP, Telnet, and SSH are
common for switch configuration and various console access.
Standards bodies such as the DMTF (Distributed Management Task Force) are involved
in developing the next generation of management standards called CIM (Common
Information Model).

2.2.2

General Purpose
Network

The general purpose network
function
will carry many protocols, as it is expected to be
less regulated than the other network functions.

Standard commodity networking
protocols are to be expected here, such as DHCP, ICMP, SSH, etc.

2.2.3

Video Network

Th
e video network function will carry some form of streaming protocol.

2.2.4

Control
s

Network

The control
s

network will carry many protocols. Most will not be determined until such
time as the control system framework is selected. In general, however, we can expec
t
so
me combination of the following, each categorized with a generic acronym.


DOP (Distributed Object Protocol)


The purpose of DOP is to provide location independent access to services. The access is
typically synchronous, although asynchronous notificat
ion is sometimes provided. There
are language
-
specific DOP such as Java RMI/JINI, and language
-
independent DOP such
as CORBA/IIOP and Web Services. Given the timescale of a project such as ILC, we
believe a language neutral DOP is a better choice.


MQP (
Message Queuing Protocol)

The MQP provides the ability to asynchronously pass information between decoupled
applications and services. Logging and alarms are a prime example. These messages must
typically have guaranteed delivery and persistence. There are

many forms of MQP such
as Java JMS and IBM MQSeries.


Control System Architecture Model for ILC Reference Design

ILC Tech Note


SRTP (Soft Real
-
Time Protocol)

A SRTP provides high performance access to fine
-
grained data points. The protocol
typically dispenses with complex serialization (marshalling) and deserializatio
n
(unmars
halling) found in DOP. EPICS Channel Access and the Fermilab ACNET
protocol are examples in this category.


DBP (DataBase Protocol)

The DBP is a well defined protocol category. There are language dependent choices here
such as Java JDBC, or language indepe
ndent choices such as ODBC. Given the breadth
of available relational database tools, it is common to support multiple protocols.


FBP (Field
-
Bus Protocol)

The FBP is usually dependent upon the vendor of a particular piece of equipment. The
goal is to lim
it the breadth to a reasonable set. Here we find protocols such as Modbus,
CAN, PROFIBUS, and DeviceNet.


2.2.5

Dedicated Networks

The remaining network functions; interlock, machine protection, fast feedback, timing,
and synchronization, are all typically HRTP

(Hard Real
-
Time Protocol). The behavior of
the protocol is deterministic to guarantee timely communication for critical functions.


In particular, the timing network will provide an event system that will be utilized by
both hard and soft real
-
time system
s, as well as the more conventional computing hosts.
Automation and flexible pulse
-
to
-
pulse feedback algorithms are implemented by a
coordinated set of software services that work together through global coordination and
distributed execution. The distribu
ted execution is synchronized with the machine pulse
rate v
ia the timing event system that

can produce so
ftware interrupts
and pulse IDs
where
needed.


3

Physical Model

The ILC control system must reliably interact with more than 100,000 technical system
dev
ices that could collectively amount to several million scalar and vector process
variables distributed across many kilometers of beam line and surface facilities.
Information must be processed and distributed on a variety of timescales from
microseconds to

several seconds. The overall philosophy is to develop an architecture
that can meet the requirements, while leveraging the cost savings and rapid evolutionary
advancements of commercial off
-
the
-
shelf (COTS) components.


We focus here on the
control system

network and hosts, and some of the secondary
supporting networks such as video, out
-
of
-
band monitoring, and general purpose field
networking. There will undoubtedly be a number of specialized networks for applications
such as fast feedback and timing even
ts, however those are not covered here.


Control System Architecture Model for ILC Reference Design

ILC Tech Note

3.1

Main Controls Network

The
general control system network carries the data for process control, data acquisition,
and slow feedback (slow meaning the pulse repetition rate of the accelerator).
Commercial off
-
the
-
shelf

Ethernet using layer
-
2 and layer
-
3 switching at 1 Gigabit and
10 Gigabit speeds

is the core technology envisioned in this model

[3]
.
Fully redundant
connections are used throughout; with a combination of dual star and loop/mesh
interconnect a
t various
lay
ers. Figure 2

shows the topology as applied to the main linac,
however we expect the same basic structure for all other area systems. Note that
redundant connections are not shown to make the figure readable.



Control System Architecture Model for ILC Reference Design

ILC Tech Note


FIGUR
E 2
:
Control System Physical Model


Th
e upper layer of the figure comprises the main control room and data center controls
computing services. There will be one primary controls room and data center, and a
number of much smaller facilities distributed over the physical machine, most likely one

per tunnel access shaft. The systems at each location will be dual
-
star networked to a
switch with a high
-
capacity uplink leading to a set of underground switches.



Control System Architecture Model for ILC Reference Design

ILC Tech Note

Th
e underground switches are the main controls network and will be interconnected via a
c
ombination of loop and mesh topology yet to be determined.
At each such underground
backbone switch, there will be a set of computing nodes for distributed data acquisition
and feedback, and an aggregator switch to collect lower bandwidth connections from
the
many controls front
-
end systems.


The many front
-
end hosts

will be co
-
located with the technical equipment in the service
tunnel. Here we will have a combination of controls shelves serving several purposes.
The controls shelves will operate the instr
umentation digitizers,

local timing receivers,
and any other large form
-
factor (ex. ATCA) boards. The controls shelves also

serve
control system protocol
-
specific functions such as name servers, forward gateways, and
reverse gateway
s. These are necessary t
o scale the control system through broadcast and
connection management
, while retaining the ability to have any controls shelf obtain data
from any other controls shelf.


The front
-
end controls shelves will also manage communication over many Ethernet fie
ld
networks to a variety of embedded systems further distributed within or near technical
systems such as modulators, low
-
level RF, commercial power supplies, valve controllers,
etc.
The goal is to have a standard Ethernet port be the field connectivity st
andard.
The
many relatively low
-
bandwi
d
th connections from such embedded controllers will be
further aggregated through small 1U switches distributed among technical system (non
-
controls) racks. This will allow separate assembly and installation of racks b
y minimizing
inter
-
rack controls cabling
.


3.2

Other Networks

A general Ethernet network will be deployed throughout the underground tunnels and
surface facilities to accommodate more ad
-
hoc connections, such as laptops and portable
test equipment, and other l
ess closely regulated equipment that should be isolated from
direct access to the controls Ethernet.


A video network will be necessary to distribute the large number of expected video
streams. It is possible that such a function can be managed as part of
the general Ethernet,
but we have chosen to cost a separate physical network for video.


Lastly, the large number of controls resources must be managed
remotely
in a reliable
fashion. The numerous networking switches must
be monitored and configured remote
ly.
The console ports of the computing blades, embedded processors, and FPGA’s must be
remotely accessible for configuration, rapid diagnosis, and repair. An out
-
of
-
band
monitoring and management network shall be deployed for these purposes. This out
-
of
-
ba
nd monitoring network serves a similar function to terminal servers. The goal is to put
the secondary ports of all equipment through this network.



Control System Architecture Model for ILC Reference Design

ILC Tech Note

We also assume there will be many dedicated networks for timing and event distribution,
fast feedback, inter
locks, machine, and personnel protection. Those are not covered in
this technical note.


4

Realizing
the
Functional
Model

The realization of the network functions on the physical model described in the previous
section is fairly straightforward. In all case
s to date, we have assumed a separate physical
network for each function, with each network itself being redundant.


The realization of

the 3 tiers of

software functions on the physical model is more
c
omplex
. This depends in part upon the selection of a pa
rticular control system and its
associated protocols. Nonetheless, we can make some assumptions just based
on basic
data flow constraints and the use of service oriented architecture

[4]
.


We assume that control consoles with graphical and scripting applic
ations will reside on
the hosts in the control rooms
, and on portable diagnosis stations. These client tier
applications will naturally reside where their human users are. The various services that
these client applications depend upon
will be physically l
ocated o
n various hosts
throughout the physical network.

Service location is largely transparent to client
applications, and so can be located to optimize bandwidth and latency.


Feedback services will be distributed in several locations. Central feedback
configuration
and status will be deployed in the controls computing services (control room and/or data
center)
. Feedback algorithms will be deployed on hosts distributed along the underground
backbone switches. The loop/mesh connectivity of these switches
will allow for
necessary
data flow [5
]. More localized feedback algorithms will be distributed to the
front
-
end controls shelves.


Archiving ser
vices will also be distributed o
n at least two levels. Individual archive
engines with reduction algorithms and

local storage will be distributed on hosts
distributed along the underground backbone switches. Higher level archiving engines
with access to central database servers

and disk farm will be located o
n t
he controls
computing services hosts.


References

[1]

Gotz A., Schmidt D., Clausen M., “Middleware in Accelerator and Telescope Control
Systems” 9
th
International
Conference on Accelerator and Large Experimental Physics
Control Systems
,
Gyeongju, Korea, Oct. 13
-
17, 2003
,

ICALEPCS
Conference
Proceedings
, pp 3
22
-
326

(
2003
).

[2] R.W. Downing, R.S. Larsen, 'High Availability Instrumentation Packaging Standards
for the ILC and Detectors' SLAC
-
PUB
-
12208
http://www.slac.stanford.edu/cgi
-
wra
p/pubpage?slac
-
pub
-
12208


Control System Architecture Model for ILC Reference Design

ILC Tech Note

[3] Clark, K., Hamilton, K., “Cisco LAN Switching”, Cisco Press 2001.

[4] Erl, Thomas, “Service
-
Oriented Architecture”, Prentice Hall PTR 2005.

[5] Votava, M., Bowden, M., Gottschalk, E., “Commodity Computing Architecture”, ILC
-
do
c
-
332 http://docdb.fnal.gov/ILC
-
public/DocDB/ShowDocument?docid=332