DOC - ILC Document Server

chinchillatidyNetworking and Communications

Oct 26, 2013 (4 years and 8 months ago)


Control System Architecture Model for ILC Reference Design

ILC Tech Note

March 19, 2007

Authors: Cla
ude Saunders, John Carwardine, Ned Arnold, Margaret Votava, Vince
Pavlicek, Ray Larsen, Bob Downing, Steve Wolbers, Bakul Banerjee, Schinichiro
Michizono, Stefan Simrock, Jim Patrick, Sharon Lackey, Paul Kasley, Paul Joireman,
ay Rehlich, Kazuro Furokawa, Erik Go
ttschalk, Frank Lenkszus, Ron



ILC control system architecture is evolving as requirements are gathered and the
research and development effort gains steam. We present the
general purpose contro
architecture as of the time of writing the ILC Reference Design Report.

We do not
cover specialized subsystems such as timing, fast feedback, and machine protection.

rationale for establishing

at this stage is threefold: we requir
e a model from
which to make a cost estimate; we require a model to validat
e that we are meeting the
control system requirem
ents; and
we must communicate our vision to the technical
system groups early and often.


architecture described here

is not full


a number of distinct design choices are made

Future requirements gathering,
research, and development will complete the picture.



The architectural model

of the ILC control system is presented here from both functiona
and physical perspectives. This mod
el has served to produce a cost

estimate, as well as to
document our coverage of the control system requirements to date.
We also connect the
functional and physical models by showing where various functions are (or cou
ld be)
physically realized.

While specific technologies such as Ethernet are
, these
are ultimately intended to be representative technologies, and may well change by the
time the actual project construction phase begins.


Functional Model

The fu
nctional perspective combines software functions and network functions into a 3
tier model.
The 3
er approach (or design pattern) is common

in modern software
architecture. This approach provides many benefits including separation of concerns, re
use, lo
ad management, and change management.
Older and smaller scale control systems
consist of only 2 tiers; the client or application tier, and the technical systems tier. The 3
tier model introduces an intervening services/mid
dleware tier where many essenti
shared functions reside [1].

Figure 1


the internal structure of the 3 tiers, as well as the networking
functions required within and between the tiers. The remainder of this section is de
to elaborating on this figure.

Control System Architecture Model for ILC Reference Design

ILC Tech Note

trol System Functional Model


Software Functions


Client Tier

The client tier consists of applications with which people directly interact (graphical user
interfaces or GUI). Applications will range from engineering
oriented control consoles to
level p
hysics control appli
cations to system
management applications. Engineering
oriented consoles are focused on the direct operation of the underlying accelerator
equipment in the front
end tier. High
level physics applications will utilize a blend of

from the service tier that in turn combine data from the front
end tier and
supporting data from the relational database.


Services Tier


the perspective of a

user of the client tier, the serv
ices tier is largely invisible.
goal of the services ti
er is to manage the execution of logic in the problem domain, and
leave the problems of user interaction and graphical presentation of data and status to the
client tier. The services tier provides services that coordinate many activities while

Control System Architecture Model for ILC Reference Design

ILC Tech Note

providing a

set of well
defined non
graphical interfaces. Device abstractions such as
magnets and BPMs that incorporate engineering, physics, and control models are
represented in this tier. This makes it possible to relate high
level machine parameters
with low
l equipment settings in a standard and centralized way. This centralization
of control provides many benefits in terms of coordination, security,
optimization, and conflict avoidance. For example, a parameter save/restore service can
prevent tw
o client applications from simultaneously attempting to restore a common
subset of operational parameters.

Some candidate services are:

Device Abstraction

access to devices that combine front
end channels and
physics parameters.

Model Interaction

ess to engineering, physics, and control models of the


configuration of slow and fast feedback loops.


performance background archiving of data along with access to
past data.


provides central control over
operational parameters.


central logging of status from software components.


coordinates alarms from front
end tier and other sources.


provides explicit management of software/firmware deployment.


provides state
machines for repeatable, complex procedures.

The use of a relational database has become commonplace in control systems.
Traditionally considered to be a single point of failure, today’s high
performance, highly
managed, redundant databases can be an inte
gral part of a control system. This applies
during machine operations as well. Just as with a large
scale consumer web site, the
underlying database can be separated into one or more tightly controlled OLTP (OnLine
Transaction Processing) and OLAP (OnLine
Analytical Processing) databases. Each is
tuned to a particular purpose. There is ample evidence of the use of relational databases
in high availability scenarios.


end Tier

The front
end tier is where the technical equipment controls reside, often r
eferred to as
Input Output Controllers (IOCs).
This tier provides access to the field I/O, including field
buses of various sorts, as well as
discrete analog and binary I/O

. This tier is
configured and managed by the services tier, but can run autonomo
usly. The primary
abstraction in this tier is a channel, or process variable, roughly equivalent to a single I/O
point. Some control systems also provide the ability to group channels at this level into
abstract devices that generally represent a piece of
equipment such as a whole motor or
beam position monitor.

Control System Architecture Model for ILC Reference Design

ILC Tech Note


Network Functions

The functionality and performance requirements within and between the three tiers varies
considerably. We show in Figure

a set of network functions.
Each network function
will h
ave a physical realization, and will carry one or more protocols.

Below we begin to
characterize the network functions and types of protocols we expect the physical network
to carry.

We mention specific protocols in some cases, but these examples are simpl
meant to clarify, and don’t represent an actual choice yet.


Deployment and Management Network

The deployment and management network function provides out
band monitoring and
management of the many controls resources, both hardware and software compone
We classify the protocols here as DMP (Deployment and Management Protocol).
Typically found here are management protocols like SNMP (Simple Network
Management Protocol) and RMCP (Remote Management and Control Protocol). FTP,
TFTP, Telnet, and SSH are
common for switch configuration and various console access.
Standards bodies such as the DMTF (Distributed Management Task Force) are involved
in developing the next generation of management standards called CIM (Common
Information Model).


General Purpose

The general purpose network
will carry many protocols, as it is expected to be
less regulated than the other network functions.

Standard commodity networking
protocols are to be expected here, such as DHCP, ICMP, SSH, etc.


Video Network

e video network function will carry some form of streaming protocol.




The control

network will carry many protocols. Most will not be determined until such
time as the control system framework is selected. In general, however, we can expec
me combination of the following, each categorized with a generic acronym.

DOP (Distributed Object Protocol)

The purpose of DOP is to provide location independent access to services. The access is
typically synchronous, although asynchronous notificat
ion is sometimes provided. There
are language
specific DOP such as Java RMI/JINI, and language
independent DOP such
as CORBA/IIOP and Web Services. Given the timescale of a project such as ILC, we
believe a language neutral DOP is a better choice.

Message Queuing Protocol)

The MQP provides the ability to asynchronously pass information between decoupled
applications and services. Logging and alarms are a prime example. These messages must
typically have guaranteed delivery and persistence. There are

many forms of MQP such
as Java JMS and IBM MQSeries.

Control System Architecture Model for ILC Reference Design

ILC Tech Note

SRTP (Soft Real
Time Protocol)

A SRTP provides high performance access to fine
grained data points. The protocol
typically dispenses with complex serialization (marshalling) and deserializatio
halling) found in DOP. EPICS Channel Access and the Fermilab ACNET
protocol are examples in this category.

DBP (DataBase Protocol)

The DBP is a well defined protocol category. There are language dependent choices here
such as Java JDBC, or language indepe
ndent choices such as ODBC. Given the breadth
of available relational database tools, it is common to support multiple protocols.

FBP (Field
Bus Protocol)

The FBP is usually dependent upon the vendor of a particular piece of equipment. The
goal is to lim
it the breadth to a reasonable set. Here we find protocols such as Modbus,
CAN, PROFIBUS, and DeviceNet.


Dedicated Networks

The remaining network functions; interlock, machine protection, fast feedback, timing,
and synchronization, are all typically HRTP

(Hard Real
Time Protocol). The behavior of
the protocol is deterministic to guarantee timely communication for critical functions.

In particular, the timing network will provide an event system that will be utilized by
both hard and soft real
time system
s, as well as the more conventional computing hosts.
Automation and flexible pulse
pulse feedback algorithms are implemented by a
coordinated set of software services that work together through global coordination and
distributed execution. The distribu
ted execution is synchronized with the machine pulse
rate v
ia the timing event system that

can produce so
ftware interrupts
and pulse IDs


Physical Model

The ILC control system must reliably interact with more than 100,000 technical system
ices that could collectively amount to several million scalar and vector process
variables distributed across many kilometers of beam line and surface facilities.
Information must be processed and distributed on a variety of timescales from
microseconds to

several seconds. The overall philosophy is to develop an architecture
that can meet the requirements, while leveraging the cost savings and rapid evolutionary
advancements of commercial off
shelf (COTS) components.

We focus here on the
control system

network and hosts, and some of the secondary
supporting networks such as video, out
band monitoring, and general purpose field
networking. There will undoubtedly be a number of specialized networks for applications
such as fast feedback and timing even
ts, however those are not covered here.

Control System Architecture Model for ILC Reference Design

ILC Tech Note


Main Controls Network

general control system network carries the data for process control, data acquisition,
and slow feedback (slow meaning the pulse repetition rate of the accelerator).
Commercial off

Ethernet using layer
2 and layer
3 switching at 1 Gigabit and
10 Gigabit speeds

is the core technology envisioned in this model

Fully redundant
connections are used throughout; with a combination of dual star and loop/mesh
interconnect a
t various
ers. Figure 2

shows the topology as applied to the main linac,
however we expect the same basic structure for all other area systems. Note that
redundant connections are not shown to make the figure readable.

Control System Architecture Model for ILC Reference Design

ILC Tech Note

E 2
Control System Physical Model

e upper layer of the figure comprises the main control room and data center controls
computing services. There will be one primary controls room and data center, and a
number of much smaller facilities distributed over the physical machine, most likely one

per tunnel access shaft. The systems at each location will be dual
star networked to a
switch with a high
capacity uplink leading to a set of underground switches.

Control System Architecture Model for ILC Reference Design

ILC Tech Note

e underground switches are the main controls network and will be interconnected via a
ombination of loop and mesh topology yet to be determined.
At each such underground
backbone switch, there will be a set of computing nodes for distributed data acquisition
and feedback, and an aggregator switch to collect lower bandwidth connections from
many controls front
end systems.

The many front
end hosts

will be co
located with the technical equipment in the service
tunnel. Here we will have a combination of controls shelves serving several purposes.
The controls shelves will operate the instr
umentation digitizers,

local timing receivers,
and any other large form
factor (ex. ATCA) boards. The controls shelves also

control system protocol
specific functions such as name servers, forward gateways, and
reverse gateway
s. These are necessary t
o scale the control system through broadcast and
connection management
, while retaining the ability to have any controls shelf obtain data
from any other controls shelf.

The front
end controls shelves will also manage communication over many Ethernet fie
networks to a variety of embedded systems further distributed within or near technical
systems such as modulators, low
level RF, commercial power supplies, valve controllers,
The goal is to have a standard Ethernet port be the field connectivity st
many relatively low
th connections from such embedded controllers will be
further aggregated through small 1U switches distributed among technical system (non
controls) racks. This will allow separate assembly and installation of racks b
y minimizing
rack controls cabling


Other Networks

A general Ethernet network will be deployed throughout the underground tunnels and
surface facilities to accommodate more ad
hoc connections, such as laptops and portable
test equipment, and other l
ess closely regulated equipment that should be isolated from
direct access to the controls Ethernet.

A video network will be necessary to distribute the large number of expected video
streams. It is possible that such a function can be managed as part of
the general Ethernet,
but we have chosen to cost a separate physical network for video.

Lastly, the large number of controls resources must be managed
in a reliable
fashion. The numerous networking switches must
be monitored and configured remote
The console ports of the computing blades, embedded processors, and FPGA’s must be
remotely accessible for configuration, rapid diagnosis, and repair. An out
monitoring and management network shall be deployed for these purposes. This out
nd monitoring network serves a similar function to terminal servers. The goal is to put
the secondary ports of all equipment through this network.

Control System Architecture Model for ILC Reference Design

ILC Tech Note

We also assume there will be many dedicated networks for timing and event distribution,
fast feedback, inter
locks, machine, and personnel protection. Those are not covered in
this technical note.



The realization of the network functions on the physical model described in the previous
section is fairly straightforward. In all case
s to date, we have assumed a separate physical
network for each function, with each network itself being redundant.

The realization of

the 3 tiers of

software functions on the physical model is more
. This depends in part upon the selection of a pa
rticular control system and its
associated protocols. Nonetheless, we can make some assumptions just based
on basic
data flow constraints and the use of service oriented architecture


We assume that control consoles with graphical and scripting applic
ations will reside on
the hosts in the control rooms
, and on portable diagnosis stations. These client tier
applications will naturally reside where their human users are. The various services that
these client applications depend upon
will be physically l
ocated o
n various hosts
throughout the physical network.

Service location is largely transparent to client
applications, and so can be located to optimize bandwidth and latency.

Feedback services will be distributed in several locations. Central feedback
and status will be deployed in the controls computing services (control room and/or data
. Feedback algorithms will be deployed on hosts distributed along the underground
backbone switches. The loop/mesh connectivity of these switches
will allow for
data flow [5
]. More localized feedback algorithms will be distributed to the
end controls shelves.

Archiving ser
vices will also be distributed o
n at least two levels. Individual archive
engines with reduction algorithms and

local storage will be distributed on hosts
distributed along the underground backbone switches. Higher level archiving engines
with access to central database servers

and disk farm will be located o
n t
he controls
computing services hosts.



Gotz A., Schmidt D., Clausen M., “Middleware in Accelerator and Telescope Control
Systems” 9
Conference on Accelerator and Large Experimental Physics
Control Systems
Gyeongju, Korea, Oct. 13
17, 2003

, pp 3


[2] R.W. Downing, R.S. Larsen, 'High Availability Instrumentation Packaging Standards
for the ILC and Detectors' SLAC

Control System Architecture Model for ILC Reference Design

ILC Tech Note

[3] Clark, K., Hamilton, K., “Cisco LAN Switching”, Cisco Press 2001.

[4] Erl, Thomas, “Service
Oriented Architecture”, Prentice Hall PTR 2005.

[5] Votava, M., Bowden, M., Gottschalk, E., “Commodity Computing Architecture”, ILC