Robust Adaptive Topology Control (RATC)

awfulcorrieAI and Robotics

Oct 29, 2013 (4 years and 12 days ago)

438 views

RATC Task 1 Report

Applied Communication Sciences








Robust Adaptive Topology Control (RATC)


Identification of Data and Control Flows in the RATC Solution

Task 1 Report


Project Deliverable



Prepared by
Applied Communication
Science
s

for
the

Department of Energy (DOE)

Advanced Research Projects

Agency



Energy (ARPA
-
E
)

a
nd

Texas A&M University



Octo
ber
3
1
, 20
1
2





Applied Communication Science
s Contact
s
:

S. Ayyorgun and

J
. S
ucec

Texas A&M Contact: M. Kezonuvic





RATC Task 1 Report

Applied Communication Sciences



1

Introduction

................................
................................
................................
...........................

1

2

Overview of RATC Data and Flows

................................
................................
....................

3

3

RATC Communication Architectures

................................
................................
................

6

3.1

Centralized

................................
................................
................................
......................

6

3.2

Decentralized and Distributed
................................
................................
.........................

7

3.3

Legacy System Data Sources

................................
................................
..........................

9

3.4

Future System Data Sources

................................
................................
.........................

10

4

RATC Network Flow Event Sequences

................................
................................
............

12

4.1

Centralized Optimization Communication Event Flow Example

................................

12

4.2

Decentralized Substation Communication Event Flow Example

................................
.

14

5

Candidate Substation
Physical Topologies and Performance Considerations

.............

17

6

Typical WAN Design for Power Systems and Related QoS Considerations

.................

21

7

Logical Topologies and Communication Architectures

................................
..................

25

7.1

Intra
-
Substation Communications and Availability Considerations
.............................

25

7.2

Inter
-
Substation Communications and QoS
-
Related Performance Considerations
......

31

7.3

Communications Between Substation and Control Center and WAN Capacity
Constraints

................................
................................
................................
................................

32

7.4

Intra
-
Control Center Communications and Potential EMS Network Contention

........

34

8

Cyber
-
Security Considerations
................................
................................
..........................

36

9

References

................................
................................
................................
............................

38

10

Appendix A: Acronyms

................................
................................
................................
......

39



RATC Task 1 Report

Applied Communication Sciences



1

1

Introduction

This
Task 1 report for the

Robust Adaptive

Topology Control (RATC) project

presents
the
network data and control flows pertinent to the RATC algorithm communications in the context
of
the
projected
power systems operations (PSO) network
infrastructure
.

The RATC algorithm
will consi
st of
potentially
both centralized and distributed components with RATC software
processes computing global system management signals at the utility control centers
(CCs)
and

performing

decentralized local
system/data management

f
unctions at
utility
substa
tion and power
generation sites.

Broadly speaking t
he goals of this

report are as follows.

1.

Identify the RATC
communication
network data and control flows including flow source,
destination, data type/format and applicable

Application Layer

standards/protocols.

2.

Identify the RATC
communication
network flow event sequences including the
applicable operations and relevant actors.

3.

Identify
typical
PSO
communication
network architectures, including physical and
logical
network topologies,
as they

relate to RATC communication needs
(
i.e
.

intra
-
substation, inter
-
substation, substation to/from control centers, and among control
centers).

4.

Identify the
communication
network
constraint
s

and challenges that may limit the
availability of data required by
RATC algorithm modules and potentially present issues
in terms network flow performance

or

cyber
-
security risks.

(This is in fact part of Task 2
(Q2
-
Q4)
per
the ACS
SOW, but
this

work
has been started in Quarter 2

and reported
additionally as part of Task
1
as these are important for communication

architecture
consideration
s
.
)

This report
identifie
s the communication network data flows between the RATC system and
external systems (e.g. CC databases) as well as between the RATC system and Operators.
Furthermore,
the required communication flows between RATC subsystem components are also
presented.
Figure
1

shows the high
-
level breakout of RATC subsystems/componen
ts. The RATC
Analytics
subsystem consist
s

of the RATC algorithm components and exchange inputs/outputs
with one another largely via the RATC Analytics file system. The RATC Data Management
subsystem consists of components developed by Grid Protection Allia
nce (GPA) and includes a
suite of interface adapters to communicate with external systems, Operators as well as RATC
Analytics components.


Figure
1

High
-
level view of RATC components/subsystems

RATC Task 1 Report

Applied Communication Sciences



2

Last, while a centralized RATC depl
oyment is understood to be the focus of

the

near
-
term RATC
software development effort, this report additionally considers the communication network flows
for an optional future distributed, decentralized RATC implementation where RATC
components are deplo
yed at multiple CCs and at substation sites. The distributed RATC
processes will leverage the utility’s inter
-
site network
s

to communicate with one another and the
intra
-
site networks to obtain data from substation sensors and from existing power system data
repositories situated at the utility’s CC.



RATC Task 1 Report

Applied Communication Sciences



3

2

Overview of RATC Data and Flows

ACS has analyzed the
RATC communication
network data flows

and has identified the following
source

destination flow descriptions listed in
Table
1
. The first block of rows, shaded light
blue, corresponds to

the data communication flows for external
CC

data sources to the RATC
data management components. The second block, shaded orange, corresponds to data
communication flows for PSO data from the RATC databases to the RATC Analytics
(algorithms). The third
block, shaded light green, corresponds to data communication flows
between the individual RATC Analytics algorithms. The fourth block, shaded in yellow,
corresponds to communication flows for receiving and responding to Operator requests. The last
block, s
haded in lavender, corresponds to future data communication flows for substation RATC
components.

Table
1

Summary
of
RATC Network Data Flow Descriptions

Source

Destination

Data Type

Data Format

Protocol(s)

RC CC SCADA

CC RATC GMC
1

DFR event records

COMTRADE

ICCP

RC CC SCADA

CC RATC GMC

DPR event records

COMTRADE

ICCP

RC CC SCADA

CC RATC GMC

CBM event records

COMTRADE

ICCP

RC CC SCADA

CC RATC GMC

Load flow
2

TBD

ICCP

RC CC SCADA

CC RATC GMC

Generator availability

Static
text
file

ICCP

RC CC SCADA

CC RATC GMC

Current breaker state

(TBD)

ICCP

RC CC SCADA

CC RATC GMC

Historical CB data

(TBD)

ICCP

RC CC ENMD

RATC RDBMS

Static model data

(TBD)

DB
-
to
-
DB

RC CC ENMD

RATC RDBMS

Dynamic model data

(TBD)

DB
-
to
-
DB

RC CC Planning

RATC
RDBMS

Line availability

(TBD)

DB
-
to
-
DB

RATC RDBMS

Optimization

Static model data

Text file (
CPLEX
-
format
)

SSH/SFTP, RATC
file system

RATC RDBMS

Optimization

Line availability

Text file (
CPLEX
-
format
)

SSH/SFTP, RATC
file system

CC RATC GMC

Optimization

Generator availability

Text file (
CPLEX
-
format
)

SSH/SFTP, RATC
file system

CC RATC GMC

Optimization

Topology data

Text file (
CPLEX
-
format
)

SSH/SFTP, RATC
file system

CC RATC GMC

Optimization

Load flow

Text file (
CPLEX
-
format
)

SSH/SFTP, RATC
file system

RATC RDBMS

Cascade Detection

Static model data

Text file (TBD
format, e.g. CDF)

SSH/SFTP
/HTTPS
?,
RATC file system

CC RATC GMC

Cascade Detection

Load flow

Text file (TBD
format, e.g. CDF)

SSH/SFTP
/HTTPS?
,
RATC file system

CC RATC GMC

Cascade Detection

DFR

event records

Text file (TBD
format, e.g. CDF)

SSH/SFTP
/HTTPS
?,
RATC file system

CC RATC GMC

Cascade Detection

DPR event records

Text file (TBD
format, e.g. CDF)

SSH/SFTP
/
HTTPS?,

RATC file system

RATC RDBMS

CBOE

Static model data

Text file (TBD
format,
e.g. CDF)

SSH/SFTP
/HTTPS
?,
RATC file system

CC RATC GMC

CBOE

CBM event records

Text file (TBD
format, e.g. CDF)

SSH/SFTP
/HTTPS
?,
RATC file system

CC RATC GMC

CBOE

Current breaker state

Text file (TBD
format, e.g. CDF)

SSH/SFTP
/HTTPS
?,
RATC file system




1

The Grid Measurement Component (GMC) is the combined Real
-
Time Database and Real
-
Time Data File Store

2

It is understood based on discussions with GPA that “Load Flow” is essentially measurements of V, A, MW and
MVAR.

RATC Task 1 Report

Applied Communication Sciences



4

Source

Destination

Data Type

Data Format

Protocol(s)

CC RATC GMC

CBOE

Historical CB data

Text file (TBD
format, e.g. CDF)

SSH/SFTP
/HTTPS
?,
RATC file system

RATC RDBMS

Stability Check

Static model data

Text file (TBD
format, e.g. CDF)

SSH/SFTP?, RATC
file system

CC RATC GMC

Stability Check

Topology

data

Text file (TBD
format, e.g. CDF)

SSH/SFTP?, RATC
file system

CC RATC GMC

Stability Check

Load flow

Text file (TBD
format, e.g. CDF)

SSH/SFTP?, RATC
file system

RATC RDBMS

Overvoltage Check

Static model data

Text file (TBD
format, e.g. CDF)

SSH/SFTP?,
RATC
file system

RATC RDBMS

Overvoltage Check

Dynamic model data

Text file (TBD
format, e.g. CDF)

SSH/SFTP?, RATC
file system

CC RATC GMC

Overvoltage Check

Topology data

Text file (TBD
format, e.g. CDF)

SSH/SFTP?, RATC
file system

CC RATC GMC

Overvoltage Check

Load flow

Text file (TBD
format, e.g. CDF)

SSH/SFTP?, RATC
file system

RATC RDBMS

Relay Settings
Adjustment

Static model data

Text file (TBD
format, e.g. CDF)

SSH/SFTP
/HTTPS
?,
RATC file system

Cascade Detection

Optimization

Line
availability update

Text file (
CPLEX
-
format
)

RATC file system

CBOE

Optimization

Line availability update

Text file (
CPLEX
-
format
)

RATC file system

Optimization

RATC Stability
Check

Lines to be switched

Text file (TBD
format)

RATC file system

Optimization

Overvoltage Check

Lines to be switched

Text file (TBD
format)

RATC file system

Optimization

Relay Settings
Adjustment

Lines to be switched

Text file (TBD
format)

RATC file system

Stability Check

Optimization

Ok/Not ok

Text file (TBD
format)

RATC file system

Overvoltage Check

Optimization

Switching overvoltage
levels; Ok/Not ok

Text file (TBD
format)

RATC file system

Optimization

Operator

Lines to be switched

Text file (
CPLEX
-
format
)

SSH/SFTP, RATC
file system

RC Operator

RATC DB
Management

Optimization request

Simple message
(TBD)

IP socket
-
based C/S
request
? Pub/Sub?

RATC DB
Management

Optimization

Optimization request
notification

Simple message
(TBD)

IP socket
-
based C/S
request

Optimization

RATC DB
Management

Optimization complete

Simp
le message
(TBD)

IP socket
-
based C/S
noti
ce

RATC DB
Management

RC Operator

Optimization notification

Simple message
(TBD)

IP socket
-
based C/S
noti
ce
? Pub/Sub?

CBOE

Operator

Maintenance report

Text file (TBD
format)

SSH/SFTP
/HTTPS
?,
RATC file system

Cascade Detection

Operator

Cascade detection, fault
report

Text file (TBD
format)

SSH/SFTP
/HTTPS
?,
RATC file system

Relay Settings
Adjustment

Operator

Zone 2 and 3 settings
update

Text file (TBD
format)

SSH/SFTP
/HTTPS
?,
RATC file system

Substation CBM

SA

RATC GMC

CBM report

Text file (
XML

format
?
)

DNP3 (IEC 61850
-
8/MMS)

Substation DFR

SA RATC GMC

Fault report

Text file (format =
COMTRADE
)

DNP3 (IEC 61850
-
8/MMS)

RATC Task 1 Report

Applied Communication Sciences



5

Source

Destination

Data Type

Data Format

Protocol(s)

Substation DPR

SA RATC GMC

Fault report

Text file (format =
COMTRADE
)

DNP3 (IEC 61850
-
8/MMS)

SA RATC GMC

CC RATC GMC

Substation compressed or
filtered RATC data

Text file
(TBD
RATC
-
defined)

GEP

SA RATC GMC
(A)

SA RATC GMC
(B)

Compressed or filtered
RATC substation data

Text file
(TBD
RATC
-
defined)

GEP

Of particular interest from a communication network data flow analysis standpoint is the right
-
most column of
Table
1

which identifies the associated Application Layer

(Layer 5)
protocol/interface for the data flow. The underlying Transport (Layer 4) and Network (Layer)
Layer protocols are understood to be IP
-
based. Furthermore, the underlying Data Link (Layer 2)
and Physical (Layer 1) Layer protocols/standards are unde
rstood to be Ethernet
-
based.



RATC Task 1 Report

Applied Communication Sciences



6

3

RATC Communication Architectures

Based on the collaborative discussions
among ACS, GPA, ASU, UCB, and TAMU,
ACS has
mapped the

RATC communication network data flows to actual physical communication
network topology components
. Per TAMU request, this has been performed for (1) completely
centralized RATC deployment and (2) a decentralized and distributed RATC deployment
[ACS/TAMU conference call, 2012
-
09
-
28].

3.1

Centralized

Figure
2

shows the communication network architecture with RATC network data flow paths
and related protocols for a centralized deployment. The centralized communication architecture
shown here corres
ponds to a RATC system implementation should RATC be deployed today.


Figure
2

RATC network flow communication paths and applicable protocols/standards for centralized
architecture

Referring to
Figure
2

above, descriptions of the centralized (i.e. CC
-
oriented) RATC
communication network data flow Application Layer protocols/interfaces are as follows.

A.

Real
-
time measurement data will be obtained by the Grid Measurement Component
(GMC) from SCADA/EMS information systems. The interface for this real
-
time data
exchange is based on the
Inter
-
Control Center Communications Protocol

(ICCP).

B.

For the RATC Relatio
nal Database Management System (RDBMS), standard database
-
to
-
database Application Layer interfaces (e.g. SQL
-
based, Oracle
-
based) will be used to
obtain utility Planning Data and Electrical Network Model Data (ENMD).

C.

Secure file transfer utilities based on

SSH/SFTP
will be used by RATC Data
Management to load input data files to the RATC Analytics file system for the RATC
RATC Task 1 Report

Applied Communication Sciences



7

algorithms (e.g. Optimization)

to use

in their computational procedures
3
. These input files
will be loaded to a file system location know
n by the algorithms so that file system read
operations
are
sufficient for them to obtain the input data needed. Similarly, when the
RATC algorithms compute outputs (e.g. optimized lines to be switch), th
e

output
s

are

written to the RATC file system and
ca
n be
obtained by the RATC Data Management or
by
external users
via

SSH/SFTP.

D.

Simple TCP/IP socket
-
based client/server (C/S) message passing will be used by RATC
components to send and receive event triggers. These include (a) optimization run
requests from

the Operator, (b) request and input file notification trigger
s

from the RATC
Data Management component to the RATC Analytics, (c) output file notification
s

from
the RATC Analytics to RATC Data Management and (d) optimization run complete
notification
s

sen
t to the Operator. TCP sockets are recommended for this pu
rpose
(instead of UDP sockets)
s
ince

they ensure reliable transport and should afford
sufficiently low
-
latency communications for the expected types of C/S requests and
notification triggers.

E.

As one

of the pertinent non
-
RATC interfaces, ICCP is understood to be used by utility
systems to share SCADA/EMS information between the Transmission Operator (TO
p
)
4

CC and Reliability Coordinator (RC) CC.

F.

As another pertinent non
-
RATC interface, ICCP is
also
un
derstood to be used
communicate the RATC
-
computed optimized lines to be switched
from the RC CC to the
TOp

CC.

Other candidate Application Layer protocols/interfaces for RATC data exchange include HTTPS
and pub/sub tools. However, based on discussions with

RATC team members, such methods are
not needed
for the initial software development effort and likely not needed even for the final
software product.

3.2

Decentralized and Distributed

Figure
3

shows the candidate communication network architecture with RATC network data
flow paths and related protocols for a
future

decentralized/distributed deployment. The
decentralized communication architecture is distributed across multiple substations and CCs
corresponds to the future RATC system deployment. The figure additionally shows
communication
interface
s
for
substation

a
utomation (SA)

RATC
data flow
s (lavender block of
Table
1
). Of particular interest are the communications with legacy substation elements. It is
expected that initiall
y such intra
-
substation communications between the substation RATC
components and the substation IED/RTU elements will be DNP3
-
based

(over TCP/IP)
. Long
term, though, there may be a future transition to substation communications based on IEC 61850.
In gene
ral, the implementation concepts for a decentralized/distributed RATC deployment are
still evolving and subject to change.




3

The operator may potent
ially load file inputs as well via SSH/SFTP.

4

NERC model description uses the term Transmission Operator (TOp). The term “Transmission System Operator
(TSO)” may be used alternatively in the RATC Architecture document
[8]
.

RATC Task 1 Report

Applied Communication Sciences



8


Figure
3

RATC network flow communication paths and applicable protocols/standards for
a
distributed
archit
ecture

Referring to
Figure
3

above, descriptions of the
Application Layer protocols/interfaces for
decentralized and distributed

(i.e. substation
-
oriented)

RATC commu
nication network data flow
s

are as follows.

G.

The substation GMC
can

obtain real
-
time measurements from substation IED, relay and
front end processor (FEP) elements. The Application Layer protocol/interface for
obtaining this measurement data is understood to be DNP3
from the substation RTU
encapsulated over TCP/IP
.

This i
nterface will also support

processing

of

COMTRADE
files from DFRs and CBMs
. The applicable RATC algorithms that potentially consume
this substation data include Circuit Breaker Operation Evaluation (CBOE).

H.

The substation RATC Analytics will
potentially
send commands directly to substation
relay elements. The applicable RATC algorithm for this operation is the Relay Settings
Adjustment algorithm
which would send

a specified pre
-
computed and pre
-
loaded
protective relay setting

index to relays within the su
bstation
. The Application Layer
protocol/interface for such RATC commands is TBD
but is assumed to

be DNP3 over
TCP/IP.

I.

For inter
-
substation communications between RATC elements, this presumably w
ill

be
for
coordinating relay setting changes as well as det
ermination of line availability
(breaker state) in adjacent substations. A mechanism for CIP
-
compliant exchange of
information among substations is
use of the

Grid Protection Alliance (GPA) Gateway
Exchange Protoco
l (GEP).

GEP is an open source

publish/sub
scribe

protocol

that
is used
for phasor data exchange
.

RATC Task 1 Report

Applied Communication Sciences



9

J.

Similarly for communications
among the many

substation GMC elements and th
e CC
GMC, GEP could be

used for sharing or aggregating measurement data.

Based on discussions with GPA, there is an additional
WAN design
considera
tion of relevance
for decentralized WAN communications between substation and the RC CC.
Referring again to
Figure
3
, the RC CC is shown internetw
orked via an RC private util
ity network
to
a T
O
p

CC

that
collects SCADA information from substation sites
via

the private utility SCADA network. There
is an implicit understanding in the communication flows shown that direct peer
-
to
-
peer
communication betw
een the substation RATC Data Management host and the RC RATC Data
Management
host
is not possible, perhaps due to firewall security policies. As such, substation

measurement are collected at
a
T
O
p

RATC Data Management host and then delivered to the RC
RATC

Data Management system. This introduces the following additional communication flow.

K.

F
or communications
between

the
T
O
p

GMC elements and the
R
C GMC, GEP could be
used for sharing
measurement data.

Last, there was at one point in time, a notion that RATC
Analytics command and control
messaging might be sent over the WAN between the CC and substation RATC elements.
However, it is not clear whether such communications are part of the future vision for the RATC
Analytics. Further discussions with the RATC al
gorithm designers are needed to determine
whether th
ese communications are

envisioned
as

part of a decentralized RATC deployment.

3.3

Legacy System Data Sources

As shown above in
Figure
3
,

DNP3 over TCP/IP will be utilized by the substation RATC Data
Management functionality for communications with the substation IED, relay and front end
processor (FEP) elements.
Essentially, the DNP
3 protocol is encapsulated end
-
to
-
end by

a
TCP/IP connection as illustrated below
Figure
4

(adapted from
[5]
)

wher
e the substation RATC
Data Management system receives sensor data over an Ethernet LAN
.
One important finding
from
[5
]

is that DNP3 over TCP/IP incurs delays that

are likely excessive
for
highly delay
-
sensitive applications such as relay protection.
While the TCP/IP encapsulation

does

incur
additional processing delay at the
sender and receiver,
the bulk of the problematic delay is
actually incurred by the
DNP3 pr
otocol components.

DNP3 over TCP/IP, however, may still be
a viable interface to legacy systems for state reporting and file transfer applications that are less
sensitive to communication delay than relay protection. It is understood

that

for the RATC Grid
Measurement Component (GMC)

the DNP3 adapter will be based on the Green Energy DNP3
adapter product [ACS/GPA teleconference, 2012
-
Sep
-
26].

RATC Task 1 Report

Applied Communication Sciences



10


Figure
4

DNP3 over TCP/IP stacks


3.4

Future System Data Sources

Among the
candidate

data object models applicable to
future
RATC
substation
communications
is the IEC 61850 standard for substation automation (SA). IEC 61850 defines mappings
between SA data models and underlying communication protocols, of which two mappings are
particularly important here. First, IEC 61850
-
9 defines the mapping of time
-
critical raw
measurement data samples and Generic Object Oriented Substation Event (GOOSE) data
directly to the Ethernet link layer. (The time
-
critical GOOSE data items are disse
minated across
the LAN by Ethernet
-
based multicast
[7]
.) Second, IEC 61850
-
8 defines the mapping of client
-
server communications to the Manufacturing Message Speci
fication (MMS) protocol which is an
industry standard protocol that operates at the Application Layer in the TCP/IP protocol stack.
Figure
5

below, adapted from
[10]
[12]
, depicts the interface between

the

IEC 61850 model and

the

communic
ation protocol stack

including a potential future IEC 61850 RATC interface
adapter
.

RATC Task 1 Report

Applied Communication Sciences



11


Figure
5

IEC 61850 o
bject
m
od
el interface with p
rotocol
s
tack

Based on discussions with

GPA

[2012
-
10
-
11]
, it is projected that future

IEC 61850 interface
adapter
would comprise the entire set of IEC 61850 data model and services.




RATC Task 1 Report

Applied Communication Sciences



12

4

RATC Network Flow Event Sequences

RATC network data and control flow
s

are driven by (1) the RATC algorithm inputs/outputs, (2)
the type of data or control s
ignals that must be communicated, (3) the sources and receivers of
the network flows and (4) the communication network infrastructure to be traversed by the
network flows. The steps performed for completing RATC network flows may be described by a
sequenc
e of operations, or
flow event sequences
, that
culminate in the delivery of power systems

data or RATC control signals.

In the following sub
-
sections, examples are given for
communication event flows for the projected centralized deployment architecture (
Section
4.1
)
and for a possible decentralized, substation deployment architecture (Section
4.2
).

4.1

Centralized Optimization Communication Event Flow

Example

As an example of a data flow communication event sequence, the scenario of an RC operator
request for optimization is considered.
Figure
6

shows the flow event sequence for this scenario

in the context of a centralized RATC deployment in the RC CC
. It is understood that as
preconditions the RATC Relational Database Management Sy
stem (RDBMS) and Grid
Measurement Component (GMC) have been populated
with
static/quasi
-
static model and
dynamic measurement data, respectively [GPA “RATC Optimization Suite” schematic, 2012
-
10
-
01].


Figure
6

RATC data flow sequen
ce: Optimization algorithm run

Figure
7

shows the corresponding detailed communication data flow event sequence diagram for
this scenario. As shown in this sequence di
agram figure, there are

6 lifelines corresponding to the
Operator and

5

RATC components interacting in the communication flow sequence: RATA Data
Management, RDBMS, GMC, RATC Analytics

(Optimization algorithm)

and RATC File
System (matching the 5 RATC comp
onents shown in the communication network architecture of
Figure
2
). These components communicate with one another to trigger and/or control subsequent
operations in
the communication event flow via a mix of IP socket
-
based client/server
transactions, standard SQL database calls, custom database (i.e.
GMC
) calls, standard secure file
transfer protocol (SSH/SFTP) and standard file read/write access. The detailed step
-
by
-
step
sequence of operations triggered by these communications are numbered as 1 to 11 in
Figure
7

with the relevant Application Layer communication protocol/action s
hown in bold.

RATC Task 1 Report

Applied Communication Sciences



13


Figure
7

Data flow event sequence diagram for a RC Operator optimization request

Operation 8 (Compute optimized lines to be switched) of the above figure includes calls to the
Overvoltage Check and Stability Check
algorithms (and, therefore, file system read/write
communications are also annotated for this operation).

Figure
8

shows communications data
flows for the iterative i
nteractions between the Optimization algorithms and the Overvoltage and
Stability Check algorithms.

RATC Task 1 Report

Applied Communication Sciences



14


Figure
8

Data flow event sequence diagram for Overvoltage and Stability checks

4.2

Decentralized Substation Communication Event Flow

Example

As a representative

example of a RATC communication network flow event sequence using
WAN communications

for a future decentralized, substation
-
oriented RATC architecture
,
Figure
9

shows the event sequence for substation

data

collection to support the RATC circuit breaker
operation evaluation (CBOE) application. CBOE is one of the proposed algorithm modules in
the RATC architecture
[8]
.
In this example there are RATC interactions both between RATC
components
collocated within a substation, between RATC Data Management components
situated at the substation and CC, and b
etween RATC components collocated at the CC.

The
intermediary RATC components
shown
at the T
O
p

CC

are, again, based on the understanding
discussed earlier in the context of
Figure
3

that there may not be direct peer
-
to
-
peer
internetworking between substation and RC CC systems.

RATC Task 1 Report

Applied Communication Sciences



15


Figure
9

Example RATC data flow event sequence: Data c
ollection

for
circuit breaker operation e
valuation

Figure
10

shows
a

sequence diagram rendering of the data collection for CBOE.
As shown

in
this sequence diagram
, there are
11

li
felines corresponding to substation automation (SA) RATC
components (SA
-
RATC Analytics, SA
-
RATC Data Management, SA
-
RATC GMC, SA
-
RATC
File System), the substation CBM IED and CC RATC components (CC
-
RATC Data
Management, CC
-
RATC
GMC, CC
-
RATC File System,

CC
-
RATC Analytics

plus the
intermediary CC
-
RATC Data Management and CC
-
RA
TC GMC components at the T
O
p

CC
). In
addition to
using
many of the Application Layer protocols/interfaces employed for the
Optimization request communication event sequence diagram (
Figure
7
), the sequence diagram

below

also shows the use of
DNP3 over TCP/IP

for interactions with the CBM IED and
Gateway
Exchange Protocol

(GEP) for WAN communications between the RATC Data Management
components at the substation and CC
.

RATC Task 1 Report

Applied Communication Sciences



16


Figure
10

Sequence diagram rendering of data collection for circuit breaker operation evaluation






RATC Task 1 Report

Applied Communication Sciences



17

5

Candidate
Substation Physic
al Topolog
ies and Performance Considerations

For
RATC
communications within a substation, timely delivery of IED measurement represents
arguably the most critical QoS
-
related network performance
criterion
. This section, therefore,
identifies the candidate

physical network topologies relevant to the SA RATC intra
-
substation
communications. Th
e chief
RATC
data flow type of interest

within a substation

is the reporting
of measurement data from the substation IED elements to the RATC (or substation)
Data
Mana
gement system
.
Figure
11

summarizes the RATC data flow dependencies on the intra
-
substation physical topology.

Relevant RATC data:

Digital fault recorder (DFR), digital protection relay (DPR) and circuit
breaker recorder (CBR) event records

Relevant standards/protocols:

DNP3 over TCP/IP,
IEC 61850 object model

(
future
)
, MMS,
TCP/IP, Ethernet extensions (e.g. IEEE 802.1q)

Risks/Issues:



For DNP3 access to legacy systems, millisecond delivery response likely not possible



For
future system
sub
-
millisecond delivery of time
-
critical IED data, switched network
should be Gigabit

Ethernet



Ethernet link for RATC data concentrator co
uld be a bottleneck due to collection of IED
data and dissemination of such data with the RATC modules and the

CC

Mitigation(s):

Future n
etwork elements should support

mapping of time
-
critical IEC 61850
objects to IEEE 802.1p QoS extensions
. RATC specific
communication control mechanisms
need to be developed.

Figure
11

RATC data flow dependencies on intra
-
substation physical topology

As discussed earlier, RATC intra
-
substation data communications for a future decentralized
RATC dep
loyment will potentially demand of mix of legacy (e.g. DNP3) and future (e.g. IEC
61850) protocols and standards. Furthermore, RATC communications must consider potentially
a mix of physical network topologies.
Substation communications have evolved over t
he years
from conventional discrete copper wire interconnections. While conventional wiring is still
applied in some substations, other substations have migrated to high
-
speed, redundant
,

switched
Gigabit Ethernet local area networks (LANs). Therefore, s
ubstation physical topologies
typically match one of several stages
in the transition process from discrete wire connectivity to
connectivity based fully on switched Ethernet.

The following figures adapted from

[1]

show examples of typical substation physical topologies
ranging from legacy conventional wiring connectivity (
Figure
12
)

to complete
ly

switched
Ethernet connectivity (
Figure
15
)
.

The migration to Ethernet
-
based substation communication
network (SCN) architectures has been driven by three key factors. First,

switched Ethernet
offers attractive scalability and ease
of management/maintenance advantages over conventional
wiring.

Second,
the advancement of high
-
speed (i.e. Gigabit per second) Ethernet technology
now satisfies the low
-
latency delivery needs of time
-
sensitive raw measurement data and
GOOSE events.

Third
, the deployment of IEC 61850
-
compliant intelligent electronic devices
(IEDs) provides a

single unifying

interface to the TCP/IP stack (and Ethernet) that was not
available in legacy devices.

RATC Task 1 Report

Applied Communication Sciences



18

Figure
12

shows the most basic transition to an Ethernet
-
based SCN architecture. The individual
substation bays still rely on conventional wiring to interconnect the bay control unit (BCU) and
relay elements with the bay switchgea
r and current transformer (CT) and voltage transformer
(VT) devices.

However, Ethernet
connectivity
is employed
between individual substation bays
and the substation
computing systems where distributed RATC substation algorithm
s

(e.g.
CBOE


Circuit Break
er Operation Evaluation)

and the
SA
-
RATC Data

M
anagement

component

reside.

The s
ubstation gateway router also affords remote access to the BCU and bay
relay elements from the control center (CC) in addition to interactions between the distributed
substati
on RATC
components
and the CC instance of the RATC
system
.


Figure
12

DNP3 (legacy) or
IEC

61850
-
8
(future)
station bus and conventional wiring to the process

Figure
13

shows the evolution of the substation physical topology to additionally include direct
serial connections, according to IEC 61850
-
9, with CT/VT elements within the individual
substation bays.

The deployment of such CT/VT elements combined with the IEC 61850
interface enhances equipment monitoring
by ensuring interoperability across different vendors

and reduces the number of proprietary interfaces

in the logical path

between the SA
-
RATC
compo
nents and the end devices
.

RATC Task 1 Report

Applied Communication Sciences



19


Figure
13

DNP3 (legacy) or
IEC

61850
-
8

(future)

station bus
with

IEC

61850
-
9 links to non
-
conventional
instrument transformers

Figure
14

shows the evolution of the SCN architecture to additionally include

the complete
incorporation of switched Ethernet within individual substation bays. Here, the deployment of
modern switchgear with IEC 61850 interfaces to switched

Ethernet affords significant savings
through the reduction in copper wiring required and the simplification of interconnectivity with
devices.

SA
-
RATC Data Management, however, still obtains the bulk of the substation/bay
measurement through the front end

devices with TCP/IP interfaces.


Figure
14

Hierarchical communication networks with
DNP3 (legacy) or
IEC61850
-
8

(future)

as station bus
and a process bus using both IEC 61850
-
8 and IEC

61850
-
9

within substation bays

RATC Task 1 Report

Applied Communication Sciences



20

Last,
Figure
15

shows the complete migration

to

switched Ethernet technology. This non
-
hierarchical topology affords fully decentralized peer
-
to
-
peer communications among s
ubstation
elements.

Direct switched Ethernet access to the substation/bay measurement devices minimizes
the delay in RATC data acquisition.
Furthermore, access via the substation gateway router
potentially supports direct communications between the contro
l center and individual substation
bay sensor elements.


Figure
15

One single station
-
wide communication network
bus
using

DNP3 (legacy) or

both IEC

61850
-
8 and
IEC

61850
-
9

(future)

with

additional

bay
-
level digital fault recorded

(DFR) additionally shown




RATC Task 1 Report

Applied Communication Sciences



21

6

Typical WAN Design for Power Systems

and Related QoS Considerations

The wide area network (WAN) for power systems operations (PSO) supplies the key network
infrastructure for communications between geographically
-
dispersed
power utility sites/facilities.
Among the facility types interconnected by the WAN include control centers suc
h as a
transmission operator

(
TO
p
)
CC
and
a
reliability coordinator

(R
C)

RC
,
plus
various types of
power generation plants
such as
potentially a
mix of
fossil plants (FPs), hydroelectric plants
(HPs), nuclear plants and combustion turbine plants, as well as numerous substations.

Of particular importance to RATC

WAN

communications is the ability for the utility network
operator to manage how traffic

is serviced with

Differentiated Services

(
DiffServ
)

according to a
packet’s priority/urgency as indicated by it
s

D
iffServ C
ode
P
oint

(
DSCP
) field. In utility
-
owned
portions of the WAN infrastructure (i.e.
Figure
18
), the marking and servicing of RATC IP
packets should be coordinated with utility network operation procedures to ensure they are
treated appropriately. In portions of the WAN infrastructure not owned by
the utility (i.e.
Figure
19
), however, explicit QoS control based on DiffServ may not be possible. Furthermore, as a
general WAN issue, WAN link outages and degradati
on can adversely impact inter
-
site
communications, particularly for those sites with non
-
redundant, single
-
homed connectivity
to
the utility’s WAN core. These issues are summarized in
Figure
16

below.

Risk/Issue 1:

Using non
-
utility
-
owned WAN resources means less control over network flow
QoS

Mitigation:

RATC WAN flows should be deployed in close coordination with other utility
flows
so as to use WAN resources in a manner that protects urgent/priority data

Risk/Issue 2:

Single
-
homed remote are highly
-
vulnerable to WAN link failure/degradation

Mitigation:

Communication performance disturbance analysis is needed for
RATC
WAN
traffi
c

Figure
16

Summary of RATC WAN communication risks/issues

The following figures adapted from
[4]

together form an

illustrative exa
mple o
f a power system
WAN.

Figure
17

shows a

high
-
level
view of the

WAN design
.

Here, a utility
-
owned high
-
speed
redundant backbone network interconnects eight core netw
ork sites.

Based on the
network
infrastructure used by the Tennessee Valley Authority (TVA) WAN
given in

[4]
, the core
backbone WAN
should be an IP
-
based network
consisting of ISP
-
grade site routers
interconnected by dedicated (utility
-
owned) high
-
speed G
igabit Ethernet

and OC
-
3 primary
path
s
and DS
-
1/DS
-
3 secondary
path
s.
Furthermore, the core link topology should incorporate at least
ring
-
based path redundancy t
o support automatic failover to a backup path when a link in a
primary path fails.

RATC Task 1 Report

Applied Communication Sciences



22


Figure
17

Illustrative high
-
level WAN design

As mentioned above, remote utility sites (e.g. substations and power generation plants) are
connected by various non
-
backbone networks to the core WAN infrastructure. These satellite,
non
-
backbone WAN components that hub into the WAN core may consist eith
er of entirely
utility
-
owned network components or may leverage public network infrastructure.
Figure
18

shows
utility
-
owned

non
-
backbone WAN connectivity to remote u
tility sites.
Depending on site
network reliability and performance requirements, a mix of link technologies may be deployed
to
satisfy
disparate site internetworking needs. As shown here, dedicated T
-
1 links with backup
56Kbps backup links, facilitated
by a 1/0 Dig
ital Access Cross
-
connect System

(DACS),
internetwork two of the remote
sites
shown

connected

to

a core

WAN router. Radio links are
also shown to reach two additional remote sites.

K
ey advantage
s

of the utility
-
owned
infrastructure
include th
e ability to reach remote sites not adequately serviced by public networks
and improved cyber
-
security control. Of course, on the other hand, installation and maintenance
costs for utility
-
owned WAN links represent an obvious drawback for this approach.

RATC Task 1 Report

Applied Communication Sciences



23


Figure
18

Illustrative example of r
emote site connectivity using utility
-
owned infrastructure

In contrast with the previous figure,
Figure
19

shows
an example of
non
-
utility owned non
-
backbone WAN connectivity to remote sites

and external utility networks such as the corporate
network
.
Of particular interest here are the additiona
l cyber
-
security infrastructure requirements,
implied
by the firewall icons in the TO
p CC
, for remote site connectivity via the Internet. Other
security measures such as IPSec and
transport layer security (
TLS)
likely are required for utility
communicatio
ns over public infrastructure to ensure data privacy, integrity and authentication.
While the utility will lose some control over the network infrastructure and incur additional
cyber
-
security risks using the architecture of
Figure
19
, it potentially will reduce WAN
infrastructure installation and maintenance costs.

RATC Task 1 Report

Applied Communication Sciences



24


Figure
19

Ill
ustrative example of remote site

co
nnectivity using non
-
utility
owned infrastructure




RATC Task 1 Report

Applied Communication Sciences



25

7

Logical Topologies and Communication Architectures

This section presents the logical topologies and communication architecture pertinent to RATC
communications. Of particular interest to RATC is the identification of potential
network
performance
-
related and network availability
-
related risks, issues and challenges. Where
possible, candidate mitigation measures are suggested.

7.1

Intra
-
Substation Communications

and Availability Considerations

As discussed in Section

5
, a key requirement of intra
-
substation communications is low
-
latency
transport of time
-
sensitive raw measurement and GOOSE event data. As such, the transition to
high
-
speed Gig
abit Ethernet from discrete wire connectivity has been an important
modernization effort of the physical topology within the substation to reduce installation,
maintenance and management costs while still providing timely delivery of substation data. In
t
his section, candidate logical topologies for intra
-
substation networking are presented. Of
particular interest here
are the benefits

various logical topologies
provide
in terms of network
reliability/availability
.

However, even highly redundant, reliabl
e logical topology designs are
vulnerable to some (ideally, small) non
-
zero probability of
network connectivity failure.
The

related

risks to RATC intra
-
substation communications are summarized in
Figure
20
.

Risks/Issues:



Limited physical access to remote

sites

makes SCNs highly vulnerable

to

failure events

(e.g. long mean
-
time
-
to
-
repair)



Cyber attacks on SA systems may disable even highly
-
redundant SCNs

Mitigation(s):

Impact of intra
-
substation communication performance disturbance on RATC
procedure should be analyzed. In particular, communication failure within a single substation
should not adversely impact operation of RATC modules at other sites.

Figure
20

Potential intra
-
substation network availability risks

The following figures adapted from
[2]

show examples of logical topologies for intra
-
substation
communications. The topologies range from basic star (
Figure
21
) and ring (
Figure
22
)
topologies to a parallel redundant double ring topology (
Figure
25
).
Figure
26

presents a
topology option that is also fully redundant but is achieved with a reduced hardware
commitment.

The selectio
n of the appropriate logical topology sh
ould first be based on the
network availability requirements for substation communications. A useful definition for
availability metric for the intra
-
substation communications network context is the expected
fractio
nal amount of time that the network is capable of supporting communications between all
network elements (i.e. computer workstation hosts, IEDs, switches, routers, etc.). For example,

a
requirement of

99
.9
9% availability corresponds

approximately

to

not m
ore than
53

minutes of
network downtime/outage, on average, per year.
The second criterion for

logical

topology
design
is the network management/maintenance cost. Of particular relevance to maintenance
costs are the mean time between failure (MTBF), mean

time to repair (MTTR) and mean time
between replacement (MTBR) metrics.

Figure
21

shows the most basic candidate substation logical topology. Likely, this is an
ina
dequate topology for intra
-
substation communications because each Ethernet switch element
represents a point where failure could cut off portions of the network. In particular, failure of the
centralized substation switch will effectively sever all commun
ications between the substation
bays and the substation management systems.

RATC Task 1 Report

Applied Communication Sciences



26


Figure
21

Star SCN

architecture

Figure
22

shows a ring topology
, the most elementary redundant configuration for protection
against single link failure events. Here, a second substation switch is also present to provide
some degree
of protection against device failure at the substation switch level. However,
connectivity to the individual bays and the
RATC
computing system
s

are single
-
homed so a
single link or single switch device failure can still sever communications to individual

host
systems or to an entire substation bay of IEDs.


Figure
22

Ring SCN

architecture

Figure
23

shows a redundant star logical topology. This clearly is an improvement over the
basic star configuration
Figure
21
. Furthermore
, the addition of the backup substation switch
RATC Task 1 Report

Applied Communication Sciences



27

and the dual
-
homed connectivity from the substation switches to the
RATC

system
s

and gateway
router provide complete single event failure protection for these network elements.
How
ever,
the bay switch elements still represent single points whose failures can completely sever
communications to/from individual substation bays.


Figure
23

Redundant star SCN

architecture

Figure
24

shows a single ring topology with completely redundant dual
-
homed link connectivity
to individual substation, management computing systems and the gateway router. This topology
provides complete protection against any single link or switch element failur
e event.

Likely,
because of it
s

completely dual
-
homed nature, this configuration satisfies the minimum
availability needs for intra
-
substation communications.

Nevertheless, to ensure high network
availability, the network elements, in particular the Ethe
rnet switches, must be continuously
monitored to ensure that device failure events are quickly detected, reported and resolved.
That
is, once a single Ethernet switch fails, should a second fail then a network partition will be
created that must repaired.

For example, to ensure 99.99% availability for a given a calendar
year would likely require a maintenance individual to be physically onsite in order to replace a
failed Ethe
rnet switch within 53 minutes following

a double
device failure. However, onsit
e
presence of maintenance personnel is likely not realistic for many substations.

So, while the
redundant single ring logical topology may afford a mean time between network failure events
that is sufficiently large, the mean time to repair may be prohibi
tively long without onsite
personnel.

RATC Task 1 Report

Applied Communication Sciences



28


Figure
24

Redundant single ring SCN

architecture

Figure
25

shows a completely redundant, parallel doub
le
-
ring logical topology.

Not only are
individual substations and network element dual
-
homed as per
Figure
24
, but the redundant
parallel ring of
backup
Ethernet swi
tches

provides complete failover protection should a partition
failure in the primary Ethernet ring occur.

(Of course, if both switches homing into a particular
individual bay fail then that bay will be disconnected but the core of Ethernet switches will
not
be partitioned.)
Whil
e the initial installation cost

for such a parallel redundant topology

may be
significant
, the level of failure protection afforded by this configuration is likely necessary for
remote substation where physical access to repair an
d replace switch elements is difficult.

RATC Task 1 Report

Applied Communication Sciences



29


Figure
25

Redundant parallel double ring SCN architecture

Figure
26

shows a modification to the redundant parallel ring topology of
Figure
25

to reduce
somewhat the cost

and space required of a redundant parallel ring of Ethernet switches. Here,
any two Ethernet switch elements can fail and still the core of Ethernet switches will still remain
connected. This modified parallel double ring topology combined with a remote

network
management system to monitor the health/status of the substation switch elements likely
provides a practical balance
of
intra
-
substation network availability and cost.

RATC Task 1 Report

Applied Communication Sciences



30


Figure
26

R
edundant parallel double ring SCN archite
cture with reduced numbe
r of switches

Last,
Figure
27

shows the modified redundant parallel double ring logical topology of
Figure
26

with the breakout of elements of substation bay 1 presented in detail.
Of particular importance
here is the dual
-
homed connectivity of individual bay elements to the redundant Ethernet bay
switches.

This is crucial for protection against single link failure events within an individual
substation bay.

RATC Task 1 Report

Applied Communication Sciences



31


Figure
27

R
edundant
parallel d
ouble
ring SCN a
rchitecture with reduce number of switches and redundant
bay unit architecture

shown

Of course, double
-
failure protected SCNs as shown in
Figure
27

may not be provided for all
sites.
Furthermore, network faults due to targeted malicious

attacks

may thwart

even

well
-
designed, redundant SCNs.

7.2

Inter
-
Substation Communications

and QoS
-
Relate
d Performance Considerations

Recent advances in Ethernet technology have afforded
extension

of Ethernet
deployment

from
the LAN to the WAN
[3]
[6]
.
Figure
28
, adapted from
[8]
, shows an example of inter
-
substation
connectivity based on Gigabit Ethernet virtual private LAN service (VLPS).

To
help
support
preferent
ial treatment of time
-
sensitive or high
-
priority traffic it is recommended that the VPLS
WAN deployment leverage the Ethernet standard’s

for VLANs (IEEE 802.1q) which
incorporates the

extension

for media access control (MAC) level quality of service (QoS).

RATC Task 1 Report

Applied Communication Sciences



32


Figure
28

Inter
-
substation connectivity via wide area VPLS

In practice, however, many inter
-
substation connections will leverage low
-
capacity WAN links
(e.g.
Figure
18

and
Figure
19
)

and
even
with VPLS available, there are potential communication
issues.
Figure
29

s
ummarizes the
related RATC inter
-
substation communication
challenges
.

Relevant RATC data:

Notification of fault events and batch IED (e.g. DPR, DFR, CBR) data

Relevant standards/protocols:

DNP3,
IEC 61850

object model, MMS, TCP/IP, IEEE 802.1q

Risk/Issue 1:

Even with Gigabit Ethernet VPLS, sub
-
millisecond delivery of data between
substations over a the WAN is not likely feasible

Mitigation:

SA decisions that demand responses on the order of milliseconds af
ter IED event,
likely need to be made based on data already available to the local RATC agent

Risk/Issue 2:

When high
-
speed WAN connectivity is not available, e.g. due to failure of primary
link) or WAN coverage limited to slow (56Kbps) links, RATC
inter
-
substation throughput will
be low

Mitigation(s):

RATC agents should be “resource
J
aware” and filter (e.g. drop) or mark (e.g.
apCm⁶alueF acc潲摩ng t漠the 灲i潲it礯urge湴 潦 the inter
J
su扳tation 摡ta

Figure
29

Inter
-
substatio
n communication challenges


7.3

Communications Between Substation and Control Center

and WAN C
apacity
Constraints

The key communication content pertinent to RATC between substations (and power generation
plants) and the Control Center (CC) is the supervisory control and data acquisition (SCADA)
information employed by a utility’s
energy management system

(EMS). As sh
own in
Figure
30

(adapted from
[13]
)
, SCADA information is exchanged betw
een the CC systems and remote
terminal units (RTUs) situated at power generation plants and substations inter
-
connected by the
utility WAN. Modern RTU elements are either packaged as part of IP
-
enabled intelligent
electronic devices (IEDs) or connected to

front end processor (FEP) units that provide an IP
-
based interface with the utility communication network.
Measurements from the RTU elements
are delivered near
-
real
-
time
to
local engineering tools (such as distributed RATC
components
).
Computing system
s collocated at the CC site host EMS
-
related applications
(including
RATC Task 1 Report

Applied Communication Sciences



33

engineering tools such as the centralized RATC
components
)
and
are
interconnected by the CC
campus network
s
.


Figure
30

Overview of SCADA and EMS communications for control center management of RTU systems

In terms of related risks and potential issues, it is important to emphasize that
RATC elements
situated at the CC likely will share network resources with EMS elements

and that
RATC
comm
unication
s between CC
-
RATC and SA
-
RATC
element
s will share the utility WAN
resources with SCADA communications
.

Figure
31

summarizes the risk and p
otential mitigation
associated with sharing backhaul WAN resources from substation
s

with SCADA
communications.

Relevant RATC data:

SA
-
RATC agents send periodic status and event
-
driven alerts to CC
-
RATC; CC
-
RATC sends topology control commands to SA
-
RATC ag
ents
.

Risk/Issue:

If a WAN path between CC and substation
is

congested, critical RATC event
reports
or command
s

may be lost or unacceptably delayed

Mitigation(s):

DSCP values of RATC data packets should be marked according to
priority/urgency and

the

utili
ty WAN should support DiffServ. RATC
-
specific communication
control algorithms

may

need to be deployed.

Figure
31

RATC substation


control center communications risk


RATC Task 1 Report

Applied Communication Sciences



34

7.4

Intra
-
Control Center Communications

and Potential EMS Network Contention

Figure
32
, adapted from
[13]
, shows
the keys functional areas of the control center

(CC)

energy
management system (EMS) plus its interactions with the control center business management
system (BMS) and remote site operations.

RATC modules at the CC access the SCADA/IED
information and mode
l data maintained for the EMS and BMS. RATC access to these shared
databases is via the CC EMS
system enterprise

network.

Sharing the EMS data will help reduce
the RATC dependency on communications o
ver

the WAN to obtain IED and model data.

The
RATC syst
em will also make its outputs (e.g. lines to be switched as computed by the
Optimization algorithm) available for presentation to an Operator via a TBD user interface.

For
initial design considerations, it is understood that the RATC outputs will available

as text files
written to the RATC Analytics file system.


Figure
32

Control
center EMS and BMS interactions

Figure
33

shows a breakout RATC

intra
-
CC communications identified in
Figure
2
.
Of particular
importance for QoS and cyber
-
security considerations
are

the CC campus network
s

that
interconnect RATC components (i.e. RATC Data Management, RATC Analytics), SCADA
components
and utility database systems. It is
an

understanding for the initial RATC design that
the underlying network
infrastructure
for intra
-
CC RATC communications will be high
-
speed,
redundant Gigabit Ethernet.

RATC Task 1 Report

Applied Communication Sciences



35


Figure
33

Breakout of RA
TC
intra
-
CC communication

network flow
s
: (A) ICCP for SCADA information;
(B)
Database interfaces for ENMD and Planning data; (C) SSH/SFTP for file
-
based information exchange;
(D) IP socket
-
based messaging for triggers/notifications.

While the CC EMS networ
k is likely to be a high
-
speed switched Ethernet LAN,
the sheer
magnitude of
power systems data
content maintained on servers located at the CC represents a
potential source of LAN congestion should it be accessed in an unregulated manner.
The
potential r
isk and associated mitigation for RATC intra
-
CC communications is summarized
below
in
Figure
34
.

RATC Task 1 Report

Applied Communication Sciences



36

Relevant RATC data:

As discussed previously, CC
-
RATC
Data Management
w
ill
leverage

CC
campus
network resources to obtain SCADA/RTU information

via ICCP
.

Furthermore, the CC
-
RATC

Data Management

will also access Model data (static, dynamic) including model data
(line, generation, load) and current state updates

via a database
-
to
-
database interface
.

Risk/Issue:

Unconstrained CC
-
RATC
access
over

the

CC campus communication
network to
the voluminous quantity of data maintained at CC
databases
could potentially dis
rupt

CC
network performance
.

Mitigation(s):

RATC int
ra
-
CC communications should incorporate network resource
-
awareness
procedures (TBD) to intelligently access data based on the available CC network capacity
.

Figure
34

RATC intra
-
CC communications data, risk and mitigation


8

Cyber
-
Security Considerations

Cyber
-
security considerations and mechanisms should be an integral part of communication
architecture for RATC.

Since RATC would be adapting the topology of
the
power grid much
more frequently than traditional power
-
system ope
rations, it exposes a serious attack surface.
Communications to and from RATC devices (in substations and control centers) need to be
hardened while providing the required performance (as discussed in previous sections).

An initial
cyber
-
security
assessmen
t of the protocols and communication interfaces identified
Table
1

has been performed.

Table
2

provides a brief summary of this analysis.

Table
2

Summary of Preliminary Cyber
-
Security Protocol/Interface Assessment

Protocol/Interface

Cyber
-
Security Considerations

ICCP

Network
-
based security, e.g. IPSec or VPN tunnels

Database
-
to
-
Database

Role
-
based access control, e.g. achieved through domain
-
based security

SSH/SFTP

Password
-
based or certificate
-
based SSH secures SFTP file transfers

File System

Likely, digital

signatures should be applied to ensure file integrity

GEP

TBD, though, VPN tunnels are a logical solution

DNP3 over TCP/IP

TBD, though, password
-
based device access control is likely present

In general, though, e
nterprise
cyber
-
security solutions
may

not

be

sufficient to protect
against
various vulnerabilities (in particular for communications both
within

and
between

substations)

involving RATC
. This is due to performance restrictions (e.g. involving encryption).

We
might
need to
investigate

both prot
ocol
and

application (
i.e.
RATC) specific, stateful cyber
-
security
mechanisms
.
RATC
cyber
-
security agents need to be installed at each substation and the CC.

One novel approach (that we will pursue) is to develop finite
-
state
-
machine models of proper
behav
iors of protocol (e.g.
IEC
61850) actions specific to RATC (e.g. control of RATC object
models). These FSMs would be used as “sensors” to detect potential attacks and to coordinate
mitigation strategies.

We illustrate an example to such sensors by consider
ing communications among RATC devices
in CCs.
For communications and coordination among RATC computers in CCs, we would create
ICCP associations

as part of the RATC communication architecture
.
The
RATC head
-
end may
need to execute certain SBO

(Select Befor
e Operate)

programs/devices in both the associated
control centers and related substations. An adversary could inhibit these operations by stagin
g
RATC Task 1 Report

Applied Communication Sciences



37

various attacks.
Figure
35

shows
a
n

FSM sensor example to detect such attacks.

In this example,
an adversary would inhibit selection and/or operation of RATC SBO devices/programs by
staging MMS ‘read response’ or ‘event notification requests’ failures. This sensor allows us to
monitor ongoing RATC ICCP communications and set statist
ical checkers to detect onset of such
attacks.

Our future work will

consist of
specifying a cyber
-
security architecture for RATC where RATC
cyber
-
security agents would work with RATC
components
to ensure
cyber
-
protection
.

We will
specify representative vu
lnerabilities
. Should we adopt FSM checker approach (briefly described
in this section) then we could
identify corresponding FSM checkers

for those vulnerabilities
.
Another novel

concept

that we can

consider is incorporation of RATC
-
specific states into th
ese
checkers (not just protocol
-
specific checkers).

Determining these vulnerabilities and
corresponding FSM checkers will require us to understand the standard protocol specifications
(e.g.
DNP3,
IEC 61850, ICCP, C37.118, etc.) in detail, which we will try

to tackle (as allowed
by our budgeted hours).

In anyhow, the cyber security architecture for RATC is the subject of a
later Task, however we started our investigation as it is related to the communication
architecture.

For instance, a RATC security agent
may have to be placed in front of all critical
RATC components (e.g. GMC). We will elaborate these in detail in the respective Task.



Figure
35

A FSM sensor for RATC, involving ICCP communications




RATC Task 1 Report

Applied Communication Sciences



38

9

References

[1]

L. Andersson, C. Brunner and F. Engler, “Substation automation based on IEC 61850 with new
-
process close
technologies,”
Proc. IEEE PowerTech 2003
.

[2]

A. Ali and M. S. Thomas, “Substation communication networks architecture,”
Proc. IEEE POWERCON
2008
.

[3]

G. Chiru
volu, A. Ge, D. Elie
-
Dit
-
Cosaque, M. Ali and J. Rouyer, “Issues and approaches on extending
Ethernet beyond LANs,”
IEEE Comm. Mag.
, March 2004, pp. 80
-
86.

[4]

ESP
-
TCP
-
LTA
-
2008, Rev. 0000,
Assessment of the PSO Telecommunications Infrastructure
, pp. 21
-
37
.

[5]

X. L
u, Z. Lu, W. Wang and J. Ma, “On network performance evaluation toward the smart grid: a case study
of DNP3 over TCP/IP,”
Proc. IEEE GLOBECOM 2011
.

[6]

F. Matera et al. “Network performance investigation in a wide area gigabit Ethernet test bed adopting all
-
optical wavelength conversion,”
IEEE Photonics Technology Letters
, Vol. 20, No. 24, December 15, 2008,
pp. 2144
-
2146.

[7]

S. Mohagheghi, J. Stoupis and Z.

Wang, “Communication protocols for power systems


current status and
future trends,”
Proc. IEEE PSCE 2009
.

[8]

T. Popovic,
Robust Adaptive Topology Control Architecture Specification
, DRAFT, August 28, 2012.

[9]

M. Qureshi, A. Raza, D. Kumar, S.
-
S. Kim and H.
-
S.

Yang, “A communication architecture for inter
-
substation communication,”
Proc. IEEE Intl. Conf. Comp. Info. Tech. Workshops
, 2008.

[10]

T. Sidhu and P. K. Gangadharan, “Control and automation of power system substation using IEC61850
communication,”
Proc. IEEE

Conf Ctrl App
, August 28
-
31, 2005.

[11]

Teleconferences with TVA, August 23 and September 6, 2012.

[12]

M. Thomas and I. Ali, “Reliable, fast and deterministic substation communication network architecture and
its performance simulation,”
IEEE Trans. Power Delivery
, Vol. 25, No. 4, October 2010, pp. 2364
-
2370.

[13]

F. F. Wu, K. Moslehi and A. Bose, “Power system control centers: past, present, and future,”
Proc IEEE
,
Vol. 93, No. 11, Nov. 2005, pp. 1890
-
1908.




RATC Task 1 Report

Applied Communication Sciences



39

10

Appendix A
: Acronyms

Table
3

identifies acronyms related to the RATC algorithm, PSO and communication networks
that are pertinent to this report.

Table
3

Acronym List

Acronym

Meaning

BCU

Bay
Control Unit

BPU

Bay Protection Unit

CBM

Circuit Breaker Monitor

CC

Control Center

CDF

Common Data Format

CT

Current Transformer

DACS

Digital Access Cross
-
connect System

DNP

Distributed Network Protocol

FEP

Front End Processor

FP

Fossil Plant

GEP

Gateway Exchange Protocol

GMC

Grid Measurement Component

GOOSE

Generic Object Oriented Substation Events

HP

Hydroelectric Plant

ICCP

Inter
-
Control Center Communications Protocol

IEC

International Electrotechnical Commission

IED

Intelligent
Electronic Device

ONU

Optical Network Unit

PMU

Phasor Measurement Unit

PSO

Power Systems Operations

PT

Potential Transformer

RDBMS

Relational Database Management System

RTU

Remote Terminal Unit

SA

Substation Automation

SCADA

Supervisory Control And

Data Acquisition

SCN

Substation Communication Network

TOp

Transmission Operator

TSO

Transmission System Operator

VPN

Virtual Private Network

VT

Voltage Transformer

WAN

Wide Area Network