INTEROPERABILITY TEST AND CERTIFICATION MANAGEMENT ASSISTANCE ANALYSIS

verdeagendaElectronics - Devices

Nov 21, 2013 (3 years and 4 months ago)

122 views

Interoperability

Test

and Certification Management Assistance Analysis

Version


0.
80

i

UCA International Users Group

February
2
4, 2014


INTEROPERABILITY TES
T
AND CERTIFICATION
MANAGEMENT ASSISTANC
E
ANALYSIS




Prepared for:

The

UCAIug OpenADE
Task Force,
UCAIug SG



Prepared
by:

The
UCAIug OpenADE
Task Force and
QualityLogic, Inc




Managed by
:

UCAIug OpenADE Task
Force






Version

0.
80





Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

ii

UCA International
Users Group

February
2
4, 2012


Revision History

Rev

Date

Summary

Marked

0.01

2011
-
12
-
21

Initial draft for team review

N

0.70

2012
-
02
-
14

Updated based on OpenADE Task Force feedback and comments by
Marty Burns and Anne Hendry

N

0.80

2012
-
02
-
24

Incorporated comments from 2012
-
02
-
21 OpenADE meeting

and

feedback received via email.

N

Open Editorial Items and Issues Log

As open items and issues are addressed in new versions of this document, they are removed from this list.


Item
No.

Date

Provided
By

Summary of the Issue

Status /
Disposition









Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

iii

UCA International
Users Group

February
2
4, 2012


Executive Summary

Smart Grid Technical Standards are critically important to
development of interoperable
products, increasing

competition
,
reducing costs of Smart Grid components
, and
speeding

implementation

of new technologies
. These key principles
enable

mainstream market
adoption o
f Smart Grid
products and services
.

This is
important for utilities and other
market participants to
be able to make viable

infrastructure investments,
and
reduce
project costs and product developme
nt investments

required to leverage the opportunities
that Smart Grid technologies make available
.

The UCA International OpenADE
Task Force

developed the requirements that led to the
North American Energy Standards Board, (NAESB) REQ18/WEQ19, PAP10 Energy
Usage Information and NAESB REQ21 Energy Services Provider Interface (ESPI)
standards.

T
hese are specifically referenced in the November 8
, 2011,

US Department of
Energy Funding Opportunity to demonstrate or pilot innovative applications based on
standardi
zed energy usage availability.

OpenADE/ESPI

is a key standard in
the NIST
V2
.0 Smart Grid Standards Roadmap

and is the most imp
or
tant standard addressing

technology standardization

for enabling consumer access to energy usage information
.


Two
of the
impor
tant tasks that the
OpenADE
Task Force

plans to undertake are
:

-

Develop a conformance, cer
tification, and testing process and
program for
OpenADE
, coordinated with entities such as standard development organizations
(SDOs), user groups, and Smart Grid activ
ities.

-

Develop programs to allow vendors to develop, test, and demonstrate their ability
to integrate with
OpenADE

communications protocols
.

Creating a test and certification program is a challenging activity, especially in light of
the rigorous requiremen
ts embodied in V1.0
and
D
raft Version 2.0
of the Smart Grid
Interoperability Panel (SGIP) Test and Certification Committee’s (TCC) Interoperability
Process Reference Manual (IPRM)
1
.

This
UCA International OpenADE
Task Force

is
aiming

to establish

a
n
accelerated test
and certification program for OpenADE/ESPI (NAESB REQ 21) that can
:

1.

Accelerate the development and implementation
of
a successful interoperability
certification program for
OpenADE

that meets or exceeds

the
applicable
SGIP
TCC IPRM
requirements




1

See SGIP TCC TWIKI at
http://collaborate.nist.gov/twiki
-
sggrid/bin/view/SmartGrid
/SGTCCIPRM
,
V1.0, November 18, 2010.


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

iv

UCA International
Users Group

February
2
4, 2012


2.

Create a standard test specification,
test cases,
test
scripts,
test harness
,

and other
tools
that can be used by vendors and others to test

intero
perable
OpenADE

products

prior to certification

3.

Establish a maintenance and update process and p
rogram to
e

ensure

currency of
the certification program and pre
-
certification tools

The balance of this
requirements document
will review the requirements of the SGIP
TCC IPRM;
provide commentary on the challenges in achieving the IPRM goals; outline
the tasks required to achieve the goals of
OpenADE

for the
interoperability

test and
certific
ation
;
and suggest

a set of next steps for the
Task Force
.




Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

v

UCA International
Users Group

February
2
4, 2012


Table of Contents

SGIP IPRM REQUIREMEN
TS

................................
................................
................................
......................

7

TRADE ALLIANCE FUNCT
IONS

................................
................................
................................
...................

9

1.1.

ITCA Test and Certification Tasks Based on the IPRM

.....

Error! Bookmark not defined.

1.1.1.

Organize the ITCA (Implied Tasks)

................................

Error! Bookmark no
t defined.

1.1.2.

Manage and Promote the Standard (Implied Tasks)

..............

Error! Bookmark not defined.

1.1.3.

Organize the Certification Program

...............................

Error! Bookmark not defined.

1.1.4.

Define Certification Program (Explicit Tasks)

....................

Error! Bookmark not defined.

1.1.5.

Establish Vendor Partnerships (Implied Tasks)

...................

Error! Bookmark not defined.

1.1.6.

Implement Certification Program (Explicit Tasks)

................

Error! Bookmark not defined.

1.1.7.

Improvements in the Standard and the Certification Program (Explicit Tasks)
Error! Bookmark not defined.

1.1.8.

Cyber
-
Sec
urity (Explicit Tasks)

................................
....

Error! Bookmark not defined.

1.1.9.

Governance (Explicit Tasks)

................................
........

Error! Bookmark not defined.

NEXT STEPS

................................
................................
................................
................................
............

11

SCHEDULE

................................
................................
................................
................................
..............

17

SUMMARY

................................
................................
................................
................................
.............

18







Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

vi

UCA International
Users Group

February
2
4, 2012


Authors

James Mater
, QualityLogic

Steve Van Ausdall
, Xtensible

/ SCE


Edited by: Dave Jollota
, QualityLogic



Interoperability Test and Certification Management Assistance Analysis

Version


0.
70

7

UCA International Users Group

February 14, 2012


1

SGIP IPRM Requirements

1

The
OpenADE
Task Force

is fortunate to have access to a ground
-
breaking guide to
2

developing and managing a world
-
class test and certification program.

Indeed, until the
3

issuance of Version 1.0 of the SGIP IPRM, nothing like it existed for the Smart Grid (or
4

an
y other industry
) that we know of
.

Every trade alliance like
UCA International
5

OpenADE

Task Force

has
had to create its own program
,

which has resulted in

a great
6

deal of variation in how such programs
have
evolved.

Having a roadmap such as the
7

IPRM can
greatly accelerate achieving the goals of product interoperability based on a
8

specific standard.

9

It is also critical to understand that the IPRM defines the standard against which the SGIP
10

will be assessing the quality and maturity of certification program
s for Smart Grid
11

standards.

Although NIST
2

is not directly bound to accept the conclusions and
12

recommendations of the SGIP (in terms of which standards to adopt), it is clear that the
13

SGIP process is closely watched by NIST managers

as well as many regulat
ors and other
14

Smart Grid industry stakeholdes.

SGIP assessments of certification program maturity
are

15

expected to influence the decisions
all stakeholders make
.

16

The IPRM
Version 1
identifies some 86 formal requirements in
five

distinct areas

that
17

serve as a specification for what a good test and certification program for a
Smart Grid
18

technology
standard should look like.

In addition, a number of guidelines are discussed
19




2

NIST is the National Institute of Standards and Technology, US Department of Commerce.

NIST
established the Smart Grid Interoperability Panel specifically to engage a broad range of stakeholders in
assisting with thei
r mission to identify Smart Grid technology standards.


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

8

UCA International
Users Group

February
2
4, 2012


that further clarify how such a program can achieve its goals of interope
rable and
20

conformant products based on the specification.


21


22

Interoperability Test and Certification Management Assistance Analysis

Version


0.
70

9

UCA International Users Group

February 14, 2012


2

Trade

Alliance Functions

23

Like the UCA International IEC 61850 and CIM User Groups, the OpenSG Committee,
24

OpenADE
Task Force

of UCA
,

is taking on the functions of an industry trade alliance
25

when it
assumes the

responsibility for developing and operating an inter
operability
26

certification program for OpenADE/ESPI.

In doing so, it is accepting responsibility for
27

pioneering an activity within the traditional OpenSG and UCA International charter.

The
28

key characteristics of an interoperability certification program tha
t distinguish it from
29

prior UCA International efforts are:

30

1.

The OpenADE
Task Force

is a typical

OpenSG
“requirements” specification
31

activity that assumes other organizations will adopt and formalize an actual
32

national or international technical standard to
implement the requirements defined
33

by OpenADE.

Taking on the certification functions is perhaps a logical extension
34

of the
Task Force

activities but is without precedent in OpenSG.

Implementing a
35

certification program implies a series of new activities tha
t OpenADE will be
36

responsible for including:

37

a.

Defining the conformance certification profile

38

b.

Making major decisions concerning implementation of the certification
39

program as outlined in the IPRM and including additional decisions
40

concerning resources, the c
ertification business model,
and
developing and
41

managing relations with test labs, product developers and others that are
42

new and different from normal OpenSG relationships

43


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

10

UCA International
Users Group

February
2
4, 2012


c.

E
nsuring
quality control and maintenance of the certification program

44

d.

Managing the
issuance and currency of certifications and the awareness of
45

them in the user community

46

e.

Marketing the technology standard and the certification program to

ensure

47

participation and customer awareness

48

2.

The last standard that evolved from OpenSG Committee
work was OpenADR
49

Version 2.

The functions of a trade alliance were taken on by a separate industry
50

organization designed just for this purpose
,

rather than UCA International.

51

3.

UCA International has two precedent activities in the IEC 61850 and CIM
52

interoper
ability test programs.

Both are conducted on a volunteer basis but neither
53

of them meet
s

the requirements set in the IPRM.

To do so will require significant
54

new investment by UCA International
,

as well as taking on additional functions.

55

OpenADE/ESPI could
model itself after the IEC 61850 and CIM User Group activities
56

and may choose to do so

(ignoring SGTCC IPRM conformance)
.

The concerns would be
57

in terms of timeframe and available volunteer labor to develop and manage a successful
58

ITCA program.


59

In sett
ing up and operating an Interoperability Test and Certification Authority (ITCA)
60

(for lack of a better term to describe the functions incumbent on UCA and
61

OpenADE/ESPI), there are a series of activities and responsibilities that are addressed
62

specifically
or implied in the SGTCC IPRM
, most of them enumerated in a separate
63

section.

These will need to be addressed whether the UCA takes on this role
,

or some
64

other entity.

65

The following is an attempt to organize the IPRM explicit and implicit requirements and
66

s
uggested best practices

for an ITCA

into an actual task list as
summarized

in Appendix
67

B, “
ITCA Test and Certification Tasks Based on the IPRM


.

68


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

11

UCA International
Users Group

February
2
4, 2012


3

Next Steps

69

The UCA International
OpenADE

Task Force

can move forward in different ways
,

but

a
70

process could be

the following
:

71



Review
the above Task List and make
key decisions

on how to

develop a formal
72

ITCA and manage it.

The initial key decisions are probably

73

o

Whether or not to develop a formal ITCA

[Should there be an ITCA for
74

OpenADE/ESPI?


Why or why not?]

75


76

AH: Yes.


Because:

77


78

a) the responsibilities of an ITCA, as per
79

http://www.ucaiug.org/default.aspx, appear to encompass most of the
80

declared mission of UCAIug:

81

Influence, select, and/or endorse open and public standards appropriate to
82

the energy and utility

market based upon the needs of the membership.

83

Specify, develop and/or accredit product/system
-
testing programs that
84

facilitate the field interoperability of products and systems based upon
85

these standards.

86

Implement educational and promotional activities

that increase awareness
87

and deployment of these standards in the energy and utility industry.

88

Influence and promote the adoption of standards and technologies specific
89

to the ever
-
increasing Smart Grid initiatives worldwide.

90


91

b) the SGIP proposes use of t
he ITCA model to ensure successful
92

implementation of the IPRM to "increase the quality of standards
-
based,
93

secure, and interoperable products" ... "resulting in increased end
-
user
94

customer satisfaction"

95


96

c) since ESPI is tightly coupled to SGIP if stands t
o reason that following
97

the primary interoperability and conformance approach developed by the
98

SGIP SGTCC gives additional credence, visibility, and additional support
99

to using the ITCA model approach

100


101

d) if an ITCA for ESPI is created and it implements th
e IPRM there is
102

additional marketing value through the SGIP

103


104

[Marty] Yes. This should be part of the UCAIug ITCA and a subproject of
105

it.

106


107


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

12

UCA International
Users Group

February
2
4, 2012


Steve: seems like we have consensus emerging on this

108

Anne: seems like charter of UCAIug reads like the goals of an ITC
A

109

James: thinks that decision about who is the ITCA is independent of the
110

need to create one. Have concerns about whether there are resources
111

people want to put into it. What we know about the technology needs
112

suggests that an ITCA is desirable.

113


114

Consensus
:
Yes, there should be a formal ITCA.

115


116

o

Whether or not to set up formal membership and sponsorships in a legal
117

structure for the standard



[Should there be a distinct legal ITCA
118

organization that can accept membership fees and sponsorship
119

contributions?


Should supporters of OpenADE/ESPI ITCA be required to
120

(able to) be a member of a distinct organization?


Could this be
121

structured within
UCA?


How?]


122


123

[[[FW: no; would lead to an endless proliferation of such]]]

124



125

AH: I don't understand the need for this.


Too early to constrain
126

participation at this point and define such a formal structure for something
127

that is still evolving.


This could
be an organization that will eventually
128

evolve, more rightly, from industry, similar to OpenADR.

129


130

Marty: No. I think it should be structured within UCA. Any per
-
group
131

differences in cost should be realized through the testing fee structures.
132

The Smart Grid

is spread very thin. We need as few different organs as
133

possible to minimize complexity and spread overhead. UCAIug can
134

provide a mechanism for specific sponsorships for scopes of work needed
135

by the sub
-
groups.

136


137

James: Has to have the formal structure. Wh
ether this is part of UCAIug
138

or a parallel structure can be determined.

139

Steve: Should it be an autonomous task group that works under the
140

umbrella? Would it be possible for the OpenADE membership to be where
141

the decisions about the certification program ar
e made?

142

Marty: I think the answer to Steve’s question is yes.

143

Anne: Doesn’t think overly constraining things at an early stage is
144

optimal.

145

o

Whether or not to raise funding to contract with a manager



[Do we need
146

funding to contract with a manager (UCA or o
ther) to develop and
147

operate an ITCA?


How do we raise that funding?


How do we decide
148

how much to raise?]


149


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

13

UCA International
Users Group

February
2
4, 2012



150

[[[FW: if you do this, do it right; estimate the fully loaded cost up front
151

and do not try to skimp]]]

152



153

AH: I don't have the understanding of the

UCAI (I can't find a "UCA")
154

and it's subgroups to know how that relationship would be established, but
155

it seems a little odd that a User's Group would contract with a parent
156

organization to manage a program the User's Group defines ... need more
157

info.


Ge
nerally, I think I'd answer 'no' until the relationships and scope
158

are more clear.

159


160

[Marty] No. I think there is adequate voluntary participation in UCAIug. I
161

do believe that the development of artifacts can benefit from specific
162

funding. The best approach

is to determine a specific scope of work,
163

estimate its value, and solicit funding to award it.

164

o

Whether or not to develop all of the artifacts with volunteer labor or raise
165

funding to contract with appropriate vendors

[An alternative to 3 is to
166

develop and

operate the ITCA with all volunteer labor (donated by
167

companies).


Does this seem preferable and feasible?


Elaborate]


168


169


[[[FW: I don't think this would work, as two unpredictable self
-
selected
170

populations would emerge: those who were highly motivated a
nd those
171

who were not
--

either one makes hash of accountability]]]

172


173

AH: Do not think developing/managing and ITCA with 'volunteer' labor is
174

feasible, nor desirable.


I can see an approach where the OpenADE TF,
175

under guidance/support from overall UCAI mana
gement/board/etc creates
176

a project proposal which includes funding of such project by UCAI
177

overall, primarily assuming that members who would derive benefit would
178

be willing to contribute.


Just a thought.


How does UCAIug usually raise
179

funds for the activ
ities listed on http://www.ucaiug.org/default.aspx?

180


181

[Marty] I think it is reasonable to attempt to complement volunteer efforts
182

with funding. I believe that contributions of funding and of personnel
183

resources is best done at the task level or project leve
l according to my
184

recommendation at question 3). Those that will financially benefit from a
185

quality certification mark for ESPI and Green Button will be willing to
186

contribute to it if they believe the investment will produce the anticipated
187

value. One majo
r standards organization I am familiar with (I have in the
188

past been a grantee) operates a development fund that is used by technical
189

committees to perform heads down comprehensive work in support of the
190

organization. Whether UCAIug implements a “pool” app
roach to such a
191


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

14

UCA International
Users Group

February
2
4, 2012


fund, or, specific project solicitations are important details that can be
192

worked out.

193

o

Whether or not to operate a formal
third
-
party certification program or use
194

a self
-
certification program

[Certification could be by a 3rd party lab or
195

us
ing a self
-
certification program.


Which would be best for
196

OpenADE/ESPI?


Why?]


197


198

[[[FW: as suggested above, primary reliance should be placed upon
199

attestation through use of a certified reference implementation.


The
200

question then becomes: would participa
nts be more comfortable showing
201

their code to a neutral third party than to a body that, in principle, may
202

include their competitors?]]]

203


204

AH: 3rd party.


Becoming a certification body that complies with
205

ISO/IED 65 is a non
-
trivial undertaking.


Is this suggesting that UCAIug
206

operate an ITCA, and the certification body? If so I think that is more than
207

should be considered at this point given

what I seem to see as the
208

resources (acknowledging there is a LOT I don't know about UCAIug
209

and/or OpenADE :)).


A second concern is that it raises the issue of
210

conflict of interest, and so, to follow the last part of IPRM 4.3 para 1,
211

would require additi
onal bureaucracy to ensure all considerations were
212

made without influence.


Never fun.

213


214

[Marty] I think we will want both. Self
-
certification by tools available for
215

purchase or not are beneficial to allow developers to achieve a successful
216

result. Accredit
ed lab results will provide the ITCA with confidence that
217

the appropriate testing has been performed. In the end the value of a
218

service mark is based on the actual interoperability achieved.

219

o

Whether or not to contract with an independent test lab for certi
fications
220

(depends on above)



[If we contract with a 3rd party lab, how many
221

should be engage to do certifications?


How do we go about doing this?]


222


223


[[[FW: certainly exactly one, else there would be lab
-
shopping]]]

224



225

AH: 3rd party.


Same reason as abo
ve but w.r.t. ISO/IEC 17025.

226


227

[Marty] It is a good idea to contract with an independent test lab that has
228

proven an implementation based on an open
-
source “golden unit”
229

reference. It is likely that one selection is best, at least initially, to provide
230

enou
gh financial incentive for a quality process.

231


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

15

UCA International
Users Group

February
2
4, 2012


o

Whether or not to take on the marketing aspects of promoting the standard
232

and certified vendor products
[Should the ITCA also promote the standard
233

and certified products?


This involves a web site, trade show
booths,
234

presentations, marketing literature, etc.


Why or why not?]


235


236


[[[FW: NO, just not our job at all]]]

237


238

AH: Yes.


Same reason as per question (1), answer (a).


Those activities
239

fall within the UCAIug mission (http://www.ucaiug.org/default.aspx) and
240

a
lso within the responsibilities of an ITCA (see #2 under 'other questions'
241

below).

242


243

[Marty] I think that marketing is an appropriate function for the UCAIug.
244

However, this should be completely independent of the testing and
245

certification and standards
-
supp
ort activities. The reason for this is that
246

marketing, while important, tends to suck the air out of any related efforts;
247

determining arbitrary timetables and otherwise compromising on technical
248

achievement. This approach has been shown to kill an otherwis
e useful
249

standard that it thereby prevented from reaching a sufficient actual
250

interoperable result. An example was the CEBus (Consumer Electronics
251

Bus) effort in the 80s and early 90s.

252

o

What
are the

organization
al requirements that

would be needed to
253

implem
ent the key decisions

254


255

AH: What I think is even more important, though, at the moment, is to get
256

some clarity on what parts of the UCAIug/OpenSG organization are
257

required to MAKE the key decisions to enable (provide support to) the
258

activities implied by qu
estions 1 and 7.

259


260

[Marty] UCAIug has structured an organization based on a top level
261

testing structure with subprojects. These organs should be fleshed out to
262

provide the capabilities needed.

263

o

How much funding would be required and how best to pursue it

264



Ba
sed on the key decisions above, develop a formal or informal request for
265

proposal to soli
cit proposals for operating an O
penADE/ESPI ITCA as envisioned
266

by the OpenADE
Task Force
.

267



Based on the key decisions, conduct a specific discussion with UCA
International
268

to understand the tasks that it can take on and the funding requirements to do so.

269

Does UCA International have (or can it develop) the needed organizational
270

structure to accomplish the goals of the OpenADE
Task Force
?

271


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

16

UCA International
Users Group

February
2
4, 2012




Determine if the UCA can

meet the requirements of the OpenADE
Task Force
,
272

including the schedule, funding and marketing activities (if any), RFP and
273

contracting activities (if any), etc.

274



Depending on the above determination, solicit additional responses to the
275

OpenADE/ESPI ITCA R
FP.

276

Interoperability Test and Certification Management Assistance Analysis

Version


0.
70

17

UCA International Users Group

February 14, 2012


4

Schedule

277


278

Task

Completion Date

To Be Determined





Interoperability Test and Certification Management Assistance Analysis

Version


0.
70

18

UCA International Users Group

February 14, 2012


5

Summary

279

In summary, this initial
requirements document

has the potential to greatly accelerate the
280

schedule and quality of both the
OpenADE

test and certification program and the overall
281

OpenADE

industry interoperability.

282


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

19

UCA International
Users Group

February
2
4, 2012


Appendix A

H
igh
-
level overview of
283

the key requirements

284

The IPRM defines the structure and functions of an
organization that takes industry
285

responsibility for a test and certification p
rogram for a standard. The organization is
286

identified as an ITCA, or Interoperability Test and Certification Authority, and is
287

characterized by its authority and competence to design and implement a program.

288

1.

Section 5.1:
General Test Policies

provides a hi
gh
-
level overview of key policies
289

an ITCA should consider and adopt.

These include:

290

a.

The level and types of information provided to vendors for certifications

291

b.

The information that should be included in the final test report

292

c.

Any conditions or expiration
limits placed on certifications

293

d.

Conformance versus interoperability certifications.

Conformance does not
294

necessarily imply interoperability between products
.

295

e.

Trade
-
off between certification testing and economic/business
296

considerations with attention to saf
ety and
Cybersecurity

related issues

297

f.

Establishing policies to
ensure

adequacy of test tools used in certifications

298

2.

Section 5.2 details the requirements for a
Test Suite Specification

(TSS) and
299

includes details.

Section 5.3 is related and deals with
attributes of a Test Profile
300

in lieu of a complete TSS
.

A TSS consists of a suite of tests, categorized into
301

logical functional areas, such as use cases or well
-
defined features.

Each test suite
302

consists of many related test cases corresponding to a partic
ular feature set or use
303

case.

A test profile evaluates a subset of a TSS and is used to target specific areas
304

of product interoperability.

305

a.

A common Test Suite Specification (TSS)
3

shall be established when
306

multiple test labs are deployed to test the same s
tandard and/or profile. If
307

common unique test procedures are required to support this test suite, then
308

they shall also be defined. The TSS should be test tool agnostic. Test Suite
309

Specifications (TSS) used for interoperability or conformance testing shall
310




3

A Test Suite Specification (TSS) consists of a suite of tests, categorized into logical functional areas, such
as use cases or well
-
defined features.

Each test suite consists of many related test
cases corresponding to a
particular feature set or use case. Test cases would include both valid and invalid behavior tests. Each test
case is further described step
-
by
-
step with test procedures and well defined pass / fail / indeterminate
criteria, along
with references.


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

20

UCA International
Users Group

February
2
4, 2012


be managed in a well
-
defined, open and formal manner with change
311

control.

312

b.

The TSS shall be subject to revision control, including revision history,
313

revision numbering, and a defect and expansion management process. The
314

TSS should clearly identify the test

purpose, references, resource
315

requirements, test setup, procedures, observable results and possible
316

problems / lessons learned with the test approach. Observables should
317

clearly identify pass / fail / indeterminate requirements and informational
318

elements.


319

c.

A Test profile
, which

MUST be a subset of a TSS, specifies all mandatory
320

and optional elements and restrictions of the standard specification and is
321

treated as a companion to the technical standard, including submission to
322

an SSO for formal standardizati
on.

323

3.

Section 5.4 enumerates the
Technical Design

of Test and Certification Programs.

324

This section covers areas of general technical, inheritance, version control,
325

general
testing
,

conformance, testing
,

interoperability, performance, tools and test
326

lead.

There are 35 specific technical requirements for an ITCA. These are
327

summarized below:

328

a.

The ITCA
MUST

specify in the test program requirements those features
329

that are mandatory, and those features that are optional.

The ITCA shall
330

require and enforce tha
t vendors declare the optional features implemented
331

in a product.
(Tech
-
1 and
-
2)

332

b.

The ITCA
MUST

require that implementations of optional features be
333

tested and certified for conformance and interoperability. Furthermore, the
334

ITCA shall define common test c
ases for that optional feature to be used
335

by all test labs when testing for that optional feature.
(Tech
-
3)

336

c.

An ITCA MUST
define the record handling and retention requirements to
337

be followed by the TL and CB functions, consistent with requirements of
338

ISO 17
025 and ISO Guide 65.

(Techn
-
4)

339

d.

The

ITCA
SHALL

specify conditions under which the use of components
340

that have been certified by other programs can be used in products to be
341

certified by the ITCA program.

Basically, OpenADE could decide that
342

specific compon
ents or classes of components are “pre
-
certified” but the
343

ITCA remains responsible for
e
nsuring conformance and interoperability
344

of the OpenADE
Task Force

certified products.

The ITCA SHALL
345

implement a Compliant Portion Description (CPD)
4

to be used as a g
uide
346




4

See Glossary of Terms
in the IPRM
for definition and further explanation of CPD


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

21

UCA International
Users Group

February
2
4, 2012


for assembling a product based on compatible sub
-
components.

(Tech
-
5
347

to Tech
-
8)

348

e.

Another section deals with differing versions of certified products and
349

how the certification program should handle them.

ITCAs need to manage
350

re
-
certifications and the i
dentification of current versions that are certified.

351

(Tech
-
9 to Tech
-
12)

352

f.

There shall be a defined correlation between implementations and required
353

testing, commonly called a Proforma Implementation Conformance
354

Statement (PICS).

(Tech
-
14)

355

g.

The testing and

certification program shall maintain a current and
356

upcoming list of applicable test cases to be called a Test Case Reference
357

List.

These test cases should be defined in an open, consensus
-
driven
358

fashion. These test cases will be used by all test labs appr
oved by the
359

ITCA. There shall be a Test Plan derived from the Test Case Reference
360

List and used by all authorized test labs. Tests shall be identified using the
361

test plan.

(Tech
-
13, Tech
-
15
-
16

362

h.

The testing and certification program SHALL require that a sta
tic
363

conformance review
5

take place prior to testing a product.

(Tech
-
17)

364

i.

The testing and certification program shall first validate the tests, and
365

implement them utilizing validated test tools. Golden reference test
366

equipment may be utilized where appropr
iate.

(Tech
-
18)

367

j.

The testing and certification program shall assure that defined product test
368

cases cover application profiles for specific feature sets and functions
369

defined by the specific application profile, and implement interoperability
370

evaluation wi
thin that application profile.

Where practical, the testing and
371

certification program shall assure that defined product test cases cover all
372

feature sets and functions.

(Tech
-
20 to Tech
-
22)

373

k.

The ITCA SHALL classify common or major market products according

374

to their application profiles and conduct certification tests based on those
375

applications.

The testing and certification program SHALL assure that
376

defined product use cases are covered in application profiles.

377

Interoperability testing and evaluation SHALL

be implemented within
378

those application profiles.

(Tech
-
23)

379




5

A review of designed feature sets versus the specified PICS to determine the extent to which the features
are supported by the IUT. This is the first step

when a product enters a testing program. Generally the test
lab requests that the implementer declare all supported feature sets in a product. This information is used to
create the test plan for that product.


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

22

UCA International
Users Group

February
2
4, 2012


l.

A section deals with interoperability testing. The section deals with what
380

are commonly called “plugfests” but also addresses the selection and use
381

of “golden” reference units.

Plugfests are opti
onal for application interface
382

standards.

(Tech
-
24 and Tech
-
25)

383

m.

ITCAs SHALL use reference test tools
6

where appropriate to the
384

technology under test (hardware and/or software) to provide a consistent
385

and replicable approach in generating test results across ITCA test labs.
386

Successful testing programs assure that there is a known reference or
387

constant
,

to
which the system is evaluated against the desired metrics to
388

determine conformance. ITCA program tests that are performed across
389

multiple test facilities SHALL implement processes to assure they are
390

each measuring against a common known reference and achie
ving
391

repeatable results regardless of location.

(Tech
-
26 and Tech
-
27)

392

n.

When used, a minimum of two golden units are to be selected

by a defined
393

process (Tech
-
28 to Tech
-
30)

394

o.

If an ITCA Certification Program involves multiple Smart Grid systems,
395

then the Pr
ogram Requirements SHALL support end
-
to
-
end testing of
396

Smart Grid systems involving multiple product implementations to the
397

fullest extent possible. An ITCA SHALL involve all relevant parties to
398

define various business logic models for the end
-
to
-
end syste
m testing,
399

and make scenarios and test harness systems available for testing.

(Tech
-
400

30
-
31)

401

p.

The testing and certification program shall ensure that when functional
402

performance requirements are defined in an application profile, the
403

performance test profile
(s) shall be designed to implement test cases for
404

evaluating these requirements. (Tech
-
3
2
)

405

q.


The ITCA SHALL ensure that test tools have a complete mandatory
406

feature
-
set coverage of a standard. In cases where two or more
407

implementations of optional features
are available, the ITCA shall
408

incorporate those feature
-
sets in the test tool. The ITCA shall define
409

procedures and processes to validate the use of test tools and reference
410

implementations.
(Tech
-
33 to Tech
-
34)

411

r.

An ITCA shall develop criteria for surveilla
nce to insure that certified
412

products and vendors fulfill the certification agreement(s). The
413

surveillance is to be carried out by its certification body(ies). (Tech
-
35)

414




6

A number of terms are used in describing re
ference test tools such as “common test harness”, “golden
reference test equipment”, and “golden reference test products”. Generally, these each represent test tools
available to a test lab or end user to provide a consistent baseline test either as a stan
dalone implementation
or in concert with the many other types of test tools available.


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

23

UCA International
Users Group

February
2
4, 2012


4.

Section 5.5 addresses
Program and Field Experience Feedback

and points to the
415

import
ance of end
-
user and product vendor experience with the standard and the
416

certification program.

417

5.

Section 6 of Draft
4
, Version 2 of the IPRM is new and addresses
Cybersecurity

418

Testing
.

ITCAs are responsible for coordinating and overseeing the
Cybersecurity

419

criteria as applicable to the testing and certification programs that they operate.

420

Section 6.7 includes specific requirements of an ITCA for
Cybersecurity

testing
421

but these are still in discussion in terms of the ITCA responsibility in this area.
422

This sec
tion includes proposed requirements that:

423

a.

The ITCA SHALL define the procedures and processes
that

will be used
424

to validate interoperability
Cybersecurity

requirements.

Such tests are
425

usually initiated by each vendor, to verify that a vendor product meets
an
426

industry established level of security, and tested by independent third
-
427

party labs using the same testing procedures that are pass/fail in nature
.

428

(Sec
-
1)

429

b.

The testing and certification program shall ensure that
Cybersecurity

430

functional performance requi
rements are defined, and test cases designed
431

to evaluate the requirements. Further, ITCAs are responsible to qualify
432

testing personnel for
Cybersecurity

training and experience
.

(Sec
-
2 and
433

Sec
-
5)

434

c.

If applicable, ITCAs are responsible for Digital Certificate

programs.
435

(Sec
-
3 and Sec 4)

436

d.

ITCAs are responsible for requiring widely
-
accepted security stress
437

testing
,

including static analysis and penetration testing; assuring security
438

policy models drive testing; ensuring that vendors submit threat analyses
439

as part

of certification process; document
ing

programs and standards used
440

for security testing
;

and incorporating component
-
based
Cybersecurity

441

concepts in the testing program.

442

One interpretation of this proposed section is that an ITCA needs to establish a
443

distinct certification program just for
Cybersecurity

issues and include this testing
444

in the certification program.

The IPRM is not yet clear on the qualifications for a
445

lab t
o do
Cybersecurity

testing.


446

6.

Section 7.2, titled
Governance
,

provides a structural prescription for
an

ITCA.
447

The OpenADE
Task Force

intends to establish the ITCA for the OpenADE/ESPI
448

and related standards
,

although one purpose of this document is to assist

in
449

determining what organization will actually operate the ITCA for
450

OpenADE/ESPI.

Key governance requirements are:

451


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

24

UCA International
Users Group

February
2
4, 2012


a.

The ITCA defines and documents the interoperability test program for the
452

standard and oversees its implementation, including roles
, r
esponsi
bilities
453

and resources.

The ITCA SHALL provide oversight to provide confidence
454

that implementations of Standard(s) in certified products are indeed
455

interoperable.

This will be the primary task of the OpenADE
Task Force
.
456

(Gov
-
1 and Gov
-
5 in D3Rev
4
)

457

b.

The ITCA

determines whether
first
-
party testing,
third
-
part
y

testing or
458

both
are

allowed and defines the circumstances and process for submission
459

to a certification body (CB) as well as the CB responsibilities
.

(Gov
-
2 and
460

Gov
-
3 in D3Rev
4
)

461

c.

The ITCA SHALL define a c
orrective process for resolving reported
462

interoperability problems (e.g.
,

in the field or as part of the test) for
463

products for which they are responsible.
7

Further, it SHALL implement
464

preventive processes to avoid recurrence of such problems.

A problem
465

ma
y be associated with the specification, the test processes and procedures
466

or the test data. (Gov
-
4 in D3Rev
4
)

467

d.

A key function is to ensure that issues that arise through the certification
468

test process are fed back to appropriate parties for clarification or

469

inclusion in subsequent versions of the standard.

(Gov
-
6 in D3Rev
4
)

470

e.

The ITCA SHALL maintain a publicly available certified product and
471

systems list. (Gov 7 in D3Rev
4
)

472

f.

The ITCA shall maintain a test case reference and modification history
473

list
8
.

This is a current
m
aster list of all tests that are to be included in a
474

product certification test plan.

This helps a product implementer in
475

preparing fully conforming and interoperable products for an upcoming
476

certification and launch. (Gov
-
8 in D3Rev
4
)

477

g.

A

common TSS SHALL be established when multiple test labs are
478

deployed to test the same standard and/or profile.

If common unique test
479

procedures are required to support this test suite, then they SHALL also be
480

defined.

The TSS should be test tool agnostic.

Test Suite Specifications
481

(TSS)
9

used for interoperability or conformance testing SHALL be
482

managed in a well
-
defined, open and formal manner with change control.

483

(Gov 9 and Gov 10 in D3Rev
4
)

484

h.

The ITCA shall minimize divergence of interoperability
requirements
485

interpretations.

If an ITCA has multiple testing laboratories and certifying
486




7

The ITCA should use best efforts in contacting a standards body with respect to a specification; however,
it is not their responsibility to resolve issues with the spe
cification.

8

See Glossary of Terms in the IPRM for definition and explanation of the test case reference list.

9

See Glossary of Terms in the IPRM for definition and explanation of the TSS.


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

25

UCA International
Users Group

February
2
4, 2012


bodies, processes shall be in place to avoid quality differences and assure
487

repeatable testing between the laboratories. One way to minimize
488

divergence of interpretat
ions is to limit the number of labs to only one.
489

Another option for minimizing divergence is to have a technical lead (also
490

known as a lead lab) responsible for properly interpreting conformance
491

and interoperability issues.

(Gov
-
12 in D3Rev
4
)

492

i.

Any organiza
tion that certifies the actual test labs and processes needs to
493

conform to ISO/IEC 65 Guidelines and SHALL include the International
494

Classification for Standards (ICS) Codes applicable to the technologies for
495

which certification activities are performed.

F
urther, Accreditation
496

SHALL be by an accreditation body that is signatory, in good standing, to
497

the International Accreditation Forum (IAF) multilateral agreement for
498

“Product.”

This set of guidelines generally specifies the formal
499

documentation and proces
ses required of such a body as well as criteria
500

for eliminating or minimizing potential conflict of interest issues.

A brief
501

discussion of these Guidelines is contained in the IPRM Annex on page
502

66.

(Gov
-
11 in D3Rev
4
)

503

j.

The IPRM
REQUIRES

that product certifi
cation be issued by an ISO/IEC
504

65 accredited third party independent of the testing organization
.

(
Section
505

1.4: Intended Audience in Draft
4

of Version 2
)

506

k.

The proposed IPRM Version 2 (Draft
4
)
includes

an additional
507

REQUIREMENT over and above ISO/IEC Guide 65


the independent
508

trusted
third
-
party certification authority MUST only allow the statement
509

that products are interoperable only if the products actually demonstrated
510

interoperability during testin
g. They MUST not assume interoperability
511

from conformance testing. They MUST demonstrate it before it may be
512

part of the products certification statement. (
Section 2.2: Overview of
513

ISO/IEC Guide 65, Draft
4

of Version 2
)

514

7.

Section 7.3 is titled
Lab Qualifica
tion

and basically requires that lab selection
515

SHALL be done in a uniform and transparent procedure and that any labs
516

performing certification of products for the ITCA be ISO 17025 certified.

Further,
517

the ITCA SHALL define requirements to qualify the perso
nnel involved in the
518

certification and testing process for its standard.

A discussion of ISO 17025 is
519

contained in the IPRM Annex.

In summary, ISO 17025
focuses on two major
520

areas of laboratory operations: 1) management requirements
;

and 2) technical
521

requi
rements. The management requirements address issues such as a lab’s
522

documented practices (i.e.
,

both administrative and technical), impartiality of the
523

lab in its operations, responsibilities for continuous improvement and issues
524

resolution, and the active

support and involvement of lab management in assuring
525

commitment to complying

with these criteria. The technical requirements focus
526

on areas such as ensuring that lab staff are competent in performing their testing
527


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

26

UCA International
Users Group

February
2
4, 2012


duties, assuring that the lab environmen
t is adequate for services performed,
528

assuring that test plans and other necessary operating instructions are documented
529

and available, and that necessary equipment and software used for testing is
530

calibrated, maintained and appropriate for its intended us
age.

531

8.

Section 4 deals with
improvements

to the overall process, the standard
532

documentation itself, test labs, the test and certification program, etc.

Reference is
533

made to the preference that an ITCA solicit direct feedback from customers of the
534

certified
products to assess that they meet customer interoperability needs.

535

The IPRM includes additional recommendations for best practices for interoperability
536

and conformance testing certification programs.

Much of the material is covered in the
537

requirements them
selves but there are a number of useful recommendations that the
538

OpenADE
Task Force

could utilize in designing their own programs.


539


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

27

UCA International
Users Group

February
2
4, 2012


Appendix B

ITCA Test and
540

Certification Tasks Based on
541

the IPRM

542

According to the IPRM proposed Version 2,
D
raft
4
:

543

An Interoperability T
esting and Certification Authority (ITCA) is the program
544

management organization, providing oversight for testing and certification
545

activities associated with one or more standards or specifications, that takes
546

responsibility to insure that interoperable p
roducts within the scope of the
547

specific ITCA program are brought to market. The ITCA coordinates the
548

participation of certification bodies and test labs for its program.

549

The following are the tasks that OpenADE/ESPI and/or UCA International will need to
550

c
omplete as part of its ITCA goals under the IPRM Version 1 and proposed Version 2.


551

5.1.2

Organize the ITCA (Implied Tasks)

552



Develop a business plan for the ITCA

553

o

Determine the level of conformance to the IPRM that the ITCA will aspire
554

to implementing, including
independent ISO/IEC 65 and ISO 17025
555

certification requirements

556

o

Establish charter, scope and legal framework, including legal documents
557

as needed

558

o

Establish Governance policies and procedures

559

o

Determine and organize ITCA budget and Treasury functions

560

o

Determ
ine staff support requirements and resources available

561

o

Determine overall budget and schedule for a prudent period of time (3
-
5
562

years)

563



IPR Policy development and implementation

564



Recruit members who can contribute both financial and staff resources

565



Set sponso
rship and dues levels or acquire other sources of funding

566



Select leadership

567



Establish meeting and working protocols

568


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

28

UCA International
Users Group

February
2
4, 2012


5.1.3

Manage and Promote the Standard (Implied
T
asks)

569



Recruit vendors and customers to adopt and use the OpenADE/ESPI standard

570



Continue to support

development of the standard (working with the appropriate
571

SSO
10
)

572



Conduct conferences, meetings, trade show exhibits and plugfests

573



Represent the interest of the ITCA members at appropriate events and
574

organizations to promote the standard

575



Develop and
maintain a web site for the standard as part of promotion, for listing
576

certified products and for acknowledging sponsors, members and contributors

577



Put in place intellectual property protections


copyrights, trademarks, etc.

578

5.1.4

Organize the Certification Prog
ram

579



Organize the ITCA certifications and determine what organization will act as the
580

Certification Body
.

A
ctual certifications must be issued by independent
third

581

parties which are not the test labs and are accredited to ISO/IEC 65 Guidelines.

582

The ITCA cou
ld become an ISO/IEC 65 certified body and issue certifications of
583

conformance and interoperability itself or contract with such an organization to do
584

so
.

585

o

Certification bodies (CBs) should be accredited to ISO Guide 65,

General
586

Requirements for Bodies
Operating Product Certification Systems


587

o

Test laboratories should be accredited to ISO 17025,
General
588

Requirements for the Competence of Testing and Calibration
589

Laboratories

590

o

The ITCA should have an agreement with an accrediting organization(s)
591

to assure th
at Certification Body and Test Lab accreditation is being
592

performed in accordance with the ITCA program scheme.

593



An ITCA should have a strong relationship with the SSO associated with the
594

standard for the purpose of feedback towards standard improvement and

595

clarification where there may be ambiguities

596

5.1.5

Define Certification Program (Explicit Tasks)

597



Define the certification program for OpenADE/ESPI products

598




10

SSO = Standards Setting Organization. This may be an international standards organization such as ISO,
IEC, OASIS, IEEE, etc., or in some cases it may be a trade association such as the ZigBee Alliance.


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

29

UCA International
Users Group

February
2
4, 2012




The IPRM
REQUIRES

that product certification be issued by an ISO/IEC 65
599

accredited third party independen
t of the testing organization

600



The proposed IPRM Version 2 (Draft 3)
includes

an additional REQUIREMENT
601

over and above ISO/IEC Guide 65


the independent trusted
third
-
party
602

certification authority MUST only allow the statement that products are
603

interoperab
le only if the products actually demonstrated interoperability during
604

testing. They MUST not assume interoperability from conformance testing. They
605

MUST demonstrate it before it may be part of the products certification statement.

606



Establish a detailed work

program with schedule and resources

607



Determine the business model(s) to be used

608

o

Volunteer labor for all activities

609

o

Combination volunteer and funded staff: which activities for which

610

o

Contractors and/or vendors and labs to implement aspects of the ITCA
611

manag
ement and programs

612



Agree on one or more Proforma Implementation Conformance Statement (PICS)
613

or profiles that represent the most likely use cases for products based on the
614

standard

615



Develop the high
-
level certification test specification based on the PICS

616



D
etermine how detailed test cases, scripts and test harness(es) will be developed
617

and maintained

and who will develop and maintain them

618



Develop a program overview and applicant preparation guide

619



Define defect tracking and issue tracking requirements for
both the technical
620

program and tools and the business func
ti
ons

621

5.1.6

Establish Vendor Partnerships (Implied Tasks)

622



Determine philosophy of ITCA management (volunteer or professional)

623

o

Develop RFP if professional management is determined

624

o

Identify potential ITCA
managers and solicit proposals in response to the
625

RFP

626

o

Select and develop contract with manager

627

o

Manage contract manager activities and set policy

628



Determine if one or more test labs will be required and whether or not the
629

structure needs to be developed to a
dd test labs in the future

630


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

30

UCA International
Users Group

February
2
4, 2012


o

Develop RFP for test lab(s)

631

o

Identify potential test labs and solicit proposals in response to the RFP

632

o

Select and develop contract with test lab

633

o

Manage contract activities with test lab

634

o

Follow guidelines in ISO Guide 65 to

ensure

that Labs are ISO 17025
635

accredited and maintain accreditation


636



Determine if a separate test tool vendor will be required and whether or not the
637

structure needs to be developed to add vendors in the future

638

o

Develop RFP for test tool vendor

639

o

Identify potentia
l vendors and solicit proposals in response to the RFP

640

o

Select and develop contract with test tool vendor

641

o

Manage contract activities with test tool vendor

642



Determine if a separate marketing communications vendor will be required

643

o

Develop RFP for marketing com
munications vendor

644

o

Identify potential vendors and solicit proposals in response to the RFP

645

o

Select and develop contract with marketing communications vendor

646

o

Manage contract activities with marketing communications vendor

647

5.1.7

Implement Certification Program
(Explicit Tasks)

648



Define the General Test Policies for the certification program for OpenADE/ESPI
649

products (Section 5.1 of the IPRM D3V2)

650



Establish test report template, contents and example

651



Develop and maintain a test case reference and modification histor
y list

652



Establish a Common Test Suite Specification (TSS
11



Section 5.2/3 of the IPRM
653

D3V2) if multiple test labs are deployed to test the same standard and/or profile

654



If required, define common unique test procedures to support the TSS


t
hese
655

should be te
st tool agnostic

656




11

A Test Suite Specification (TSS) consists of a sui
te of tests, categorized into logical functional areas,
such as use cases or well
-
defined features.

Each test suite consists of many related test cases corresponding
to a particular feature set or use case. Test cases would include both valid and invalid b
ehavior tests. Each
test case is further described step
-
by
-
step with test procedures and well defined pass / fail / indeterminate
criteria, along with references.


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

31

UCA International
Users Group

February
2
4, 2012




Manage the TSS in a well
-
defined, open and formal manner with change control

657



If there are multiple testing laboratories, put in place processes to avoid quality
658

differences and assure repeatable testing between the laboratories

659



Specify i
n the test program requirements
for
those features that are mandatory,
660

and those features that are optional (Section 5.4 of the IPRM D3V2)

661

o

Require and enforce that vendors declare the optional features
662

implemented in a product

663

o

If more than one vendor imple
ments the same optional feature in a
664

product, require that future implementations of that optional feature be
665

tested and certified for conformance and interoperability

666

o

Define common test cases for optional features that need to be tested as
667

part of the cer
tification program

668



Establish certification programs, terms and conditions of award and re
-
669

certification

670

o

Maintain a published list of certified products

671

o

If a logo is part of the program, create the logo and licensing agreements

672



Determine which components, i
f any, certified by other industry programs can be
673

“inherited” in product certifications

674

o

Develop the procedures for validating that pre
-
certified components
675

included in OpenADE/ESPI products do not impact interoperability and
676

conformance to the OpenADE/ESP
I specification

677



Develop common, well
-
defined standardized test cases in an open, consensus
-
678

driven fashion.

These test cases will be used by all test labs approved by the ITCA

679

o

Validate the tests and implement them utilizing validated test tools.
680

Golden ref
erence test equipment may be utilized where appropriate.
681

Define procedures and processes to validate the use of test tools and
682

reference implementations
.

683

o

Ensure that test tools have a complete mandatory feature
-
set coverage of a
684

standard.

In cases where op
tional features are included in vendor products,
685

incorporate those feature
-
sets in the test tool
.

686



Maintain a current and upcoming list of applicable test cases to be called a Test
687

Case Reference List


688

o

Work with authorized labs to derive a Test Plan from th
e Test Case
689

Reference List.

Tests shall be identified using the test plan

690


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

32

UCA International
Users Group

February
2
4, 2012




Establish and maintain a revision control system, including revision history,
691

revision numbering, and a defect and expansion management process for all tests
692

in the TSS

693



Assure that
defined product test cases cover application profiles for specific
694

feature sets and functions defined by the specific application profile, and
695

implement interoperability evaluation within that application profile

696



Define interoperability
-
specific testing pr
ocedures such as “plugfests” but also the
697

selection and use of “golden” reference units.

A minimum of two golden units are
698

to be selected
.

699

5.1.8

Improvements in the Standard and the Certification Program
700

(Explicit Tasks)

701



Develop and maintain an improvement progr
am for the overall process, the
702

standard documentation itself, test labs, the test and certification program, etc
.

703

(Section 5.5 in the IPRM D3V2)

If possible solicit direct feedback from
704

customers of the certified products to assess that they meet customer

705

interoperability needs
.

706

5.1.9

Cyber
-
Security (Explicit Tasks)

707



The testing and certification program shall ensure that
Cybersecurity

functional
708

performance requirements are defined, and test cases designed and used to
709

evaluate the requirements

710



The ITCA needs to
work with NIST and the SGIP to

ensure

that the technical
711

specification for OpenADE/ESPI standards is reviewed for cyber
-
security issues

712



The ITCA may need to establish a Digital Certificate Program if applicable

713



Determine and implement appropriate security

stress testing including static
714

analysis and penetration testing; assure security policy models drive testing;
715

ensure that vendors submit threat analyses as part of certification process;
716

document programs and standards used for security testing and incor
porate
717

component
-
based
Cybersecurity

concepts in the testing program

718



One interpretation of this proposed section is that an ITCA needs to establish a
719

distinct certification program just for
Cybersecurity

issues and include this testing
720

in the certification

program.

The IPRM is not yet clear on the qualifications for a
721

lab to do
Cybersecurity

testing.

722


Interoperability Test and Certification Management Assistance Analysis

Version


0.
80

33

UCA International
Users Group

February
2
4, 2012


5.1.10

Governance (Explicit Tasks)

723



Determine whether
first

party testing,
third

part
y

testing or both
are

allowed
724

and define the circumstances and process for submission to a certification
725

body (CB) as well as the CB responsibilities

726



Define a corrective process for resolving reported interoperability problems
727

and implement preventative processes to avoid recu
rrence of such problems.

728