ETSI Update on TTCN-3 Language Evolution

hurriedtinkleΤεχνίτη Νοημοσύνη και Ρομποτική

15 Νοε 2013 (πριν από 3 χρόνια και 9 μήνες)

85 εμφανίσεις

ETSI Update on TTCN
-
3 Language Evolution
and Tool Conformance Testing

Nikolay

Pakulin


Contributors:

Ina
Schieferdecker

Jens Grabowski

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Agenda


STF 430: TTCN3 Evolution


TTCN3 Overview


Advanced TRI


Continuous Signals



STF 433: TTCN3 Tool Conformance Testing


Motivation


Methodology


Challenges


Requirements
-
driven Test Development





Slide
2

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 Evolution

TTCN
-
3 User Conference

2012,
Bangelore
,
India


Nikolay
Pakulin



on behalf
of

STF430

TTCN
-
3 User Conference

2012,
Bangelore
,
India

STF administrative data


STF 430:
April 2011


December 2011 (78 mDays)



Members:


Gyorgy Rethy
, Ericsson


Jens Grabowski, University of Goettingen


Ina Schieferdecker, Fraunhofer FOKUS


Jacob Wieland, Testing Technologies



STF 430 sessions in 2011


Session 1: 24
-
27 May


Session 2: 27 June
-

1 July


Session 3: 26
-

30 September


Session 4: 28 November
-

2 December


Slide
4

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

IDL

XSD

Language

mappings

ASN.1

Extensions

Advanced

parameteri
-

zation


Behaviour

types


Static

configuration

Documentation

t3doc

TTCN
-
3 Structuring:

Imports, Groups, Attributes

Core

language

TTCN
-
3 Technology Overview

TTCN
-
3 Behaviour

TTCN
-
3 Data

C

C++

TRI/TCI

mapping

Java

XML

C#

Real
-
time

support

Continuous

signal

support

Advanced

TRI

Slide
5

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

TTCN
-
3 language evolution


Tasks


Resolution and implementation of CRs



Prepare the drafts for ETSI publication:


Parts 1 and 4 to 10 of the ES 201 873 series



The 4 extension packages (ES 202 781, ~782, ~784, ~785)




Cooperation with STF 160 on language requirements of LTE
testing




Two new language extension packages


Advanced TRI


Continuous signal support


Slide
6

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

The version forest


Approved new versions


TTCN
-
3 Part
-
1: Core Language


V4.4.1 (44 CRs)


TTCN
-
3 Part
-
4: Operational Semantics


V4.4.1 (1 CR)


TTCN
-
3 Part
-
5: TTCN
-
3 Runtime Interface

V4.4.1 (2 CRs)


TTCN
-
3 Part
-
6: TTCN
-
3 Control Interface

V4.4.1 (8 CRs)


TTCN
-
3 Part
-
7: Using ASN.1 with TTCN
-
3

V4.4.1 (1 CR)


TTCN
-
3 Part
-
8: Using IDL with TTCN
-
3


V4.4.1 (2 CR)


TTCN
-
3 Part
-
9: Using XML with TTCN
-
3

V4.4.1 (12 CRs)


TTCN
-
3 Part
-
10: TTCN
-
3 docum. tags


V4.4.1 (1 CRs)


TTCN
-
3 Extension: Behaviour Types


V1.2.1 (3 CRs)


TTCN
-
3 Extension: Configuration and deployment V1.2.1 (1 CR)


TTCN
-
3 Extension: Real
-
Time and Performance Testing V1.2.2 (2 CRs)



Not republished:


TTCN
-
3 Extension: Advanced Parameterization

V1.3.1



New:


TTCN
-
3 Extension: Advanced TRI


V1.1.1


TTCN
-
3 Extension: Continuous signal support

V1.1.1


Slide
7

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

New features in Part 1


5417: Support of
parametrized

map/
unmap



var

MyConfigType

cfg
:={ option := 1, lock := false};


...


map(mtc:Port4, system:PCO2)
param

(
cfg
);


5262: Partially constrained structured types


5938: Type restriction by template li
st

type
MyRecord

MyRecordSub5
(



{f1 := *, f2 := "user", f3 := pattern
"
password|Password
" },



{ f1 := (1 .. 10), f2 := "User“, f3 := ? })


5937: Generalized annotation of attributes to
declarations/members

Slide
8

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

New: Advanced TRI 1(3)


Historically, TTCN has been used to test
communication protocols which typically use
encoded messages. This has been reflected in the TRI
SA and TCI CD design of TTCN
-
3 by encoding and
decoding messages to/from bitstrings.



However, TTCN
-
3 also supports signature
-
based
communication for which the transformation of
objects into bitstrings and vice versa is cumbersome.



Therefore, an alternative API is being proposed along
which TTCN
-
3 values can be directly passed to/from
the SUT.



Old and new TRI can be used in combination!

Slide
9

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

New: Advanced TRI 2(3)


Redefinition of TRI Data Interface


TriMessageType



畳u 噡汵e


TriAddressType



畳u 噡汵e


TriAddressListType


畳u 噡汵e䱩st


TriSignatureIdType


畮捨c湧敤e


TriParameterType




畮捨c湧敤e
 潴o扥楮朠畳g搩d


TriParameterPassingModeType








畮捨c湧敤e
 潴o扥楮朠畳g搩


TriParameterListType


T
c楐慲ame瑥牌楳瑔t灥p

(by that, the TciParameterPassingMode and Values for
parameters will be used)


TriExceptionType




畳u 噡汵e

Slide
10

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

New: Advanced TRI 2(3)


Changes to Message
-
based communication operations


triSend


硴x楓敮i


triSendBC

硴x楓敮i䉃


triSendMC

硴x楓敮i䵃


triEnqueueMsg

硴x楅i煵敵e䵳M


New


Value xtriConvert(in any value, in Type typeHypothesis)

This operation shall be called whenever the TE has to convert a value. The
TE might convert immediately after reception of the value, or might for
performance considerations postpone the conversion until the actual
access to the value.


Changes to Procedure
-
based communication operations


triCall

硴x楃慬i


etc.


Changes to Miscellaneous operations


triExternalFunction


硴x楅硴i牮慬䙵湣瑩nn

Slide
11

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

TTCN
-
3 Extension: Continuous signal
support 1(4)


This new package defines concepts for testing
systems using continuous signals.


The package introduces


notions of time and sampling,


notions of streams, stream ports and stream variables,

and


definitions allowing an automaton alike control flow
structure supporting the specification of hybrid
behavior (i.e., combined definition of
discrete and
continuous behaviors)
.

Slide
12

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

TTCN
-
3 Extension: Continuous signal
support 2(4)


The following items shall give an impression about
the contents of the package. Details and further
operations can be found in the package definition.



Concepts for time and sampling:


access to global time:


now
operator


Setting stepsize for sampling:

stepsize

operator




Streams


New port type for streams, e.g.,:


type port
StreamIn

stream
{

in float
}


type port
StreamOut

stream
{

out float
}


Operations for accessing data streams:


retrieving or setting actual values:


value

operation


retrieval of time related information:
timestamp

operation


Retrieval of actual stepsize of a port:
delta

operation


Slide
13

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

TTCN
-
3 Extension: Continuous signal
support 3(4)


Streams (cont.)


navigation on streams


previous event:



prev

operation


event at a certain point in time:

at

operation


stream extraction


Retrieval of (partial) history:

history

operation



Modes


define states in hybrid automata. Modes may


be composed sequential or parallel


have entry and exit conditions


have invariant parts


have a duration


be parameterisable


etc.

Slide
14

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

TTCN
-
3 Extension: Continuous signal
support 4(4)


15

Slide
15

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

A Conformance Test Suite for TTCN
-
3 Tools

Bogdan
Stanca
-
Kaposta
,
Testing

Technologies

Andras Kovacs,
Broadbit

Nikolay
Pakulin
, ISPRAS

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Motivation


Many TTCN
-
3 compilers (~7 commercial, 2 internal)


High interest in the language


Complex standard: hundreds of pages


No way to check the compliance of TTCN
-
3 tools


Quality
of

a
test

tool

is

critical



Relevance:


For users:


Is my tool standard compliant?


Can I switch tools easily / vendor lock
-
in?


For tool vendors:


Where can I improve the standard compliance of my
tool?


Possibly later: compliance certification


For standard developers:


Where is need for clarification?


Slide
17

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

STF 433


Specialist Task Force initiated by ETSI MTS


Manpower: 3 persons for 100 days total


Dates: November 2011


Early 2012


Work is based on the existing framework.


Task:


Update the existing TTCN3 ATS to the newer TTCN
-
3
revision


Extend the conformance test suite for TTCN
-
3 tools.


Three deliverables:


ETSI TS 102 950
-
1 V1.2.1 (2012
-
05)


ICS


ETSI TS 102 950
-
2 V1.2.1 (2012
-
05)


TSS & TP


ETSI TS 102 950
-
3 V1.2.1 (2012
-
05)


ATS & IXIT


Approved by MTS in May 2012.


Downloadable

from

the

ETSI
download

area

right

now
!


Slide
18

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Methodology (1/7)


Main steps:

1.
Validate existing tests against new revision of the
standard.

2.
Develop conformance tests.


Follow the priority list discussed with MTS and vendors.

3.
Develop adapters for selected tools. Validate and fix
tests.

4.
Automate test execution.

Slide
19

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Methodology (2/7)


Adjusting Conformance ATS from TTCN3 rev. 4.2.1 to
rev. 4.3.1:


Was: 733 tests, 130 pp. of test purposes


Validate that test purposes still hold


Update modified test purposes


Revoke outdated tests


Update ATS folder structure


Approximately 30% of the STF 433 effort



New tests development


Clause priority list:


Not all clauses can be covered. Selection was made:
Vendor feedback + most basic functionality.


There is no perfect selection!

Slide
20

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Methodology (3/7)


Development of conformance tests:


The ATS is not the test automation, but the test input!


In most cases: 1 module = 1 conformance test.


Test automation provided for certain vendors


Elvior

and Testing Technologies


Test suite structure: organized according to clauses.


Expected output for verdict determination:


@verdict pass accept/reject [
expectedoutput
]


Examples:


@verdict pass reject


@verdict pass accept, ttcn3verdict:pass


@verdict pass accept,
noexecution


Purpose specification:


@purpose
documentreference
, description


Documentreference
:
part:clause


Examples:


@purpose 1:5, Ensure that when the IUT loads a module
containing some definitions before the module declaration then
the module is rejected.


Slide
21

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Methodology (4/7)


Development of conformance tests (cont.):


Complete TTCN
-
3 test input example:

/***************************************************


** @purpose 1:5.2.2, Ensure that the IUT correctly


handles the uniqueness of


variable names in its scope.


** @verdict pass accept, ttcn3verdict:pass


***************************************************/

module

Sem_050202_Uniqueness_001 {


import

from

Sem_050202_Uniqueness_001_import {




const

all
;


}


type

component

GeneralComp

{
}


function

f_funcScope
() {



var

boolean

repeatedIdentifier

:=
true
;



if
(
repeatedIdentifier
==
true
) {
setverdict
(
pass
);
}


}


testcase

TC_Sem_050202_Uniqueness_001()



runs

on

GeneralComp

{
f_funcScope
(); }


control

{
execute
(TC_Sem_050202_Uniqueness_001());
}

}

module

Sem_050202_Uniqueness_001_import {


// repeated indentifier


// from imported module


// is allowed


const

integer

repeatedIdentifier := 0;

}

Slide
22

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Methodology (5/7)


Creation of documentation:


Framework already provided by STF393 (Proforma).


Little time for documentation.


Working on a single source saves time.


Solution:


Write test purposes and expected tool outputs directly
into the test inputs.


Generate tables for the documentation from the ATS:


Test purpose tables.


Test suite structure.


Much easier to maintain!


Documentation generation tools developed as part of
STF409.

Slide
23

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Methodology (6/7)


Validation and fixing:


Validation and fixing took more than half of the
development time!



STF internal validation with
2

tools:


TTworkbench
: compile
-
time and execution.


TestCast
: compile
-
time and execution.


Developed necessary adapters


Loopback, proc. call


Lots of feedback from
TTech

and
Elvior
!

Slide
24

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Methodology (7/7)


Executing the ATS:

ATS

Sem_060101_TopLevel_001.ttcn

NegSem_060303_component_types_001.ttcn



TTCN
-
3 Tool
(
SUT
)

TTCN
-
3 File

TTCN
-
3 File

TTCN
-
3 File

Sem_060101_TopLevel_001.ttcn

Tool output

Rejection as invalid

Rejection to execute

Execution with results

(e.g., pass/inconc/fail)

Tool output evaluation


** @verdict pass accept, ttcn3verdict:pass

fail

pass

Test input

Expected

output

Slide
25

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Discussion


Compliance does not imply safety from a vendor lock
-
in!


Testing is never complete!


With v1.1.1 of the ATS, about 32% of the core standard clauses
are covered with at least one test.


Ideally, we should have many tests for each clause.


Compliance means less likelihood for vendor lock
-
in.


With better coverage and higher number of test cases, we
improve the quality of this statement.


Compliance reduces the likelihood for major tool errors


Tests range from simple arithmetic operations to
communication operations and verdict handling.


ATS checks syntax and semantics.


ATS does not check the correctness of every single tool
message, but of the overall output.

TTCN
-
3 Tool

TTCN
-
3
Conformance ATS

Meaning?

pass

Slide
26

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Challenges (1/2)


Various standard clarifications were necessary:


13 CRs for TTCN
-
3 core languages have been written.


Unclarified

test cases have been excluded from the delivery
("on
-
hold").


Tools sometimes handle the standard differently.


Tests need to be expressed as tool independent as possible.


Test Automation:


No standard command line interface for batch run


Tool
-
specific adapters


No common test log format


TTCN3 semantics


TTCN3 standard part 1 defines both static and dynamic
semantics


No definition of well
-

and ill
-

formed programs


What to check at compile time is tool
-
dependent

Slide
27

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Challenges (2/2)


Synchronize ATS with new TTCN3 revisions


Time consuming manual activity


As ATS grows the effort will grow as well


To which core language standard version does the
conformance test suite relate to?


How to maintain compatibility across different language
versions?


Against which version should the conformance test suite be
developed in the future?


Which TTCN3 revision to support


Should we certify newest TTCN3 revision only?


Slide
28

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Conclusion


The conformance test suite is available now!


ATS Metrics:


Total number of conformance tests: 1077 (340 new)


20 tests fixed


9 removed since deprecated or duplicates


Number of positive syntactic conformance tests: 138


Number of negative syntactic conformance tests: 82


Number of positive semantic conformance tests: 517


Number of negative semantic conformance tests: 340


Number of clauses (at least partially) covered: 134 (24 new)

Slide
29

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India


30

TTCN
-
3 CONFORMANCE
TEST SUITE STF 433
STATUS REPORT

Nikolay

Pakulin


Requirements

tracking
case study

Slide
30

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Case study in a nutshell


4 chapters: total 7,5 pages


12 Declaring messages,


13 Declaring timers,


23 Timer operations,


24 Verdict operations;


56 requirements elicited,


7 requirements refined into 34 test purposes;


covered by 174 test cases


Average 7,5 requirements per page, 3 test
cases per requirement

Slide
31

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Requirements tracking case study


ISP RAS contributed to the project requirements
management tool
Requality


Hierarchical grouping of requirements


Requirements elicitation from DOC/HTML documents


Requirements are connected to certain locations in the
documents


Requirements locations are highlighted


Requirements refinement through test purposes


Test purposes define specific cases to be tested in order to treat
corresponding requirement ‘covered’


Requirements coverage by test cases report


Source document change management (beta)


Update requirements when the text of the standard is modified


Requality

is an open
-
source
plugin

to eclipse


Integrated with version control software (SVN)

Slide
32

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Sample view:
R
equality

in
TTWorkbench


Slide
33

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

/***************************************************


** @author STF 433


** @version 0.0.1


** @purpose 1:12, Ensure timer can be declared in components


** @verdict pass accept, ttcn3verdict:pass


***************************************************/

/*


* #
reqname

/Requirements/12 Declaring timers/Timers can be declared in component type


**/

module

Sem_12_toplevel_timer_001 {


type

component

TComp
{


timer

t_timer
:=10.0;


}


testcase

TC_Sem_12_toplevel_timer_001()
runs

on

TComp
{


t_timer.
start
;


if

(
t_timer.
running
){


setverdict
(
pass
)


}


else

{


setverdict
(
fail
)


}


}


control
{


execute
(TC_Sem_12_toplevel_timer_001())


}

}

Requirements Tracking (1/2)

Fully qualified name
of the requirement

Slide
34

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Requirements Tracking (1/2)


Requirements
coverage report


Integrated into
Eclipse


Evaluates
coverage of
requirements
and test
purposes

Slide
35

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

TTCN
-
3 User Conference

2012,
Bangelore
,
India

Suggested use case


Add requirements markup
to TTCN3 standard


Manual


Introduce references
between requirements and
TTCN3 test cases


Manual


Track requirements
changes between TTCN3
standard revisions


Automated (beta)


Track required changes in
TTCN3 ATS


Manual, to be automated

TTCN3
standard
updated

Changes
in
reqs
.
reviewed

New
reqs
.
added

ATS
updated

Slide
36

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing

Thank You!

Questions?

Slide
37

ETSI Update on TTCN
-
3 Language Evolution and Tool
Conformance Testing