CONNECT Tunable Performance Parameters Document

burgerraraSoftware and s/w Development

Nov 18, 2013 (3 years and 8 months ago)

159 views












CONNECT

Tunable
Performance Parameters


Document


Release #: 3.
3

Doc Id:
Draft_Rel3.3_Tunable_Performance_Parameters_Document

Draft Version: 0.
3

December
22
, 2011



CONNECTing th
e Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx


Tunable Performance Parameters Document

Created: July 25, 2011

Table of Contents

Revised: December 22, 2011/Version No. 0.3

ii

Table of Contents


Revision Log

................................
................................
................................
............................
1
-
1

1

Customization Parameters for CONNECT Deployments

................................
.................
1
-
2

1.1

Customization of Parameters for Hardware Environment and Operating System

......
1
-
2

1.2

Customizati
on of Parameters for Application Servers

................................
................
1
-
2

1.3

Customization of Parameters for the CONNECT Gateway

................................
........
1
-
2

2

Customization Parameters for Glassfish Deployments

................................
....................
2
-
3

2.1

Generic Customization Recommendations for Enhanced Performance

.....................
2
-
3

2.2

Specific Recommendations that Warrant Testing Before Customization

....................
2
-
4

3

Customization Parameters for JBoss Deployments

................................
.........................
3
-
6

3.1

Generic Customization Recommendations for Enhanced Performance

.....................
3
-
6

3.2

Specific Recommendation
s that Warrant Testing Before Customization

....................
3
-
6

4

Spring Proxy Tunable Parameters

................................
................................
...................
4
-
9

4.1

Recommended CONNECT Updates

................................
................................
........

4
-
10

4.1.1

Code Review

................................
................................
................................
.....

4
-
10

4.1.2

Recommendations

................................
................................
.............................

4
-
10

5

Log4j Tunable Parameters

................................
................................
............................

5
-
12

5.1

log4j.properties

................................
................................
................................
........

5
-
12

5.2

Threshold

................................
................................
................................
................

5
-
12

5.3

Appenders

................................
................................
................................
...............

5
-
12

5.4

Buffering

................................
................................
................................
..................

5
-
13

5.5

Hibernate Logging

................................
................................
................................
...

5
-
13

5.6

Code Review

................................
................................
................................
...........

5
-
14

6

Web Service Tunable Parameters

................................
................................
.................

6
-
15

6.1

Recommended Configuration

................................
................................
..................

6
-
15

6.1.1

Web Service Configuration

................................
................................
................

6
-
15

6.1.2

Gateway to Adapter Layer

................................
................................
.................

6
-
16

6.2

Recommended CONNECT Updates

................................
................................
........

6
-
16

6.2.1

Code Review

................................
................................
................................
.....

6
-
16

7

MPI and Patient Correlation Tunable Parameters

................................
..........................

7
-
17

7.1

Recommended Configuration

................................
................................
..................

7
-
17

7.1.1

MPI Adapter

................................
................................
................................
.......

7
-
17

7.1.2

Patient Correlation Adapter

................................
................................
................

7
-
17

7.1.3

Patient Discovery Mode
-

Trust, Verify, and Passthrough

................................
..

7
-
18


CONNECTing th
e Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx


Tunable Performance Parameters Document

Created: July 25, 2011

Table of Contents

Revised: December 22, 2011/Version No. 0.3

iii

7.2

Recommended CONNECT Updates

................................
................................
........

7
-
18

7.2.1

Code Review
-

MPI Ada
pter

................................
................................
...............

7
-
18

7.2.2

Code Review
-

Patient Correlation Adapter

................................
........................

7
-
18

8

Polic
y Engine Tunable Parameters

................................
................................
...............

8
-
19

8.1

Recommended Configurations

................................
................................
................

8
-
19

8.1.1

Pass Thru Mode

................................
................................
................................

8
-
19

8.1.2

Adapter Policy Engine

................................
................................
........................

8
-
20

8.1.3

Ada
pter Policy Engine Orchestrator

................................
................................
...

8
-
21

8.1.4

Adapter Policy Engine PEP
................................
................................
................

8
-
22

8.1.5

Adapter Policy Engine PIP

................................
................................
.................

8
-
23

8.1.6

Adapter Policy Engine PDP

................................
................................
...............

8
-
24

8.2

Code Review

................................
................................
................................
...........

8
-
24

8.3

Recommended CONNECT Updates

................................
................................
........

8
-
25

9

Audit Logging Tunable Parameters

................................
................................
...............

9
-
26

9.1

Recommended Configuration

................................
................................
..................

9
-
26

9.1.1

Database Configuration

................................
................................
.....................

9
-
26

9.1.2

Spring Proxy Configuration

................................
................................
................

9
-
26

9.1.3

Large Documents

................................
................................
..............................

9
-
27

9.2

Recommended CONNECT Updates

................................
................................
........

9
-
27

9.2.1

Java Adapters and Minimizing the Number of
Audits

................................
.........

9
-
27

9.2.2

Buffering

................................
................................
................................
............

9
-
27

10

Response Aggregation Tunable Param
eters

................................
...............................

10
-
28

10.1

Recommended Configuration

................................
................................
................

10
-
28

10.1.1

Database
Configuration

................................
................................
...................

10
-
28

10.1.2

Large Documents for Document Retrieve

................................
........................

10
-
28

10.2

Recommended CONNECT Updates

................................
................................
......

10
-
28

11

Callback Handler Tunable Parameters

................................
................................
........

11
-
30

11.1

Recommended Configurations

................................
................................
..............

11
-
30

11.2

Recommended CONNECT Updates

................................
................................
......

11
-
30

11.3

Recommendations

................................
................................
................................
.

11
-
30

12

Signature Approvals

................................
................................
................................
....

12
-
31



CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Revision Log

Revised: December 22, 2011/Version No. 0.3

1
-
1



Revision Log

The table below provides a log
for

each revision of the document that has been issued.

Date

Version
No.

Description

Author

Reviewer

Review
Date

12/07/11

0.1

Converted from Wiki pages to MS
Word
doc/template format

L. Angus

M. Weaver

12/09/11

12/09/11

0.2

Made
revisions

based on Technical
Writer review/comments

M. Weaver

L. Angus

12/12/11

12/22/11

0.3

Re
-
reviewed document and made
final edits

based on M. Weaver
updates

L. Angus

N/A

N/A

















CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Customization Parameters for CONNECT Depl
oyments

Revised: December 22, 2011/Version No. 0.3

1
-
2



1

Customization Parameters for CONNECT Deployments

This
document

provides

guidelines for the
T
unable
P
arameters to help improve CONNECT performance
as volume thresholds are reached.

The investigation into these
T
unable
P
arameters included four
areas
:



Hardware



Operating System



Application Server



CONNECT Configuration

However, please note that other factors outside of these four areas can influence the performance of the
CONNECT Gateway, such as network speed, bandwidth, or bottlenecks.

1.1

Customization of Parameters for Hardware Environment and Operating System

While analyzing hardware environments and operating systems for tunable performance parameters, it
was determined that it is not feasible to provide generic recommendations for tunab
le parameters in those
areas.


1.2

Customization of Parameters for Application Servers

Glassfish on Microsoft Windows, RHEL, and Solaris
:



Customization parameters for Glassfish deployments

JBoss 5.1.0GA on RHEL
:



Customization parameters for JBoss deployments

1.3

Customization of Parameters for the CONNECT Gateway



Spring Proxy Tunable Parameters



Log4j Tunable Parameters



Web Service
Proxy Tunable Parameters



MPI and Patient Correlation Tunable Parameters



Policy Engine Tunable Parameters



Audit Logging Tunable Parameters



Respons
e Aggregation Tunable Parameters



Callback Handler Tunable Parameters







CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Customization Parameters for Glassfish Deployments

Revised: December 22, 2011/Version No. 0.3

2
-
3



2

Customization Parameters for Glassfish Deployments

Customization of Glassfish deployments can be made via an update to the domain.xml in the domain
configuration folder at the file system, via the web based administration interface, or via the command
line utility (asadmin).

Please refer to the
Sun Glassfish Enterprise
Server 2.1.1 administration guide

for
specific instructions on how to perform configuration updates.

2.1

Generic Customization Recommendations for Enhanced Performance

1.

Java version

Many performance optimizations are incorporated in newer releases of Java SE.

It is
recommended that the latest update be used.

Recommendation
: Use the latest JDK 1.6 i.e. JDK 1.6 Update 27.

2.

JVM Mode

Default Glassfish application server configuration uses the Client JVM mode.

This is to support a
development profile with faster sta
rtup and deployment times.

For higher performance,
the

Server JVM mode

is recommended
.

Recommendation
: Set the JVM to server mode using the "
-
server" JVM parameter.

3.

Access logging

The CONNECT gateway track transactions via the audit services.

It is benef
icial to reduce
unnecessary I/O associated with access logging.

Recommendation
: Turn off access logging.

4.

HTTP file cache

Static files are categorized based on
user
-
defined

limits into small, medium, and large groups.

Contents of small files are read into the JVM heap.

This increases the JVM heap size which may
lead to an increase in garbage collection.

Medium files are memory mapped, thereby increasing
the memory footprint of the process.

Number and size of static fi
les, frequency of file request, heap space configuration, and amount of
available memory must be considered while configuring the HTTP file cache.

Recommendations
:



For most cases, the default number of files cached (1024) is acceptable.



The maximum age of
a cache entry (max_age_in_seconds) can be configured to ensure a
few cache hits before the file is

removed from cache.



The space available for small files can be configured based on JVM heap space
configured.



The space available for medium files must consi
der the total amount of process
addressable memory space available (especially 4GB limit for a 32
-
bit JVM).



CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Customization Parameters for Glassfish Deployments

Revised: December 22, 2011/Version No. 0.3

2
-
4



5.

Configuration updates for web
-
apps

The default
-
web.xml defines filters and security constraints applicable to all web apps.

The
parameter developme
nt=true is default and enables JSP updates to be immediately available to
clients, with a server side cost associated. This parameter must be set to false on production
systems.

Also
,

ensure that parameter, genStrAsCharArray, is set to true.

6.

JDBC tuning

T
uning the JDBC connection pool as per application requirements is a standard recommendation.


It is beneficial to tune steady
-
pool
-
size and max
-
pool
-
size to the same value, where the rule of
thumb is to keep this value equal to the number of HTTP processor

threads.


Database drivers that support statement caching provide better performance via reuse of prepared
statements.

For MySQL databases (used with the CONNECT gateway), the properties are as
follows:


<property name="cachePrepStmts" value="true">

<pro
perty name="prepStmtCacheSize"
value="512">

<property name="useServerPreparedStmts" value="false">

Recommendations
:



Set steady
-
pool
-
size and max
-
pool
-
size to be equal to the number of HTTP processor
threads.



Set the MySQL JDBC statement caching properties
(as provided above).


2.2

Specific Recommendations that Warrant Testing Before Customization

1.

Java Heap Size

The Java Heap Size must be tuned based on the total available memory in the system, process
data model (32
-
bit or 64
-
bit), and operating system.

The
default maximum is set to 512 MB.

For 32
-
bit process models, the limits (with no OS tweaks) are:



Windows: 1.5 GB



Solaris: 3.5 GB



Linux: Between 1.4 and 3.5 GB (depending on Linux version)


For 64
-
bit process models, the maximum heap size is theoretically u
nlimited.


Larger JVM Heap sizes also imply larger garbage collection times, especially for a full GC
cycle.

Recommendation
: The Java Heap Size must be tuned based on the total available memory
in the system, process data model (32
-
bit or 64
-
bit), and oper
ating system.





CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Customization Parameters for Glassfish Deployments

Revised: December 22, 2011/Version No. 0.3

2
-
5



2.

Tune Java
G
arbage
C
ollection

Garbage collection (GC) can be analyzed using JVM switches (
-
verbose:gc,
-
XX:+PrintGCTimeStamps,
-
XX:+PrintGCDetails), and using visual tools that support such
analyses.

GC tuning is not required unless there i
s an overt GC problem.


Single process machines with small heap sizes:

Use serial GC (default)


Server class machines with multiple processors:

Use parallel GC (default)


Multi
-
threaded machines using multiple threads for minor collections:

Use parallel

GC (
-
XX:UseParallelGC)

Multi
-
threaded machines for major collections:

Use serial GC (default)


For shorter response times (shorter GC pauses) at the expense of throughput:

Use the CMS
GC collector (
-
XX:+UseConcMarkSweepGC)

Recommendations
:



GC should be
tuned only in case of overt GC issues.



Read
Tuning Garbage Collection with the JVM


3.

HTTP
A
cceptor
T
hreads

HTTP acceptor threads accept new incoming connections.

Default is 1.

Generally, 1 thread
for 1
-
4 CPU cores is suggested.

Recommendation
:

Testing is required to find an optimal number of HTTP acceptor
threads.

4.

HTTP
R
equest
P
rocessing
T
hreads

HTTP request processing threads retrieve and process incoming HTTP r
equests.

The default
number is 5.


The HTTP request processing threads should be tuned based on the number of CPUs.

During
testing, this number can be increased until throughput starts to decline (at this point, request
processing threads are contending
for CPU resources).

Recommendation
:

Testing is required to find an optimal number of HTTP request
processing threads.

5.

HTTP
T
hread
C
ount

HTTP thread count acts much like acceptor threads.

Generally, 1 thread for 8 CPU cores is
suggested.

Recommendation
:

The default value may be changed for a more capable hardware
platform.

References:

Optimize Glassfish Performance




CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Customization Parameters for JBoss Deployments

Revised: December 22, 2011/Version No. 0.3

3
-
6



3

Customization Parameters for JBoss Deployments

R
efer to the
JBoss Enterprise Application Server 5 Administration and Configuration Guide

for specific
instructi
ons on how to perform configuration updates.

3.1

Generic Customization Recommendations for Enhanced Performance

1.

Java
V
ersion

Many performance optimizations are incorporated in newer releases of Java SE.

It is
recommended that the latest update be used.

Recommendation
:

Use the latest JDK 1.6
,
i.e
. JDK 1.6 Update 27.

2.

JVM Mode

For higher performance, we recommend using the Server JVM mode.

Recommendation
:

Set the JVM to server mode using the "
-
server" JVM parameter.

3.

Configure Logging for
P
roduction

Logging at the application server level should be configured to ensure optimum performance.
JBoss 5 provides a configuration set, named Production, which provides a better starting point for
creating a better production environment, including logging confi
guration.



Console logging:

In JBoss default configuration, console logging is enabled. This is
necessary for development, but

expensive in production with unbuffered I/O.

High
-
volume applications benefit from turning it off.


Recommendation
:

Turn off co
nsole logging.

3.2

Specific Recommendations that Warrant Testing Before Customization

1.

Java Heap Size

The Java Heap Size must be tuned based on the total available memory in the system, process
data model (32
-
bit or 64
-
bit), and operating system.

The default m
aximum is set to 512 MB.


For 32
-
bit process models, the limits (with no OS tweaks) are:



Windows: 1.5 GB



Solaris: 3.5 GB



Linux: Between 1.4 and 3.5 GB (depending on Linux version)


For 64
-
bit process models, the maximum heap size is theoret
ically
unlimited.


Larger JVM Heap sizes also imply larger garbage collection times, especially for a full GC cycle.


Recommendation
:

The Java Heap Size must be tuned based on the total available memory in
the system, process data model (32
-
bit or 64
-
bit), and
operating system.



CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Customization Parameters for JBoss De
ployments

Revised: December 22, 2011/Version No. 0.3

3
-
7



2.

Tune Java
G
arbage
C
ollection

Garbage collection (GC) can be analyzed using JVM switches (
-
verbose:gc,
-
XX:+PrintGCTimeStamps,
-
XX:+PrintGCDetails), and using visual tools that support such
analyses.

GC tuning is not required unless there

is an overt GC problem.


Single process machines with small heap sizes:

Use serial GC (default)

Server class machines with multiple processors:

Use parallel GC (default)

Multi
-
threaded machines using multiple threads for minor collections:

Use parallel

GC (
-
XX:UseParallelGC)

Multi
-
threaded machines for major collections:

Use serial GC (default)

For shorter response times (shorter GC pauses) at the expense of throughput:

Use the CMS GC
collector (
-
XX:+UseConcMarkSweepGC)

Recommendations
:



GC should be t
uned only in case of overt GC issues.



Read
Tuning Garbage Collection with the JVM


3.

JDBC
C
onnection
P
ool
C
onfiguration

Database connections are expensive to set up and tear down.

Connection pool utilization may be
monitored from

the EAP JMX console, from JBoss ON, or from database
-
specific tools.


Set the minimum database connection pool size to the required level,
and th
en

set the maximum
at least 25
-
30% higher.

If the maximum is set too high the connection pool shrinks automatically.


Recommended additions to the <local
-
tx
-
datasource> from <datasources> (replace ?? with actual
value)

<min
-
pool
-
size>??</min
-
pool
-
size>

<m
ax
-
pool
-
size>??</max
-
pool
-
size>

<transaction
-
isolation>TRANSACTION_READ_COMMITTED</transaction
-
isolation>

<prepared
-
statement
-
cache
-
size>100</prepared
-
statement
-
cache
-
size>

<shared
-
prepared
-
statements>true</shared
-
prepared
-
statements>

Recommendation
:

Test
ing is required to find the optimal value of database pool size.

If the
maximum is set too high the connection pool shrinks automatically.

4.

Thread pooling

JBoss 5 has robust thread pooling, and several thread pools working together. The significant
thread
pools in the CONNECT context are:



System thread pool:

Defined in jboss
-
service.xml in the conf directory,
it’s

used for JNDI
naming.

The default setting is fine in most cases



HTTPd thread pool in JBoss Web:

Defined in the server.xml file under
<server>
/deploy/jboss
-
web
-
sar.

Threads work with inbound HTTP requests.





CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Customization Parameters for JBoss Deployments

Revised: December 22, 2011/Version No. 0.3

3
-
8



The thread pools can be monitored via the JBoss 5 Admin Console.

The admin console displays
and allows configuration of the number of active threads and the queue size.

Recommendation
:

Te
sting/monitoring thread pools
are

required to find an optimal number of
threads in the HTTPd thread pool.

References:


Best practices for performance tuning JBoss Enterpri se
App
lication Platform 5






CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Spring Proxy Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

4
-
9



4

Spring Proxy Tunable Parameters

CONNECT makes heavy use of dependency Injection/Inversion of Control through
S
pring
p
roxies.

For
more information on these concepts
refer to

the
Links

on

page

4
-
11, below
.

The main purpose for
viewing
the

S
pring
p
roxy framework in CONNECT was to provide guidance on how to configure the
proxies for maximum performance.

A secondary goal was to provide a detailed analysis of the
implementation in order to make recommendations
for updates to CONNECT.

Recommended Configurations

Recommendations for Spring
p
roxy settings are
dependent

on the deployment environment.

Given the
large number of Spring
p
roxy configurations and the parallels between them, it is not feasible to go
through each Spring
p
roxy configuration and provide tailored recommendations.


The following are
general recommendations.

It is trivial to order various Spring
p
roxy configuration

by their level of performance, and therefore
wherever possible, it is recommended to use the setting closer to the top of the list.



NoOp (No Operation)

-

NoOp Implementations do not execute any processing
code;

they simply
form an expected result and retu
rn that result to the caller.

Because these implementations do not
perform any processing, it is generally not possible to use these implementations unless your
organization does not require that a message be processed by a given interface.

For example,
if
your organization wants to take advantage of CONNECT gateway level processing of a Patient
Discovery message (correlation, etc
.
), but you do not implement policy checks, then it would be
appropriate to set the Adapter Policy Engine Spring proxy configur
ation to NoOp.




Java

-

Java implementations are quick and efficient
proxies that

make
Java

(jvm) calls directly to
the given proxy's target implementations.

Java implementations are generally recommended for
their performance and enhanced processing capab
ilities over the NoOp implementations.
However, these proxies are not capable of making calls to components deployed on separate
JVMs.




Web Services

-

Web Service implementations are relatively inefficient especially compared to
NoOp and Java, however, Web

Service implementations allow for processing to be distributed
across multiple JVMs.

This configuration would be used heavily in the Adapter/Gateway
deployment model
,

where the CONNECT Gateway and the CONNECT Adapters are deployed
on two separate logical

machines.


In that case a proxy on the gateway would be set to web
services, and a web service would be called on the adapter machine for the adapter to continue to
process the message.




Secured Web Services

-

Secured Web Service implementations are simil
ar to normal web
service implementation, except the addition processing required to secure and send the message
lead to lower performance.

These configurations would be used in a similar fashion to normal
web
services;

however these
provide the added
benefits of 2
-
way SSL and SAML assertions to
protect the web service message.






CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Spring Proxy Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

4
-
10



4.1

Recommended CONNECT Updates


4.1.1

Code Review

The Spring proxy framework is generally well written.

For each interface controlled by a Spring

proxy
configuration there is an Object Factory.

All of these various Object Factories

extend
ComponentProxyObjectFactory by providing context such as the bean a given Object Factory is
interested in and the Spring configuration xml file in which that bea
n is defined.

This abstract class
contains the heavy lifting of Spring proxy framework.

The method "getBean()" pulls the appropriate bean
as defined in the <interface>ProxyConfig.xml.

The "getBean()" method make
s

use of

a synchronized
method called "get
Context()"
. T
he synchronization keyword protects the Spring ApplicationContext
from being updated by multiple threads at any given time
.


There were only two notable findings observed while code reviewing the Spring proxy framework.



The use of the synchro
nized keyword on the getContext method could be a performance
bottleneck.

Under a heavy concurrent user load, processing a
message that

uses multiple spring
proxies, there is the potential that threads may have to suspend and wait for previous calls to
co
mplete executing this method.

It is eas
il
y possible that this blocking and waiting could happen
multiple times on the same thread or while processing the same transaction.




There are inconsistencies with the naming of WSDL files, and Spring proxy configur
ation files.
This can make it difficult to find particular WSDL or Spring proxy configuration files and should
be corrected for consistency's sake.



4.1.2

Recommendations

Based on the research conducted to identify tunable configuration parameters, the followin
g
recommendation for changes to CONNECT are to be documented and prioritized on the CONNECT
Backlog.


The ComponentProxyObjectFactory "getContext()" method is synchronous.

In order to be thread safe, it
is not an option to remove
synchroniz
ation

from this method.

Therefore
,

the only other mitigating
strategy would be to minimize the number of calls to, and the processing time in getContext.

Other

possible tactics for achieving this strategy would be to:



Reduce the number of spring proxies by rem
oving unnecessary proxy boundaries. There are many
interfaces which implement Spring proxies
.
CONNECT should be analyzed to identify Spring
proxies

and remove those

which are not providing any value. An example would be the Adapter
Policy Engine Orchestra
tor
.




Aggregating different proxies into logical groups so that more configurations are loaded in any
given trip to the file system.





CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Spring Proxy Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

4
-
11



Links

http://en.wikipedia.org/wiki/Dependency_injection

http://en.wikipedia.org/wiki/Inversion_of_control

http://static.springsource.org/spring/docs/1.2.x/refe
rence/beans.html

http://static.springsource.org/spring/docs/2.5.x/reference/beans.html






CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Log4j Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

5
-
12



5

Log4j Tunable Parameters

CONNECT uses Apache's log4j framework for logging capabilities.

Third party libraries that are bundled
with CONNECT, such as Metro, also uses log4j.

This document provides recommended logging
configuration and code updates for improving gateway performan
ce.

5.1

log4j.properties

Log statements are pervasive throughout the code base to provide transparency to the internal message
processing of the gateway, and as such, the CONNECT application is considered as a heavy logger.

By
default, the log4j.properties fi
le is set to the highest logging level and will log all statements to file.

This
is the recommended setting during development as it provides detailed information for debugging.

But in
a production environment, this setting incurs a very heavy penalty to

performance due to the frequent file
I/O access that needs to occur.

Consequently, it is highly recommended to tune log4j before deployment
to only log statements of interest.

The following sections will provide configuration recommendations to
the log4
j.properties file that will give the most significant performance improvement.

5.2

Threshold

Log4j

uses the threshold property to determine the level of logging to be written to file.

In order of
lowest to highest:



TRACE

-

designates finer
-
grained information
al events than the DEBUG
.



DEBUG

-

designates fine
-
grained informational events that are most
useful to debug an
application.



INFO

-

designates informational messages that highlight the progress of the appli
cation at
coarse
-
grained level.



WARN

-

designates
potentially harmful situations.



ERROR

-

designates error events that might still allow the application to continue running
.



FATAL

-

designates very severe error events that will presumably lead the application to abort.

There are also two special levels
that the threshold can be set:



ALL

-

has the lowest possible rank and is intended to turn on all logging
.



OFF

-

has the highest possible rank and is intended to turn off logging.

The higher the threshold, the less log statements are written.

It is recomm
ended to set the level to WARN
or ERROR in a production environment.

5.3

Appenders

The log4j.properties file included in the CONNECT install contains two appenders:

ConsoleAppender
and RollingFileAppender.

Both appenders

write to files under the glassfish domain/logs directory
.



ConsoleAppender

-

This appender writes to the server.log file.

Log statements from log4j as
well as statements from standard out are written here
.



RollingFileAppender

-

This appender writes to th
e NHA
-
Gateway.log file.

Only log statements
from log4j are written here.





CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Log4j Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

5
-
13



By default, CONNECT is configured to use both appenders.

The RollingFileAppender contains only
CONNECT related logs while the ConsoleAppender contains both CONNECT and Application

Server
logs.

This means that for each log statement invoked by the CONNECT code, two file writes will occur.


It is recommended that in a production level environment, only one appender be used to minimize file I/O
writes.

5.4

Buffering

Log4j

allows log mess
ages to be buffered before they are written to an appender file.

When this is not
configured, the logging mechanism works synchronously and does an immediate write for each log
statement encountered.

Unless real time logging is required, it is recommende
d that the appenders
be

configured to use buffering.

To enable buffering, use the following parameters:



immediateFlush

-

When enabled, this flag will flush the underlying writer after each append
operation.

By default, this is set

to true.



bufferedIO

-

Setting this flag to true will enable buffered writing.

By default, this is set to false.



asyncAppender

-

Setting this flag to true will buffer entire log4j message objects.

This has more
overhead than regular logging.

By default, this is set to fals
e.

If logging is local (ex. log file is located locally), then configure log4j to use an appender with the
following properties:

log4j.appender.Console.immediateFlush = false

log4j.appender.Console.bufferedIO = true

If network logging is used, then configu
re the appender with the following properties

log4j.appender.Console.immediateFlush = false

log4j.appender.Console.bufferedIO = true

log4j.appender.Console.asyncAppender = true


Enabling asyncAppender for network logging will lower the number of times log4
j has to go across the
network by increasing the buffer size.

Although the bufferSize parameter can also be used,
asyncAppender will ensure that entire messages are flushed out.

5.5

Hibernate Logging

CONNECT uses the Hibernate framework for object relational
mapping. By default, the hibernate
configuration files are set to log all database query statements to standard out.


It is recommended that these SQL statement dumps are turned off when the gateway is deployed in a
production environment. To disable them,

configuration files corresponding to each database table used
by CONNECT has to be modified. These cfg.xml files can be found under the glassfish domain's
config/nhin/hibernate directory. To disable logging, set the following property to false:

<property
name="show_sql">false</property>





CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Log4j Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

5
-
14



Alternatively, if logging the statements
is

desired, then it is recommended to configure Hibernate to use
log4j so that it can be managed by the framework for better performance (such as allowing buffering).

By
setting t
he org.hibernate.SQL logger to 'DEBUG' or 'ALL' and assigning it an appender, all SQL
statements will be logged.

5.6

Code Review

CONNECT uses the log4j logging mechanism properly.

No changes are required that will significantly
improve performance.





CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Web Service Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

6
-
15



6

Web
Service Tunable Parameters

All web service calls in CONNECT are handled in the same manner and processing goes through the
same set of classes.

The main purpose of this
section

is to provide recommended configuration settings
and recommended future update
s to CONNECT.

6.1

Recommended Configuration


6.1.1

Web Service Configuration

The WebServiceProxyHelper class uses the following parameters from the gateway.properties file:



webserviceproxy.timeout



webserviceproxy.retryattempts



webserviceproxy.retrydelay



webservicepr
oxy.exceptionstext


The
webserviceproxy.timeout

parameter specifies the amount of time in milliseconds that a call will
timeout if no activity occurs.

The
webserviceproxy.retryattempts

parameter specifies the number of times a call should retry if a web
service exception occurs with the given webserviceproxy.exceptionstext during invocation.

The
webserviceproxy.retrydelay

parameter specifies how long to wait before the proxy class attem
pts
another web service call.

The
webserviceproxy.exceptionstext

parameter specifies the text to look for when a web service
exception occurs during a call.

By default, this is set to SocketTimeoutException.

By default, the timeout parameter is set to 2 m
inutes.

This value is specifically set to handle service calls
that have very long response times.

The drawback to this high timeout value is that it will cause
significant slowdowns when CONNECT is communicating with unreliable gateways as each call cou
ld
potentially not return for 2 minutes.

This is especially exacerbated by the retryattempts and retrydelay
parameter as a service call could potentially suspend a socket until

timeout + (retryattempts
-
1) * (timeout + retrydelay)

milliseconds

have passed.

If the gateway is under a high load and communicating with unresponsive
servers, it could grind message processing to a halt.

Consider reducing the timeout value from the default value if it is not expected for the service calls to
take a lo
ng time. On the other hand, increase this value if
long response times are

expected. Timing out
before a service has a chance to reply will cause unnecessary retries.





CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Web Service Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

6
-
16



6.1.2

Gateway to Adapter Layer

The best way
to avoid

the performance penalty of invoking a we
b service is to not use a web service if it
is not required.

CONNECT allows you to plug in any adapters you choose to implement, and throughput
will be higher if the overhead of generating SOAP messages is avoided.

Refer to

the
Spring Proxy
Tunable Parameters

for a more thorough discussion on how to use Spring
p
roxies to switch out adapter
implementations.

As always, if the adapter is in the same host as the gateway, then a
Java

implementation proxy is
recommended.

If not, the
n you can use the web service proxies that
are

already included in CONNECT
or write your own customized adapter.

If a web service is not required, then there are more efficient ways
to communicate between the CONNECT gateway and the adapters.

For example
, you can implement
your adapters to use XML over HTTP or JSON over HTTP as both have
fewer overheads

compared to
SOAP.

If your adapters are implemented in Java, you can even use RMI to gain even better performance.
It is recommended to skip SOAP web services altogether if possible and implement Java
adapters, which

in turn leverage JMS or JSON.

For more information on how

to develop JSON adapters,
refer to

the
JSON Adapter Implementation

document.

6.2

Recommended CONNECT Updates


6.2.1

Code Review

CONNECT invokes each service call
through the same set of classes.

When creating the port object for
the service call, it first creates
,

and then

caches the corresponding service factory class.

Creating the
service factory is expensive and time consuming, and consequently, the first time

a service is called, the
response time will be much higher than subsequent calls.

Caching the instance at creation is the
recommended approach, and CONNECT already implements this method.

After the port object is created, it uses an instance of the WebSe
rviceProxyHelper utility class to do the
actual web service call.

This helper class uses
Java

reflection to invoke the web service method and reads
from the gateway.properties file for configuration parameters.

Using reflection is not ideal.

Java documen
tation recommends not
using

reflection due to performance
reason.

http://download.oracle.com/javase/tutorial/reflect/index.html





Reflection is powerful, but should not be used indiscriminately.

If it is possible to perform an operation
without using reflection, then it is preferable to avoid using it.


Performance Overhead

Because reflection involves types that are dynamically resolved, certain Java virtual machine optimizations
cannot

be performed.

Consequently, reflective operations have slower performance than their non
-
reflective
counterparts, and s
hould be avoided in sections of
code that

are called frequently in performance
-
sensitive
applications.



CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

MPI and Patient Correlation Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

7
-
17



7

MPI and P
atient Correlation Tunable Parameters

The MPI adapter in CONNECT handles all the persisting and retrieving of patient information.

Similarly,
the Patient Correlation component handles the persisting and retrieving of patient correlations that occur
during

patient discoveries and document queries.

The main purpose of this
section

is to provide
recommended configuration settings and recommended future updates to CONNECT for MPI and Patient
Correlation.

7.1

Recommended Configuration


7.1.1

MPI Adapter

By default, CONNECT is configured to use an MPI Adapter that is only meant as a reference for
developers, and as such, it should never be used in production.

This adapter stores patient information
using a simple text file and will be inadequate for any s
ituations outside of testing.

Starting with release
3.2, there is another provided implementation that stores patient information in the database.

Even so,
this adapter only offers the most basic capabilities.

CONNECT assumes that users will build their

own
MPI adapters, and consequently, the provided implementations are not meant to be used.


It is recommended that the user develop MPI adapters

themselves
.

The ones included in the CONNECT
gateway are meant as a reference implementation.

To properly co
nfigure CONNECT to use a customized
adapter, modify the AdapterMpiProxyConfig file.

If you are using a web service implementation, modify
the internalConnectionInfo.xml file: mpi or mpisecured.


If the basic MPI adapters included in CONNECT is sufficient
for the user, then it is recommended to use
the implementation that uses the database to store patient information.

To configure CONNECT to use
this implementation, modify the AdapterComponentMpiCheckerConfig.xml configuration file to use the
PatientDbChe
cker class.


If possible in your deployment environment, modify the MPI
Spring proxy

Configuration files
(AdapterMpiProxyConfig and AdapterComponentMpiProxy) to use the
Java

implementations.

7.1.2

Patient Correlation Adapter

The patient correlation adapter inclu
ded in CONNECT is responsible for two functions: adding patient
correlations and retrieving them.

Correlation is persisted using a database.


Correlation is most often
invoked during patient discovery and document query with more calls occurring in the in
itiating gateway
side.


One thing to note about the correlation code is that there exists database administration processing that is
piggy backed whenever a retrieve is performed.

Each retrieval for a specific patient id will result in a
check of the quer
y results whether a correlation has expired or not.

If it is expired, then that entry is
deleted from the database table.

Although functional, there are two problems with this approach.


For
one, if a patient correlation has not occurred for that specifi
c patient for a long time, then the first request
will likely need several database deletions before processing can move forward.

Depending on the
number of correlations for that patient, this could be expensive.


Another problem is that if a table cleanup
only occurs when a patient correlation is performed for that specific patient, then it is very likely that in a
production environment that has been deployed for a significant amount of time, that table will incl
ude
mostly expired entries.

Most patients will not be correlated often
,

which means clean up for expired
entries will not be frequent.




CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

MPI and Patient Correlation Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

7
-
18



In a production environment, it is recommended to create a cron job that will do a periodic cleanup of the
correlation
table of expired entries.

This will relieve the database administration processing that occurs in
the code.


If possible in your deployment environment, modify the PatientCorrelationConfig.xml to use a
Java

implementation.

7.1.3

Patient Discovery Mode
-

Trust,
Verify, and Passthrough

Patient Discovery's validation mode is controlled by the
patientDiscoveryResponseMode

parameter in
the gateway.properties file and can have the following values:

verify, trust, and passthrough.

Each mode
determines how patient cor
relation is performed during a patient discovery transaction.



Verify Mode

-

This mode of response indicates the responding gateway to verify the match
before agreeing on the match and create a patient correlation.



Trust Mode

-

This mode of response does no
t require the responding gateway to verify the
match
,
but instead creates a patient correlation directly based on the responding message.



Passthrough Mode

-

The responding gateway sends the message directly to the agency without
creating any patient correl
ation.

It is recommended that you consider carefully how your gateway is configured as the default value is
Varify.

Verify is the most time consuming of all modes
,

as it will query the local MPI before it creates a
correlation through the Patient Correlat
ion Adapter.

Trust mode is the second
,

as it bypasses the initial
MPI query.

Passthrough is consequently the fastest
,

as it simply passes the response message and will not
create a correlation.

7.2

Recommended CONNECT Updates


7.2.1

Code Review
-

MPI Adapter

The MPI adapter implementation does not follow the same flow as the other adapters in the CONNECT
code.

It has three layers of Spring
p
roxies
,

(AdapterMpiProxyConfig, AdapterComponentMpiProxy, and
AdapterComponentMpiChecker)
,

while other adapters only hav
e one.


It is recommended that MPI adapter processing be refactored to remove these extra layers
,

which will
greatly simplify the code base and increase the reference adapter performance.

7.2.2

Code Review
-

Patient Correlation Adapter

The implementation is fair
ly straightforward
,

but there are sections of the code that could be optimized.
One major improvement that can be made is to minimize the number of database transactions.

Database
queries should be combined whenever possible, and the Storer/Retriever clas
ses do not take this into
consideration.


It often uses Java code to post process results to compensate for not generating the correct
queries.


In addition, batch processing should be supported.

Currently, the interface to the patient
correlation adapter

only supports one patient correlation at a time.


If a patient discovery query is initiated
and a response is returned with multiple patient ids for correlation, then each of those ids will generate a
separate request to the correlation service.

This is
expensive especially if the interface is a web service.
Ideally, these requests should be batched together into one single request.


Currently, the interface does
not support this and the code will need to be refactored.




CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Policy Engine Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

8
-
19



It is recommended that the Patient

Correlation code be refactored to minimize the number of database hits
by creating more efficient combined queries when possible.


In addition, the Patient Correlation interface
should be modified to support batch processing
.


8

Policy Engine Tunable Parame
ters

The Policy Engine services in CONNECT are complex
,

but well documented (
Enterprise Service
Component
-

Policy Engine
).
,

Aside from what is documented in the previous link, it should also be noted
that in most service implementations, the notion of "pass through mode" generally means that policy
checks are skipped (however this may not always be a good option depending on a
ny additional logic that
would be short circuited via "pass through mode").


8.1

Recommended Configurations

Some general recommendations around spring proxies can be found at the
Spring Proxy Tunable
Parameters

page.

Recommendations

for spring proxy settings are dependent on the deployment environment.

Refer to the
following flow chart for organization specific recommendations.

8.1.1

Pass Thru Mode

Does your organization
implement any of CONNECT’s
Policy Engine interfaces
?
No
Yes
Does your
organization use
“pass thru” mode
?
(
Continue
)
Set “pass thru”
mode in
gateway
.
properties
Set Adapter Policy
Engine to
NoOpPermit
Yes
No


Notes
:

Policy Engine interfaces include:




Adapter Policy
Engine



Adapter Policy Engine Orchestrator



Adapter Policy Enforcement Point



Adapter Policy Information Point



Adapter Policy Decision Point



Adapter Policy Engine spring configuration is in PolicyEngineProxyConfig.xml





CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Policy Engine Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

8
-
20



8.1.2

Adapter Policy Engine

Is your organization’s
Adapter Policy Engine
implementation on a
separate JVM from your
gateway
?
Set the Adapter Policy
Engine spring proxy to
Java
.
No
Set the Adapter Policy
Engine spring proxy to
Web Services
Yes
Which interface does your
custom policy engine
adapter provision
?
AdapterPolicyEngine
Set the Adapter Policy
Engine spring proxy to
Secured Web Services
AdapterPolicyEngineSecured

Notes
:

Adapter Policy Engine
S
pring configuration is in PolicyEngineProxyConfig.xml


Custom Adapter implementations will provision one of the following interfaces:



Java

-

gov.hhs.fha.nhinc.policyengine.adapter.proxy.PolicyEngine
Proxy.java



Web Services

-

AdapterPolicyEngine.wsdl



Secured Web Services

-

AdapterPolicyEngineSecured.wsdl


If your organization has developed a custom implementation of one of the above interfaces, the following
S
pring proxy configurations will not be used
:



Adapter Policy Engine Orchestrator



Adapter Policy Enforcement Point



Adapter Policy Information Point



Adapter Policy Decision Point





CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Policy Engine Tunable
Parameters

Revised: December 22, 2011/Version No. 0.3

8
-
21



8.1.3

Adapter Policy Engine Orchestrator

Set the Adapter
Policy Engine
Orchestrator spring
proxy to Web
Services
No
Is your organization’s Adapter
Policy Engine Orchestrator on a
separate JVM from your gateway
?
Set the Adapter
Policy Engine
Orchestrator spring
proxy to Java
.
Yes
No
:

Notes:

Adapter Policy Engine Orchestrator spring
configuration is in
AdapterPolicyEngineOrchestratorProxyConfig.xml


Custom Adapter implementations will provision one of the following interfaces:



Java

-
gov.hhs.fha.nhinc.policyengine.adapter.orchestrator.proxy.AdapterPolicyEngineOrchProxy.java



Web Service
s

-

AdapterPolicyEngineOrchestrator.wsdl


If your organization has developed a custom implementation of one of the above interfaces, the following
S
pring
proxy configurations will not be used:



Adapter Policy Enforcement Point



Adapter Policy Information
Point



Adapter Policy Decision Point







CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Policy Engine Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

8
-
22



8.1.4

Adapter Policy Engine PEP

Set the Adapter Policy
Enforcement Poin
spring proxy to Web
Services
Is your organization’s Adapter
Policy Enforcement Point on a
separate JVM from your gateway
?
Set the Adapter Policy
Enforcement Point
spring proxy to Java
.
Yes
No
:

Notes
:

Adapter Policy Enforcement Point spring configuration is in AdapterPEPConfig.xml

Custom Adapter implementations will provision one of the following inter
faces:



Java
-

gov.hhs.fha.nhinc.policyengine.adapter.pep.proxy.AdapterPEPProxy.java



Web Services
-

AdapterPEP.wsdl


If your organization has developed a custom implementation of one of the above interfaces, the following
S
pring

proxy configurations will not be used:



Adapter Policy Information Point



Adapter Policy Decision Point





CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Policy Engine Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

8
-
23



8.1.5

Adapter Policy Engine PIP

Set the Adapter Policy
Information Point
spring proxy to Web
Services
Is your organization’s
custom Policy Information
Point adapter on a separate
JVM from your gateway
?
Set the Adapter Policy
Information Point
spring proxy to Java
.
Yes
No


Notes:

Adapter Policy Information Point spring configuration is in AdapterPIPConfig.xml

Custom
Adapter implementations will provision one of the following interfaces:



Java
-

gov.hhs.fha.nhinc.policyengine.adapter.pip.proxy.AdapterPIPProxy.java



Web Services
-

AdapterPIP.wsdl





CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Policy Engine Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

8
-
24



8.1.6

Adapter Policy Engine PDP

Set the Adapter
Policy Decision Point
spring proxy to Web
Services
Is your organization’s
custom Policy Decision Point
adapter on a separate JVM
from your gateway
?
Set the Adapter
Policy Decision
Point spring proxy
to Java
.
Yes
No


Notes
:

Adapter
Policy Information Point spring configuration is in AdapterPEPConfig.xml

Custom Adapter implementations will provision one of the following interfaces:



Java

-

gov.hhs.fha.nhinc.policyengine.adapter.pip.proxy.AdapterPEPProxy.java



Web Services

-

(Not Available)


Caveat:

Answering "Yes" to the questions about your organization using the CONNECT Policy
Information Point or Policy Decision Point Reference Adapters implies that your organization stores
patient consent XACML documents in your XDS.b

repository.

8.2

Code Review

The Policy Engine services code was generally well written.

As documented in the above Policy Engine
link, there are 5 major components to the Policy Engine services including; the Policy Engine Processor,
the Policy Engine Orches
trator, the Policy Enforcement Point, the Policy Information Point, and the
Policy Decision Point. This architecture is complex, however it is consistent with the extensible access
control markup language (XACML) model, as well as abstraction patterns else
where in CONNECT.


There were only two findings observed, including some duplication of efforts in the reference adapter
implementation, as well as possible over use of the
S
pring proxy design paradigm.

Both the Policy Information

Point and the Policy Deci
sion

Point appear to retrieve the same XACML
document from the document repository while executing their respective responsibilities.

This is
potentially problematic on two levels.



CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Policy Engine Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

8
-
25



Firstly, if a document is retrieved once, that effort should not need to b
e duplicated by retrieving the
document a second time.

Unfortunately this may

be a drawback to the XACML model, because while the
CONNECT Reference Adapter PIP and PDP do retrieve this document, there is no

guarantee at run time
that either of those compo
nents won't be replaced by implementation specific adapters.
Refer to

the
Recommended CONNECT Updates section for recommendations on this finding.


Secondly, each of the previously described 5 components of the CONNECT Policy Engine Reference
Adapter are s
eparated by
S
pring proxy configuration boundaries
,

which provide the option at runtime to
switch between
Java
, web services, secure web services, and "no op" implementations.

These
S
pring
proxies, while generally beneficial, have the potential to introduc
e performance issues as much as they
can mitigate them depending on the configuration.

Refer to

both the Recommended Configurations and
Recommended CONNECT Updates for recommendations on this finding.

8.3

Recommended CONNECT Updates

Based on the research
cond
ucted

here to identify tunable configuration parameters, there are also some
recommendations for changes to CONNECT to be documented and prioritized on the CONNECT
Backlog.

1.

There is little benefit of having the
S
pring proxy abstraction for the Adapter Poli
cy Engine
Orchestrator.

This layer can be removed to eliminate the overhead of checking the configuration
and dependency, injecting the appropriate implementation.

There is not enough of a distinction in
the interface or purpose of the Adapter Policy Eng
ine, and Adapter Policy Engine Orchestrator,
nor are there multiple

Adapter Policy Engine Orchestrator implementations to warrant a separate
interface.


2.

It is complex and expensive to retrieve XACML documents via XDS.b because of the registry
query, parsin
g of the result, and then the repository query.

We should consider a "one stop"
mechanism to store and retrieve these documents or work to improve the performance of the
CONNECT Reference XDS.b implementation.

Once the document is retrieved, we might also
consider a faster approach to parsing the document.

Currently, the XACML document is de
-
serialized into a JAXB object to be parsed.

Consider using xpath or xquery instead.


3.

We should examine the XACML specifi
cation as well as reference adapter implementation to
look for performance improvements.

A bug was identified while analyzing the Adapter Policy Engine Reference Adapter that should be
prioritized and fixed per the development process.

Refer to

GATEWAY
-
7
47
(
https://issues.connectopensource.org:8443/browse/GATEWAY
-
747
) for more details.






CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Audit Logging Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

9
-
26



9

Audit Logging
Tunable
Parameters

Audit logging occurs during all message transactions in the

CONNECT gateway.

For each transaction,
four calls to the audit adapter are invoked. The included audit adapter then creates a record of the message
at each call.

Consequently, two records (inbound and outbound) are added within the Gateway
-
to
-
Gateway tr
ansaction and the other two within the Gateway
-
to
-
Adapter transaction.

This document
provides recommended configuration and code updates to the audit component to optimize gateway
performance.

9.1

Recommended Configuration

To improve CONNECT audit logging performance, consider the following adjustments around database
and
S
pring proxy configurations, as well as some recommendations for large files.


9.1.1

Database Configuration

One of the bottlenecks for the auditing adapter is the database.

T
here are four database inserts per
message transaction, and each message can potentially be large
. T
he database persistence consumes most
of the processing time for audits. Under load, res
ource contention is observed with the database write
operation given that multiple threads attempt (blocking) write operations on the same database table.


One configuration modification that was tested during this analysis was to change the database stora
ge
engine for the auditrepository table.

By default, MySQL uses MyISAM.

For the test, the table was
modified to use the InnoDB engine to take advantage of row level locks rather than table level locks.

ALTER TABLE auditrepo.auditrepository ENGINE = InnoD
B;

As the audits are all inserts and all the inserts are done at the end of the table, it was expected that row
locks would decrease the number of transactions waiting for a table lock.

Indeed, when the stress test was
executed against the gateway, the nu
mber of table_locks_waited went down to zero.

SHOW status LIKE '%
\
_locks
\
_%'

But even with the change, the gateway did not seem to have gained any performance improvement.

It is
possible that the drawbacks of using InnoDB negated the improvements gained f
rom row level locking.
This is likely due to InnoDB having slower writes.

If using this approach, further configuration to
InnoDB will be needed to get faster writes.

Increase the innodb_buffer_pool_size and
innodb_file_io_threads to possibly achieve bet
ter performance.


If using the included audit adapter, it is recommended to tune the database to increase performance.


As
most of the database transactions are for writes, ensure that your configuration is tuned accordingly.

In
addition, row level locks
will be better than table level locks.

In addition, consider using other database
engines such as Oracle to better improve performance.

9.1.2

Spring Proxy Configuration

As expected, the main bottleneck in performance is the configuration to use audit web servic
es in place of
a
Java

implementation.

It is recommended that when possible, the AuditRepositoryProxyConfig.xml be
set to use
Java
.


In addition, if the user prefers, they can take over all the auditing responsibilities from the gateway.

This
is done by s
etting the AuditRepositoryProxyConfig.xml to use NoOp.

In this case, no auditing will be
conducted

at the CONNECT Gateway, and the adapters will be expected to perform all necessary audits.



CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Audit Logging Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

9
-
27



It is recommended that if the user is using their own adapter, th
ey modify the
AuditRepositoryProxyConfig.xml to use NoOp to avoid unnecessary calls to the audit component.

In
addition, they should ensure that document saves are minimized.

9.1.3

Large Documents

Audit transactions save the entire message, including document a
ttachments to the repository.

For
medium to large sized documents, this can become intractable.

A document retrieve for a large document
will have the document saved twice; once when the adapter responds to the gateway and another when the
gateway sends it to the
Nationwide Health Information Network
.


If a production level gateway plans on supportin
g the passing of large documents, the audit adapter needs
to be modified.

A possible solution is to save the document once for both audits or to have the audit share
the same document reference with the document retrieve adapter.


In addition, if the audi
t is to use the default MySQL database for large documents, the
max_allowed_packet needs to be increased to allow for large BLOBs to be saved.

9.2

Recommended CONNECT Updates


9.2.1

Java Adapters and Minimizing the Number of Audits

If the gateway is configured to us
e
Java

adapters, it may make sense to turn off 2 of the audits between
the gateway and adapter level.

Requirements dictate that there needs to be an audit every time a message
passes from one actor to another.


Fulfilling this requirement is dependent on
each deployment, and it
may be better to make the number of audits a tunable parameter.

9.2.2

Buffering

As logging is not considered an immediate priority during processing, it is recommended that the
CONNECT gateway be modified to support buffering of audit mes
sages.


Performance can be gained if
the audit occurs over the network as it will minimize the number of remote calls.

This feature will need
to be configurable so that the user will have the option to turn buffering off if the audit adapter is local.






CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Response Aggregation T
unable Parameters

Revised: December 22, 2011/Version No. 0.3

10
-
28



10

Response Aggregation Tunable Parameters

To improve CONNECT response aggregation performance consider the following adjustments around
CONNECT configuration and recommended code updates.

CONNECT configurations include database
configuration and considerati
ons for large files,
while
recommended

code updates focus on potentially
unneeded database processing and thread waits.


10.1

Recommended Configuration

Response aggregation in CONNECT occurs when an initiating gateway sends a request to multiple targets
in the
Nationwide Health Information Network
.

For each response the gateway receives, it aggregates
and combines them into a single response to send back to the initiating adapter.

As this is one of the
heavier computations involved in CONNECT message processin
g, this document provides recommended
configurations and code updates to increase gateway performance.

10.1.1

Database Configuration

Aggregation for document query and document retrieve relies heavily in the database for processing.

It
uses tables to record each

returned response from each target and combines them when all the responses
have been received.

Consequently, to achieve better performance, ensure that the database used by the
gateway is tuned properly.


It is recommended that the database used by the
CONNECT gateway be tuned for optimization.

In
addition, consider using other database engines such as Oracle to better improve performance.

10.1.2

Large Documents for Document Retrieve

For document retrieve, any
response with a large document is

saved in the fil
e system as part of the
aggregation processing.

Any files greater than the size specified in the gateway.properties
(
aggregatorMaxDbResponseSize
) will be saved in the temporary directory specified by the
aggregatorLargeResponseDir

parameter.

Any files sm
aller than the threshold will be saved directly to
the database.

Once all the responses have been received, the aggregation component reads and deletes
these files and combines them into a single response to be sent back to the requester.

Therefore
,

to e
nsure
that the file I/O time is minimized, point the
aggregatorLargeResponseDir

to a local directory.


It is recommended that the location pointed to by the
aggregatorLargeResponseDir

parameter in the
gateway.properties file is pointing to a local director
y.

10.2

Recommended CONNECT Updates

Prior to CONNECT 3.3, when a query request is fanned out by the entity service to all intended targets,
each sends
occurred

sequentially.

This is clearly a major source of higher response times and a
concurrent implementatio
n is included in the CONNECT 3.3 release.


Another problem found in the previous aggregation implementation is that there is unnecessary
processing done when there is only one target in the request.

When there
was

only one response
expected
,

the message was still marshalled/unmarshalled and read/saved to the database even though
passing it through would have sufficed.





CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Response Aggregation Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

10
-
29



In addition, aggregation processing for document query and document retrieve contained unnecessary
waits for responses.

As
previous implementations had only
synchronous

sends, these waits were
unnecessary as the responses were immediately available.

In addition, error handling for document
retrieve was not handled properly and did not halt the wait loop when a request failed.


These waits then
had to time out for an error response to be sent back.


It is recommended that the latest CONNECT version be used, as most of these problems will be fixed
with the latest version of CONNECT.






CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Callback Handler Tunable Parameters

Revised: December 22, 2011/Version No. 0.3

11
-
30



11

Callback Handler Tunable Parameters

Callbac
k Handlers in CONNECT are an implementation of a JAX
-
WS/WSIT (Metro) paradigm for
handling message level policies such as SAML and security.

CONNECT currently contains three
Callback Handlers, one for SAML, one for the certificate keystore, and one for th
e certificate truststore.
Each of these Callback Handlers
is

called every time a message that requires SAML or SSL is sent.
Callback Handlers only manage SAML and message level security for web service clients.

This means
that they are only invoked when C
ONNECT is acting as an initiating gateway and not as a responding
gateway.

11.1

Recommended Configurations

Considering that there is no configuration around this aspect of Callback Handlers and
keystores/truststores, this topic contains only recommendations for

updating CONNECT.

11.2

Recommended CONNECT Updates

The Callback Handler code is generally well written.

The only outstanding issue that was identified was
the fact that the keystore and truststore

are being read (file I/O) with every call to the Callback Handlers.
Considering the pervasiveness of secured web service calls throughout CONNECT, there is a chance that
this file I/O can cause performance
bottlenecks
.

11.3

Recommendations

Based on the researc
h conducted here to identify tunable configuration parameters, there are also some
recommendations for changes to CONNECT to be documented and prioritized on the CONNECT
Backlog.

There is an existing paradigm in the CONNECT design to read a file or configuration set when
it is needed and then cache the file for later invocations.

There is also a "last modified" tracker that is
checked to determine if a cached object is stale, in w
hich case, it is reloaded.

This paradigm could be
reused in this case to improve performance after the first execution.


While making these improvements, it would also be advisable to investigate generalizing the code which
implements this paradigm in oth
er parts of the CONNECT architecture into one generic class.

However,
consider how using one static map/hash table for all cached objects could introduce another performance
bottleneck
.





CONNECTing the Health IT Community


Doc ID:
burgerrara_54fe3aee
-
c8de
-
43e3
-
b639
-
eda02cc728f3.docx

Tunable Performance Parameters Document

Created: July 25, 2011

Signature Approvals

Revised: December 22, 2011/Version No. 0.3

12
-
31



12

Signature
Approvals


The undersigned acknowledge that
the reviewer(
s)
have reviewed the
Tunable
Performance
Parameters

document

and agree with the information presented within this document. Changes
to this document will be coordinated
with and approved by the undersigned or their designated
representatives.


Signature:


Date:


Print Name:




Title:




Role:





Signature:


Date:


Print Name:




Title:




Role:





Signature:


Date:


Print Name:




Title:




Role: