Best Strategies to avoid worst performance failures in Investment Banking

wispxylopolistInternet and Web Development

Aug 7, 2012 (4 years and 8 months ago)

333 views

BFS
-
US SSDU Research

BFS
-
US SSDU Research

Best Strategies to avoid worst
performance failures in Investment
Banking

Sujatha Badri Narayanan

Karthik Authoor Venkata



Infosys Limited (NASDAQ: INFY)

BFS
-
US SSDU Research

BFS
-
US SSDU Research

Abstract


Investment
banking functionalities are grouped
into front
office, middle
office and back office.



Performance
testing specifically related to trading applications within
investment banking plays a vital role
.



Era has shifted from traditional manual trading in stock exchanges like
NASDAQ, Globex to electronic trading.



People trade millions of shares in micro seconds with various exchanges
these days using electronic trading.

2

BFS
-
US SSDU Research

BFS
-
US SSDU Research

Why this Paper?

“Every large system that works started as a small system that worked.”


Anonymous


“Fast, good, cheap: pick any two.”


Anonymous


“Lots of methodologies have come and gone, paradigms have changed but the requirements are
always the same; Make it good and make it fast.”


Anonymous


In Electronic trading, complex heterogeneous systems are spread over diverse technologies with various
platforms ranging from legacy mainframe to today’s web based technologies are used.


Today’s systems and interfaces are built with their own protocols both at application layer and network
layer.


Even minor performance improvement in these systems will yield mammoth benefits leading to savings in
millions of dollars.


In this whitepaper we will attempt to describe some of the good performance testing
strategies that can be followed to immunize ourselves from disasters in the Investment
banking domain specifically in the trading business.





3

BFS
-
US SSDU Research

BFS
-
US SSDU Research

Few Performance experiences as seen by User

4

Sl.No

User experience Scenarios in the Trading Business

1

Session login failed.

2

Market related data reaching the end user late due to network traffic rendering the info useless

3

Market data graphs failed to pull the real time data from the server

4

Viewing the detailed transaction statements on the web application giving distorted or half information due to delay
in response from server

5

CMP(Current Market price) dashboard showing old market price for the stocks as the real time market update is
slow

6

Placing a trade order from Stock trading web application during market mayhem (lot of trading activities happening
at the same time). Server timed out.

7

Placed a trade order from web application during heavy traffic. Trade got executed but the confirmation never came
back leaving the transaction in dilemma stage

8

Making a critical payment from bank web app to a third party using secure payment network. The money got
debited from my account but due to heavy traffic on the third party server, the transaction failed.

9

Downloading “My” bank statement on the desktop which failed

10

Making a mobile payment from network switches from 3G to Wireless and thus failing the transaction

BFS
-
US SSDU Research

BFS
-
US SSDU Research

Proposed Strategies for better user experiences

Following are the key strategies proposed, that might help to provide solutions to some of the

performance issues that have been experienced by users as listed in the previous slide:


1)
Performance Testing from an Architect’s View


Performance test driven architecture

2)
Infrastructure Optimization like Server, Database, Network etc
-


3)
Optimal combination and usage of tools based on technology used


Fiddler, Web Load, VSTS,
Jmeter, LR, Wily Introscope,
Dynatrace
,
Shunra
, Strobe,
Xcap

etc.

4)
Component based testing


Stressing the system at component level

5)
Seamless Failover Testing


Time taken for switch over during disaster



5

BFS
-
US SSDU Research

BFS
-
US SSDU Research

Strategy 5:

Seamless
Failover Testing

Strategy 2:

Infrastructure Optimization

Performance Strategy layers


An End
-
to
-
End View

6

Strategy 1 :

Architect’s View

Strategy 3 :

Optimal combination and usage of tools based on technology



Load

Balancer

Web Server

Web Server

App Server

App Server

MF

DB

BFS
-
US SSDU Research

BFS
-
US SSDU Research

Strategy 1:

Performance Testing from an Architect’s View

7

Life Cycle Phases

Performance Strategy Activities

Requirements

gathering

/

analysis

/

validation


卹獴敭

un摥牳瑡r摩dg


乆N

条瑨t物rg


啮U敲獴慮摩dg

乆N

&

癡汩摡瑩湧

瑨tm


Modeling

o
NFR

o
Workload

Architecture

Analysis

&

Design


卣慬慢楬楴y

慳a敳獭敮t



瑨t

慲捨楴散eure


Validation

of

the

design

from

a

Performance

stand

point


䍡p慣楴y

慮d

獩s楮i

p污ln楮i



瑨t

楮晲慳瑲a捴ure


P敲fo牭慮捥

偲敤楣i楶e

䵯摥汩Mg



䑥獩杮楮g

瑨t

p敲fo牭慮捥

T敳e

獴牡瑥ty

BFS
-
US SSDU Research

BFS
-
US SSDU Research

Strategy 1:

Performance Testing from an Architect’s View..
Contd

Following are the performance test strategy activities (as mentioned against each lifecycle phases in the

previous slide) that can be leveraged


1)
Performance Testing from an Architect’s View


Performance test driven architecture

2)
Infrastructure Optimization like Server, Database, Network etc
-


3)
Optimal combination and usage of tools based on technology used


Fiddler,
WebLoad
, VSTS, Jmeter,
LR, Wily Introscope,
Dynatrace
,
Shunra
,
Xcap

etc.

4)
Component based testing


Stressing the system at component level

5)
Seamless Failover Testing


Time taken for switch over during disaster



8

BFS
-
US SSDU Research

BFS
-
US SSDU Research

Strategy 2:
Infrastructure Optimization

Why

Infrastructure

Optimization

?


Major

component

in

an

organization’s

CAPEX


Level

of

optimization

is

a

key

goal

for

many

organizations


Superior

user

experience

which

is

the

core

business

should

also

not

get

affected


Most

Commonly

used

servers

in

Investment

Banking



Web

Server

:

Tomcat,

IIS

App

Server

:

Tomcat,

IIS,

IBM

WebSphere

Database

:

Oracle

10
g,

11
g,

Microsoft

SQL

Server



9

BFS
-
US SSDU Research

BFS
-
US SSDU Research

Suggested optimizations on
TomCat
:




Have the latest OS patches, Service packs and JDKs installed as higher versions always have better
performance and better security.




Have windows server instead of a non server version as non server version will have various
limitations like number of users etc.




Connector tuning will yield good results


Generally Client creates the connection and once server
sends response to the request connection is closed. One may land up opening 100s of connection to
render a single page


Modify “
MaxKeepAliveRequests

to a higher value option to avoid this.




Tomcat also has a thread pool. Make the best use of the same by adjusting
minProcessors

and
maxProcessors

values.





10

Strategy 2:

Infrastructure Optimization
-

Web Server

BFS
-
US SSDU Research

BFS
-
US SSDU Research

Suggested optimizations on IIS:




Configure the following parameters like Shutdown worker processes when idle for good performance












Managing log files will result in good performance in almost all the servers.



Make sure that only required values are selected for logging and old log files are archived to avoid
space issue.


11

Strategy 2:

Infrastructure Optimization
-

Web Server …
Contd

BFS
-
US SSDU Research

BFS
-
US SSDU Research

12

Strategy 2:

Infrastructure Optimization
-

Application Server

Suggested optimizations on
WebSphere

Application Server for better performance :



Modify Initial Heap Size and Maximum Heap size to 3096



Have
Loglevel

severity to “error” or “fatal”




“show application print statements” can be disabled



IBM Service Logs can be disabled (Process
Defnitions

-
> Logging & Tracking
-
> IBM Service Logs)



Have a strategy around connection pool management in WAS

BFS
-
US SSDU Research

BFS
-
US SSDU Research

13

Strategy 2:

Infrastructure Optimization


Database Server

Suggested optimizations on Oracle DB Server for better performance :



Ensure that the database is loaded to the required volume before Load testing.


As in any other server
system resources monitoring
plays a vital role in DB server also.


Besides CPU, Memory, ensure that Disk I/ O is monitored.


Ensure that all the disks are properly loaded.



Other major aspects of
Oracle

tuning are :
-


Denormalizing a database


If your database is doing too many joins then you can consider
this option to improve performance


Indexing (B
-
Tree Concept)


-

is used in databases to speed up retrieval process

-
is one of the most time tested technique that can be used to improve performance



Full Table Scan
-

it is better to go for full table scan for smaller tables

BFS
-
US SSDU Research

BFS
-
US SSDU Research

14

Strategy 2:

Infrastructure Optimization


Database Server ..
Contd

Tuning SQL itself is another interesting topic :



You can make use of AWR logs and other monitoring tools


to know what queries take more time
and


what queries are called often and then select the queries that should be tuned.



Following are few samples of SQL tuning :
-


If you are retrieving from multiple tables, the sequence of the tables matters a lot for
performance. Optimizers (if tuned properly


cost optimization)does help in making such
decisions.


If it is a read only table read once and cache them so that we can avoid reading multiple times


Do not read the entire table when only few columns are required. DB filters can be handy
here.

BFS
-
US SSDU Research

BFS
-
US SSDU Research

15

Strategy 3:

Optimal combination and usage of tools

Selecting an optimal combination of tools plays a vital role in identifying bottlenecks.



While selecting
load generation tools

we may need to take into the account the following key
factors:


Protocol Support required for your application like :

Http, Win socket, SOAP, WAP, IMAP, POP3, EJB, RMI
-
Java, COM/DCOM, Tuxedo, Seibel, SAP
R/3, Terminal Emulation, SSL, XML, PeopleSoft, VT100
-
520, ODBC, FTP, Streaming Media,
LDAP, TCP/IP, UDP, MMS, Microsoft .NET SOAP Stack, CORBA, Oracle OCI, Ajax


Ease of Parameterization and Correlation


Network address management


User ramp up / down facility


Scheduling tests


License details


Customer Support


BFS
-
US SSDU Research

BFS
-
US SSDU Research

16

Strategy 3:

Optimal combination and usage of tools…
Contd


While selecting
monitoring tools
we may need to take into the account the following key factors:


Technology support required for your application
-

.Net
, J2EE, Mainframe etc


Supported system resources monitoring


Depth in analysis For e.g. does it help in identifying bottlenecks at code level ? Query level ?


License details


Customer Support

There are many Performance tools in the market but based on our experience in Investment banking
we find that the following tools are most commonly used.

Strategic Areas / Technology

.Net

J2EE

Mainframe

Load Generation Tools

VSTS

HP LR ,
Radview’s

Web load, Compuware
QALoad, Borland ‘s Silk Performer, Microsoft
Web Application Stress Tool (WAS)

HP LR using
RTE protocol

Monitoring Tools

IIS Debug
Diagnostics

HP Diagnostics, Wily Introscope, Dynatrace, Site
scope

Strobe

BFS
-
US SSDU Research

BFS
-
US SSDU Research

17

Strategy 3:

Optimal combination and usage of tools…
Contd

There are two types of monitoring tools


Agent based
tools like HP Diagnostics ,Wily Introscope


Agent less
tools like Site Scope

Sample Agent based Monitoring Tool output


Wily Introscope














Detailed analysis up to code / query level can be done in agent based monitoring.

BFS
-
US SSDU Research

BFS
-
US SSDU Research



SLAs should be broken down during architecture phase for each of the components.



Components should be individually load/volume tested to make sure that they meet required SLA.



This will ease bottleneck identification and help in addressing the same at an earlier stage rather
than struggling during final phase of the project.


18

Strategy 4:

Component based Testing

BFS
-
US SSDU Research

BFS
-
US SSDU Research


In investment banking as availability at the right time matters most, failover testing is essential.



In failover testing , the behavior of the system will be tested by bringing down the servers in a logical
way


web, app etc.



In many instances CPU utilization was high and later during analysis we realized that Load Balancers
were not working as expected.



Make sure that your strategy covers testing load balancers also.


19

Strategy 5:

Failover Testing

BFS
-
US SSDU Research

BFS
-
US SSDU Research


Designing for the right performance and ensuring that performance is engineered into the product /
application is key.



Specifically in investment banking business , organizations cannot afford to incur higher expenditure on
application failures and higher
opex

on unnecessary infrastructure, hence performance testing
becomes an indispensable activity for optimizing the quality of the application



We recommend organizations to view performance quality from the bigger picture and adopt the best
strategies which were discussed during these sessions so that business at large is benefited thereby
making organizations less vulnerable to competitive pressures.

20

The Way ahead

BFS
-
US SSDU Research

BFS
-
US SSDU Research

21

Conclusion and Results

Performance testers should also focus on
performance engineering standpoint
in addition to load
testing / volume testing strategies.

User experience Scenarios

Strategy

1

Session login failed.

Load beyond expected may be the reason. Capacity
Modeling could have avoided this situation


Strategy 1

2

Market related data reaching the end user late
due to network traffic rendering the info useless

Response time not meeting SLAs


Strategy 2 & 3

3

Market data graphs failed to pull the real time
data from the server

May be the server responsible for pulling the data is
down. Strategy 5

4

Viewing the detailed transaction statements on
the web application giving distorted or half
information due to delay in response from
server

Database Server response may be slow. Strategy 3

5

CMP(Current Market price) dashboard showing
old market price for the stocks as the real time
market update is slow

Possible reason could be bottleneck in one of the
components


Strategy 4

BFS
-
US SSDU Research

BFS
-
US SSDU Research

22

Conclusion and Results …
Contd

User experience Scenarios

Strategy

6

Placing a trade order from Stock trading web
application during market mayhem (lot of trading
activities happening at the same time). Server timed
out.

Load beyond expected may be the reason.
Capacity Modeling / Proper Workload

modeling

could have avoided this situation


Strategy 1

7

Placed a trade order from web application during
heavy traffic. Trade got executed but the confirmation
never came back leaving the transaction in dilemma
stage

Incomplete transaction due to poor response
time. Possible reason could be bottleneck in
one of the layers / components


Strategy 4

8

Making a critical payment from bank web app to a
third party using secure payment network. The
money got debited from my account but due to
heavy traffic on the third party server the
transaction failed.

Possible reason could be bottleneck in one of
the layers / components


Strategy 4

BFS
-
US SSDU Research

BFS
-
US SSDU Research

References and Acknowledgements

References


http
:
//msdn
.
microsoft
.
com


Tomcat



The

definite

guide

by

Jason

Brittain

&

Ian

F
.

Darwin


Performance

Tuning

for

Apache

Tomcat

by

Mark

Thomas


Oracle

Performance

Tuning

by

Peter

Corrigan

and

Mark

Gurry


Performance

Testing

Advanced

Certification

material

from

Infosys


Acknowledgements


Vijay

from

Infosys

Infrastructure

Management

Services


Sanjeeb

Kumar

from

Financial

Services

team

for

sharing

the

user

experiences


Finacle

SPE

team


BFS
-
US SSDU Research

BFS
-
US SSDU Research

Q&A:

sujatha_b03@infosys.com


Karthik_venkata@infosys.com




24