Deploying Tomcat Clusters in an Advanced BlackboardEnvironment

fallsnowpeasInternet και Εφαρμογές Web

12 Νοε 2013 (πριν από 3 χρόνια και 5 μήνες)

68 εμφανίσεις

© Blackboard, Inc. All rights reserved.

Deploying Tomcat Clusters in an
Advanced Blackboard

TM

Environment


Speaker: Steve Feldman

Director, Software Performance Engineering and Architecture

sfeldman@blackboard.com


2

Agenda and Introductions

»
Goals, Objectives and Outcomes

»
Background Knowledge

»
Hands on Learning

»
Benchmark Statistics

»
Questions…Questions…Questions

»
Total Time: 50 Minutes

3

Note to Attendees

»
Tomcat clustering is not a standard configuration of the
Blackboard Academic Suite.

»
Clusters have been set
-
up and tested by Blackboard Engineers,
but have not been certified as a valid configuration.

»
The technology is proven and used throughout the world.

»
The same can be said of server virtualization

»
Many Blackboard clients use and deploy virtual servers.

»
The guidance provided in this presentation is to help clients take
advantage of the power of clustering, but help clients understand
that they must have knowledge on setting up and managing
clusters.

4

Presentation Goals

»
The goals of this presentation are:

»
Introduce the concepts and techniques for planning and
deploying Tomcat Application Clusters.

»
Help administrators understand the value of Tomcat
clusters for performance and availability.

»
Show administrators how to set
-
up their very own
Tomcat cluster.

5

Presentation Objectives

»
Define what are clusters.

»
Define what are nodes.

»
Explain the differences between cluster nodes and load
balanced nodes.

»
Review the configuration requirements for setting up a
Tomcat cluster

»
Review the configuration requirements for setting up web
server distribution to the cluster.

»
Explain issues that might come up with clusters and
provide techniques for diagnosing and resolving the
issues.

»
Convince every customer to go home and start setting up
clusters in their development environments.

6

Presentation Outcomes

»
At the end of the session, administrators will be
able to do the following:

»
Define the difference between a load
-
balanced
configuration and a tomcat cluster.

»
Set
-
up a Tomcat cluster or a load
-
balanced nodes

»
Configure clustering for quality of service.

7

What is a Tomcat Cluster?

»
A cluster is a collection of two or more Tomcat
instances running in parallel and working together
to provide robust scalability, performance and
reliability.

»
Clusters can be multi
-
homed and reside
physically on the same hardware.

»
Clusters can be distributed and reside on
distributed hardware.

»
Clusters are indistinguishable to end users
presenting a seamless experience

8

Why Cluster?

»
Clusters distribute workload

»
Minimize unnecessary queuing

»
When configured for a small JVM it can reduce the amount of JVM
pause times.

»
Clusters can help better utilize the hardware you run on

»
Clusters enable a scale
-
up model to go along with load
-
balanced
installations (scale
-
out model)

»
Clusters can provide additional availability

»
Not recommend using Tomcat Clusters as the primary failover
solution

»
Clusters are cool…and because they are cool you should
consider them!

»
Just one person’s opinion!

9

Visual Look at a Tomcat Cluster

Web Server

Instance

Node 1

Node 2

Node N

Traffic is directed by the

Web Server like a software load balancer

Nodes know little

about there partners


Recommend 1 node

Per Core


10

What is an Application Node?

»
An application node is a single instance of
Tomcat.

»
Nodes can be part of a cluster.

»
Means they are aware of each other

»
Part of a group

»
Failover

»
Nodes can be part of a load
-
balanced
configuration.

»
A single physical server can have multiple logical nodes
not in a cluster, but technically load
-
balanced by JK

11

Clustered Nodes versus Load
-
Balanced Nodes

»
A load
-
balanced node is the following:

»
Physically distinct from a hardware perspective.

»
Has its own web server managing traffic.

»
Has a dedicated configuration and resource definition.

»
Has no understanding of failover definition.

»
Relies on third
-
party resources like load
-
balancers.

»
A clustered node is the following:

»
Can be physically or logically distinct from a hardware perspective.

»
Shares a web server with potential N+ nodes.

»
Shares configuration and resource definition with other nodes.

»
Has failover definition as part of the application configuration.

12

Load
-
Balanced and Clustered Nodes

Web Server

Instance

Node 1

Node 2

Node N

Web Server

Instance

Node 1

Node 2

Node N

Hardware or Software


Load Balancer

Cluster

Load
-
Balanced

Nodes unaware
of each other.

Nodes aware of
each other.

13

Session Replication

»
To enable session replication in Tomcat, three
different paths can be followed to achieve the
exact same thing:

»
Using session persistence, and saving the session to a
shared file system (PersistenceManager)

»
Using session persistence, and saving the session to a
shared database (JDBCManager)

»
Using in
-
memory
-
replication, using the
SimpleTcpCluster that ships with Tomcat 5
(server/lib/catalina
-
cluster.jar)

»
Recommend Using SimpleTcpCluster


14

Guidelines for Setting Up a Cluster

»
The cluster configuration is completely described in the
server.xml file under
$BLACKBOARD_HOME/apps/tomcat/conf

»
You can set
-
up a true cluster or settle for load
-
balanced
JVMs.

»
Either way you win.

»
Create multiple clusters in a multi
-
homed configuration
initially

»
Each multihomed cluster is capable of failing over to each other
rather then across physical systems

»
Eliminates the need for cross
-
over connections between servers.


15

Ingredients of a Tomcat Cluster

»
Class Name= Must have class package

»
org.apache.catalina.cluster.tcp.SimpleTcpCluster

»
Cluster Identification and Listen Port

»
Multicast Communication Address

»
JK Connector Port

»
JK Configuration Definition for Load Distribution

»
A Few Directory Copies

»
$BLACKBOARD_HOME/apps/tomcat/bin

»
$BLACKBOARD_HOME/apps/tomcat/conf

»
$BLACKBOARD_HOME/apps/tomcat/work

»
$BLACKBOARD_HOME/apps/tomcat/log

16

Example Cluster Code


<Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"


managerClassName="org.apache.catalina.cluster.session.DeltaManager"


expireSessionsOnShutdown="false"


useDirtyFlag="true"


service.mcastBindAddress="127.0.0.1"


notifyListenersOnReplication="true">



<Membership className="org.apache.catalina.cluster.mcast.McastService"


mcastAddr="228.0.0.4"


mcastBindAddress="127.0.0.1"


mcastPort="45564"


mcastFrequency="500"


mcastDropTime="3000"/>



<Receiver className="org.apache.catalina.cluster.tcp.ReplicationListener"


tcpListenAddress="auto"


tcpListenPort="4000"


tcpSelectorTimeout="100"


tcpThreadCount="6"/>



<Sender className="org.apache.catalina.cluster.tcp.ReplicationTransmitter"


replicationMode="pooled"


ackTimeout="15000"/>



<Valve className="org.apache.catalina.cluster.tcp.ReplicationValve"


filter=".*
\
.gif;.*
\
.js;.*
\
.jpg;.*
\
.png;.*
\
.htm;.*
\
.html;.*
\
.css;.*
\
.txt;"/>


</Cluster>


17

Easiest Way to Skin a Tomcat Cluster

»
Copy
$BLACKBOARD_HOME/apps/tomcat

to
$BLACKBOARD_HOME/apps/tomcat_node2

»
Change all references of

$BLACKBOARD_HOME/apps/tomcat
to now
use

$BLACKBOARD_HOME/apps/tomcat_node2


»
Modify server.xml in new directory to reflect unique server
port for AJP13 connector and shutdown.

»
Cluster ports need to be unique in order to create a socket unless
IP Addresses are defined and uniquely defined.

»
Modify CATALINA_PID in blackboard
-
tomcat.sh or .bat file
to point to the right node and log file directory.

18

Setting Up JK Load Distribution

»
Apache and IIS will route traffic based on JK
definition.

»
Best to set
-
up a new workers.properties

»
Located under
$BLACKBOARD_HOME/apps/tomcat/conf/jk/workers.properties

»
Change httpd.conf to point to new workers.properties
(Unix)

»
Managed from within IIS

»
Blackboard Web Site


Jakarta

»
Simply copy workers.properties and rename old file


19

Example Workers.Properties

# One virtual worker

worker.list=loadbalancer


# Configure the virtual worker as a load balancer

worker.loadbalancer.type=lb

worker.loadbalancer.balanced_workers=tomcat1,tomcat2

worker.loadbalancer.method=Request


# Attempt to

worker.loadbalancer.sticky_session=1


# Actual workers, not available directly

worker.tomcat1.port=8009

worker.tomcat1.host=localhost

worker.tomcat1.type=ajp13

worker.tomcat1.lbfactor=50

worker.tomcat2.port=8209

worker.tomcat2.host=localhost

worker.tomcat2.type=ajp13

worker.tomcat2.lbfactor=50

20

Web Server Configuration

»
Apache manages Tomcat worker information in
httpd.conf

»
Define workers.properties

»
Establish JK Mount (Bottom of File)

»
Probably called new worker list “loadbalancer”

»
Windows manages Tomcat worker information un
uriworkermap.properties

»
Must note new worker name for Cluster (aka: worker
list)

21

Things to Consider

»
Apache has a limit of 1024 maxclients

»
More then enough for a 2, 3 or 4 node cluster

»
Must always keep maxclients = (Tomcat Max Server
Processes) * (# of Tomcat Nodes per Cluster)

»
Recompiling Apache is one way to solve the problem


22

Advanced Feature: Setting Up QOS Models

Application Server
Tomcat
DBI
JDBC
Tomcat
Modperl
/
PerlEx
Apache
/
IIS
Change
JKMount

**JK Mount says /webapps/* but
can be changed to any single
webapp if you choose.

23

Example Benchmark Statistics

T2000 Analysis and Clustering
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
18,000
20,000
Configuration
Sessions Per Hour
Single Server 2 Nodes
Load Balanced 1 Node Each
Load Balanced 2 Nodes Each
24

Example Benchmark Statistics

Workload

R7.1 Entry
-
Level

R7.1 Mid
-
Level

R7.1 High
-
Level

R7.1 HL
Clustered

R1 (Workload of 120
Possible Concurrent
Simulations Learning
System/Community
System)

7238 Sessions/Hr

19 UPL/Second

311,656
Bytes/Second

51,888 Transactions


8080 Sessions/HR

22 UPL/Second

480,824
Bytes/Second

53,780 Transactions

8212 Sessions/HR

22 UPL/Second

488,168
Bytes/Second

54,049 Transactions


10455 Sessions/HR

25 UPL/Second

544,673
Bytes/Second

59,239 Transactions

(2 Nodes)

R2 (Workload of 240
Possible Concurrent
Simulations Learning
System/Community
System)


12459 Sessions/HR

31 UPL/Second

640,958
Bytes/Second

87,433 Transactions

13341 Sessions/HR

34 UPL/Second

729,616
Bytes/Second

90,353 Transactions

14913 Sessions/HR

33 UPL/Second

695,319
Bytes/Second

94,181 Transactions

16063 Sessions/HR

45 UPL/Second

968,128
Bytes/Second

106,659
Transactions

(4
-
Nodes)

R3 (Workload of 360
Possible Concurrent
Simulations Learning
System/Community
System)


17288 Sessions/HR

42 UPL/Second

901,103
Bytes/Second

118,754 Transactions

18455 Sessions/HR

50 UPL/Second

1,102,667
Bytes/Second

130,811 Transactions


20343 Sessions/HR

51 UPL/Second

1,145,440
Bytes/Second

145,287 Transactions

24034 Sessions/HR

65 UPL/Second

1,329,037
Bytes/Second

157,629
Transactions

(6
-
Nodes)

25

Example Benchmark Statistics

Workload

R7.1 Entry
-
Level

R7.1 Mid
-
Level

R7.1 High
-
Level

R7.1 HL
Clustered

R7 (Workload of 200
Possible Concurrent
Simulations Full
Academic Suite)

5721 Sessions/Hr

13 UPL/Second

275,672
Bytes/Second

37,313 Transactions


12548 Sessions/Hr

33 UPL/Second

728,082
Bytes/Second

84,004 Transactions

12974 Sessions/Hr

35 UPL/Second

735,846
Bytes/Second

84,970 Transactions

13804 Sessions/HR

36 UPL/Second

763,955
Bytes/Second

90,941 Transactions

(4 Nodes)

R8 (Workload of 400
Possible Concurrent
Simulations Full
Academic Suite)


11908 Sessions/Hr

34 UPL/Second

668,189
Bytes/Second

77,553 Transactions

18857 Sessions/Hr

53 UPL/Second

1,157,486
Bytes/Second

118,353 Transactions


14668 Sessions/Hr

32 UPL/Second

676,802
Bytes/Second

96,742 Transactions

(2 Nodes)

24034 Sessions/Hr

65 UPL/Second

1,392,037 Bytes/Sec

157,629
Transactions

(4 Nodes)

R9 (Workload of 600
Possible Concurrent
Simulations Full
Academic Suite)

12652 Sessions/Hr

25 UPL/Second

451,975
Bytes/Second

64,289 Transactions

23056 Sessions/Hr

63 UPL/Second

1,196,553 Bytes/Sec

149,709 Transactions

20207 Sessions/Hr

47 UPL/Second

1,014,189
Bytes/Sec

130,907
Transactions

(3 Nodes)

27997 Sessions/Hr

71 UPL/Second

1,527,433 Bytes/Sec

181,121
Transactions

(6 Nodes)


26

References

»
Tomcat: The Definitive Guide
by Jason Brittain


»
http://www.onjava.com/pub/a/onjava/2004/01/28/tomcat5.html?CMP=I
LC
-
VP9761447835&ATT=804


»
http://tomcat.apache.org/tomcat
-
5.0
-
doc/balancer
-
howto.html


»
http://www.onjava.com/pub/a/onjava/2002/07/17/tomcluster.html?page
=1

»
http://www.onjava.com/pub/a/onjava/2004/03/31/clustering.html

»
http://www.onjava.com/pub/a/onjava/2004/04/14/clustering.html

»
http://arstechnica.com/articles/paedia/cpu/hyperthreading.ars

»
http://lists.samba.org/archive/samba
-
technical/2004
-
December/038305.html

»
http://tomcat.apache.org/tomcat
-
5.0
-
doc/cluster
-
howto.html



27

Past Presentations of Note

»
B2 2006:
How We Size the Academic Suite, Benchmarking at Blackboard


»
B2 2006:
Deploying Tomcat Clusters in an Advanced Blackboard Environment



»
2006 BbWorld Presentation:
Practical Guide to

Performance Tuning and Scaling

(2 Hour Workshop)


»
B2 2005:
Introduction to Load Testing, A Blackboard Primer


»
B2 2005:
Performance Testing Building Blocks


»
Users Conference 2005:
Managing Your Blackboard Deployment for Growth and
Performance


»
Users Conference 2005:
Applied Software Performance Engineering


»
B2 2004:
Introduction to Software Performance Engineering


»
B2 2004:
Profiling Building Blocks for Performance Analysis

28

Questions?