Wideload In Perspective Documentation

moneygascityInternet και Εφαρμογές Web

8 Δεκ 2013 (πριν από 2 χρόνια και 10 μήνες)

56 εμφανίσεις

Wideload In Perspective
Release 1.0
November 22,2012
1 The What And Why Of Wideload 3
1.1 Why Do I Need To Learn About Perforamance?............................3
1.2 But I Already Know Everything!....................................3
1.3 So How Do I Get Started?........................................4
1.4 Overview.................................................4
1.5 So How Do I Use It?...........................................5
1.6 What To Expect FromThe Rest Of This Book.............................5
2 Installation 7
2.1 Overview.................................................7
2.2 What Should I Be Using?........................................7
2.3 Installing Wideload............................................8
2.4 Conclusion................................................8
3 Booting,Provisioning And Validating The VM 9
3.1 Overview.................................................9
3.2 Booting And Provisioning........................................9
3.3 Validation.................................................10
3.4 Troubleshooting.............................................11
4 First Test Run 13
4.1 Overview.................................................13
4.2 Executing Our First Test.........................................13
4.3 Conclusion................................................16
5 Analyzing The Results Of Our First Test Run 17
5.1 Outline..................................................17
5.2 Overviwe.................................................17
5.3 Analyzing The Results..........................................17
6 Appendix B - Enabling The Root User On Ubuntu 19
6.1 Overview.................................................19
6.2 References................................................19
7 License 21
Wideload In Perspective Documentation,Release 1.0
This document is a set of tutorials for the Wideload tool set.
Note:Please note that this book is a draft document that is still in a very early stage.It is incomplete and under
constant revision.
Wideload In Perspective Documentation,Release 1.0
1.1 Why Do I Need To Learn About Perforamance?
Note:For the remainder of this book,I’mgoing to use the termtechnologist to refer to anyone who builds,supports,
or tests information systems.This includes prorammers,sysadmins,and testers.
Before I start telling you about Wideload,it helps to to understand why performance tuning and optimization are such
a big deal in the first place.
1.1.1 It Is Your Single Most Important Feature
If you have ever worked on a public-facing web site,this information is old news.Simply put,no one will use your
web site if they think it’s slow.There’s far too much competition and next to no costs to switch.
Note:Howslowis too slow?You may be surprised to learn that even a half-second delay
can cause users to abandon
your site for good.
Even if you work in a corporate environment with more forgiving (and locked-in) users,they still need reliable,
responsive systems.Treating your internal users like they don’t matter will make them hate you (at best) and replace
you (at worst).
1.1.2 It Helps You Level Up As A Technologist
As you achieve a higher level of mastery as a technologist,you need to be able to understand your systems at both
higher and lower levels.Studying performance and optimization will give you a better understanding of exactly why
your system behaves the way that it does,both at micro (e.g.request queues,threads) and macro (e.g.networking)
1.2 But I Already Know Everything!
Believe it or not,most technologists (including myself for a while) actually believe that there isn’t much to learn about
performance.Sysadmins who believe this only think that you need to learn how to use top and perfmon.If the
systemis slow,add more RAM.
Wideload In Perspective Documentation,Release 1.0
Programmers who have this opinion try to make their code “fast”.They don’t always have profilers,so instead they
try to optimize blindly.They say things like “real programmers always use primitives” and “Java is slow”.
I soon learned though that there is much,much more to performance adding RAMand misguided optimization.For
example,to troubleshoot the performance of a modern,J2EE-based system,you should knowhowto do the following:
• Read a thread dump
• Query a JVMusing a tool such as JVisualVM
• Determine how many times a day your JVMis performing a “major” garbage collection and why.
Things don’t get much easier if you choose not to use Java.If you administer a LAMP-based application,then you
should know how to do the following:
• Determine if your web app is saturating your database server’s request queues
• Determine if your database disks are fast enough to support your peak load
• Report on throughput in near real-time.
And this is just the tip of the iceberg!It also helps a bunch to understand things like basic statistics and networking.
1.3 So How Do I Get Started?
There are some good books on this subject
,but they all share a problem with applicability.Unlike a programming
book that can step you through building a newRails programor a Linux book that can step you through an installation,
it is nearly impossible to set up a realistic “performance” sandbox.This is true for a lot of reasons:
• Most performance testing software is proprietary and expensive.It’s not reasonable to expect someone to
install it on their computer.
• A real systemwould include wide range of components,including the following:
– An application
– A performance load-runner
– Software to collect metrics
• It would take far too long to install all and configure all of these components.
To compensate,these books give advice that you can apply to a wide variety of applications.This is certainly better
than nothing,but I prefer hands-on training.I didn’t learn to create a Rails application by reading a book about generic
web programming - I installed all of the necessary components and created one fromscratch.
I then decided to create a “sandbox system” that I could use to learn about performance testing by doing.My goal
was to create something that was easy to setup,free and easy to use,and functionally complete.That idea eventually
became Wideload.
1.4 Overview
Wideload is a complete system that you can use learn about measuring and optimizing the performance of a web
application.At a high level,it is comprised of the following components:
• A Vagrant-powered VirtualBox VM
• A collection of Puppet modules that install and configure software on that VM.
I highly recommend Pro Java EE 5 Performance Management and Optimization if you are tuning a J2EE-based system.Regardless of
your underlying platform,I also recommend Release It!.
4 Chapter 1.The What And Why Of Wideload
Wideload In Perspective Documentation,Release 1.0
• A project skeleton for a load test that uses The Grinder.
• Convenience scripts that tie everything together.
The VMcontains numerous applications that work together to run an application and record metrics.Here are some
of the more important components:
• A instance of the Jtrac web application running on top of Tomcat and Mysql.
• An instance of Graphite,which displays application performance metrics in a very powerful and simple way.
• collectd and jmxtrans to record the metrics and feed theminto Graphite.
• A few other tools to help make running the systemsimpler,like monit,postfix,and mutt.
The Jtrac application on the VM is designed to be tested by an instance of The Grinder,which is a powerful and
easy-to-use load-testing application.Wideload comes with a simple load-testing script so you can get up-and-running
in a hurry,along with some helpers scripts that make it easier to create your own.
1.5 So How Do I Use It?
The first thing you should do is run the run the example load test,which I’ll expand on in a later chapter
.This will
let you know if you are missing any prerequisites or of there are any other issues.
Once that’s done,what you get out of this project really is up to you!Here are some example scenarios:
• Saturate The SystemAnd Find The Cause:You can write a load test that causes the systemto crash and then
find the root cause using Graphite.
• Experiment With Different Software:How does the throughput of your application change when you add a
web server like Apache?Does it get any better if you use Nginx instead?What about Postgresql instead of
I guess you can think of Wideload as a chemistry set for technologist.All you need are an idea an some curiosity.
1.6 What To Expect FromThe Rest Of This Book
This book is a set of tutorials to help you learn how to use Wideload.It is written by the same person who created
Wideload (Tom Purl) and it is designed to be more verbose than a man page but less verbose than most technical
The book is free and always will be,and you can download a copy from http://www.wideloadperf.org.You are also
free to distribute the book yourself as long as you don’t sell the book and you provide attribution.I will even eventually
provide PDF and Epub versions of the book on this site for free so you can read it on an e-reader or print it out.
If you have an comments or questions,please feel free to either open a ticket on the Github issues page or send an
email message to me.
All right,that should be enough of an overview.Let’s install Wideload and take it for a test drive.
Or you can get the “quick start” version fromthe project’s README file.
1.5.So How Do I Use It?5
Wideload In Perspective Documentation,Release 1.0
6 Chapter 1.The What And Why Of Wideload
2.1 Overview
This chapter provides the following:
• A very high-level overview of installing the prerequisite packages.
• A more in-depth set of instructions for installing the actual Wideload project.
I wanted to provide a bunch of docs on installing all of the prerequisites,but that is a very big task that changes every
six months.Instead,I will rely on the internet to provide documentation that is probably superior anyways:-).
2.2 What Should I Be Using?
Here’s the list of prereq’s and the versions that I’musing:
The Grinder
This project uses a lot of other software also,but that is all contained within the VMso we don’t have to worry about
installing it manually.
Here are some notes on my choices:
• In addition to VirtulBox,you also need to install the Extension Pack that goes along with your exact version.
Without this,a lot of features may not work correctly.
• I developed this VM on an Ubuntu 12.10 system.Everything should still work on a Mac or Windows-based
• I use the version of the ruby and vagrant packages that are installed by the Ubuntu package manager.They’re
a little behind the curve,but they work pretty well and this is a much simpler option than installing fromsource.
• I also use the Ubuntu version of VirtualBox.If you are installing a binary fromthe web site,you should probably
install version 4.1.16 because that is the same version that was used to create our vagrant “box”.
• You need a Jython runtime to run the “bundled” version of Grinder Analyzer.The 2.5.2 version of Jython is
a little old,but you can’t use a newer version because they’re not yet supported by Grinder Analyzer (as of
Wideload In Perspective Documentation,Release 1.0
We can’t move forward until you install these prereq’s,so please stop until that task is complete.
2.3 Installing Wideload
Ok,now that the prereq’s are installed let’s install Wideload.
For starters,grab the code fromthe following location:
• https://github.com/tompurl/wideload
You can either use the git tool to download the code or simply download a zip file,but it’s much easier if you actually
use git.I say that because this project contains something called git submodules,and it’s much easier to reconcile
those using using git than it is to download themmanually.
The good news is that git is cross-platform,and you’ll only have to use it a little bit.Once you have it installed,
simply run this command:
git clone git://github.com/tompurl/wideload.git
Next,run the following commands to take care of the dependency submodules:
cd wideload
git submodule init
git submodule update
To run the VM,you will need a “pre-baked” version of an Ubuntu 12.04 server VMthat has all of the Vagrant goodness
already installed.The Vagrant community calls these VM’s boxes,and the next thing we need to do is download the
box that we need:
vagrant box add precise32 http://files.vagrantup.com/precise32.box
Don’t worry,you only have to execute this step once.After that,this box is stored in a local cache.
2.4 Conclusion
You should now have all of the prerequisites and code that you need to start provisioning your VMusing Vagrant.In
the next chapter,we will start your VM,watch it build itself,and then use the built-in tools to ensure that everything
is running properly.
8 Chapter 2.Installation
3.1 Overview
We’re finally getting to the good stuff - booting and provisioning our VM.At the end of this chapter we should have a
fully-functional VMthat you can start to poke and prod.
This chapter will also devote a lot of space to validating the base functionality of your VM.This may sound like
drudgery,but luckily Wideload has a lot of software built-in that makes validation very easy.In fact,the VM is
designed to be created,provisioned,and validated in less than 30 minutes.Ideally,your only bottleneck should be
the number of free processor cycles on your machine.
3.2 Booting And Provisioning
Nowall of our work so far finally pays off with a fewvery simple commands.Simply navigate to your base wideload
directory using the command line and execute the following commands:
cd $WIDELOAD_HOME/server
vagrant up 2>&1 | tee./vu.out
So what did you just do?Well,it will take a little while to finish,so while you’re waiting you can read the following.
The first thing that happens is that vagrant will make a copy of the “box” that we downloaded earlier.This only
happens the first time you start a VM.And remember,this “box” is just a stripped-down Ubuntu VMthat includes the
bare essential stuff that vagrant needs to operate.
Next,vagrant will boot that box using VirtualBox.The VMis headless,meaning that you will not see a boot window
while the OS boots.It all happens in the background and it monitored by vagrant.
Note:This is actually one of the really cool parts about Vagrant.It gives you the ability to control and monitor you
VM’s fromthe command line.
After the VMboots,the really cool part begins.Vagrant will then start executing the Puppet recipes.These “recipes”
are scripts that install and configure all of the software that we will need on our VM.There is a lot going on here,and
it should take a little while to run.For example,on my quad-core system with 4 GB or RAM,it takes at around 11
minutes to download and install everything.
Wideload In Perspective Documentation,Release 1.0
Note:You should only have to install and configure this stuff the first time that you start your VM.After that,Puppet
is smart enough to only make changes if something has changed.
You’ll see a lot of output while you’re waiting.For now,don’t worry about that.We’ll only look at it if we have
problems with validation.
3.3 Validation
Nowthat the VMis booted and provisioned,we have one more step to complete before we can start running tests - we
need to confirm that everything important is working by visiting the VM’s Monit page.You can get there by visiting
this url:
• http://localhost:2813
• User Name:admin
• Password:monit
You should now see something that looks like this:
Figure 3.1:A Monit screen reflecting a working system.
10 Chapter 3.Booting,Provisioning And Validating The VM
Wideload In Perspective Documentation,Release 1.0
You are now looking at an instance of monit running you your VM.This tool monitors select system processes and
URL’s for uptime.Ideally,everything on this page should be green.
Tip:Monit is great for giving you a quick dashboard of your system’s health.If you are having any issues with the
VM,make sure to check the Monit page first to see if everything is running.
Chances are that your version of this site will not look exactly the same at first glance.This is because it takes a few
minutes for Monit to “catch-up” after you first provision a system.If things look funny,come back in 5 minutes and
everything should look good.
You may be wondering howwe’re looking at a web site on your VMif the URL is pointing at your machine (localhost).
Vagrant sets up port-forwarding when you start your VM,which means that it opens ports on your local machine that
forward requests to ports on a different machine.In this case,port 2813 on your local machine is forwarding requests
to port 2812 on your VM.
3.4 Troubleshooting
If everything is working for you so far,then feel free to move on to the next chapter where we finally get to start
using some of the web applications that come with the VM.If you are having issues,then check out some of these
troubleshooting tips.
3.4.1 The vu.out File
You may remember that I asked you to send the output of the vagrant up command to a out file using the tee
command.That output file was called vu.out,and it should be in the following folder:
This file is a little difficult to read using a text editor because a lot of weird characters have been added.These
characters add color to the output when it is read in a terminal,but it makes the file very difficult to read in you favorite
editor.You should therefore use the more command to read the file:
more vu.out
This will show you the contents in an easy-to-read format.
Look for any errors in the file.If you are stumped,open an issue for assistance.
3.4.2 SSH
Another good resource for troubleshooting is the ssh interface to your VM.You can access it by using the following
cd $WIDELOAD_HOME/server
vagrant ssh
These commands will drop you into the ssh shell for your VM.You can then inspect the system from the command
line.If you need to do anything as root,then simply prepend the command with sudo.No password is ever necessary
to use this account.
3.4.Troubleshooting 11
Wideload In Perspective Documentation,Release 1.0
3.4.3 The Nuclear Option
There’s a lot of moving parts in this VM,and sometimes it’s nice to be able to have a do-over.Well,Vagrant makes
this easy using the destroy target.
For example,let’s say that something is borked and you just want to start over with this VM.Simply execute these
#shut down the VM
cd $WIDELOAD_HOME/server
vagrant ssh
sudo shutdown -h now
#destroy the VM
vagrant destroy
Warning:The destroy command wipes out your VMand all manual changes that you made to it.Please use
with caution!
Now you can recreate the VMby simply executing vagrant up again.
3.4.4 If All Else Fails
If you are having issues,please feel free to open an issue.I’mlimited to being able to test this code on a Linux system,
so I am sure there are plenty of issues and challenges with other platforms.Also,I am always happy to review pull
12 Chapter 3.Booting,Provisioning And Validating The VM
4.1 Overview
In this chapter,we’re finally going to actually use Wideload to do something.We’re going run a simple load test and
the view the results,first using GrinderAnalyzer and then using Graphite.Along the way we’ll learn a bit about all of
the components and how they fit together so that you can start running your own tests.
4.2 Executing Our First Test
4.2.1 Brief Overview
This test has a lot of moving parts,and I don’t want to scare you away with all of the details yet,so here’s a quick
rundown of what we’re about to do:
1.Start our VM
2.Start the Grinder console
3.Start the Grinder agent
4.Start your load test
The load test will execute a script that does the following:
1.Visit the home page for the instance of Jtrac that is running in your VM.
2.Search for all open tickets in the “BUGS” workspace.
Our load test will simulate 10 users performing these tasks simultaneously.
The load test will then execute until we manually stop it.Once we complete it,we will viewthe results in the following
1.First,we will view a report that is generated using the information collected by the load runner.
2.Also,we will look at lower-level performance metrics that were recorded within the VM.
4.2.2 Setup
Assuming that you already installed The Grinder and Jython using the Installation Chapter,the only remaining step
is to update the following file:
• $WIDEDLOAD_BASE/loadgen/bin/setGrinderEnv.sh
Wideload In Perspective Documentation,Release 1.0
This is the script that is called by all of the other shell scripts to obtain environment variables.It should contain all of
the information that the shell scripts need to run.
Change the following variables:
• GRINDERPATH - This should be the “root” path of your installation of The Grinder.
• JYTHON_HOME - This is the “root” path of your Jython installation.
All of the other variables should remain static for most installations.They should only be changed if you have a very
good reason.
Note:Some of the variables in this script use relative paths that start with ”./”.This path is relative to the location
of the loadgen directory,not the loadgen/bin directory.This means that these shell scripts should be executed
using the loadgen directory as your PWD,not the bin directory.
4.2.3 Execution Instructions
Starting The VM
First things first,we need to make sure that your VMis running.You can do this by executing the following commands:
cd $WIDELOAD_ROOT/server
vagrant status
You should see a result that looks a lot like this:
Current VM states:
default running
If the VMis not running,then you can start it like this:
cd $WIDELOAD_ROOT/server
vagrant up
Once it’s up,you can test the health of the VMby visiting the built-in Monit page.For more information on using the
Monit page,please see the Validation section in the Booting,Provisioning And Validating The VM chapter.
Starting The Grinder
Once your VMis up,we can start our test.First,let’s open the Grinder console using the following commands:
cd $WIDLOAD_ROOT/loadgen
You should now see a window that looks a lot like this:
What you are nowlooking at is the tool that is used to coordinate load tests and present the results.However,we can’t
do much with it yet,so let’s start our agent using the following commands:
cd $WIDLOAD_ROOT/loadgen
This starts our load agent,which is the component that actually generates the load on our web app.In a more complex
test,we could run multiple agents across many machines,but our simple load test only requires one.
14 Chapter 4.First Test Run
Wideload In Perspective Documentation,Release 1.0
Figure 4.1:The Grinder Console
After executing the commands above,you should see two things.First,you should see command output that looks
something like this:
2012-11-20 23:22:37,741 INFO agent:The Grinder 3.10
2012-11-20 23:22:37,830 INFO agent:connected to console at localhost/
2012-11-20 23:22:37,830 INFO agent:waiting for console signal
Also,you should now see something on the Processes tab in the Grinder console that looks something like this:
As you can see,the console now recognizes one connected agent.Now that we have something that can generate a
load,we can run a test.
Running The Load Test
Now we get to push the big red button that starts the load test.In the Grinder console window,click Action -> Start
Processes.You should now see a pop-up screen saying that you have not selected a properties file.Our properties file
is defined in a different place,so you can safely click on the OKbutton.
The load test is now running.You should notice a few changes in the GUI:
• On the Processes tab,you should now see a row corresponding to a worker.This row will also show you how
many threads (i.e.simulates users) are running
• Regardless of your tab,you should see a TPS (transactions per second) and Collected Samples amounts on the
left side of the screen.
• The Results tab should show you some summary statistics.
• The Graphs tab should show you some higher-level statistics along with some simple graphs.
4.2.Executing Our First Test 15
Wideload In Perspective Documentation,Release 1.0
Figure 4.2:List Of Connected Agents
Now is a good time to poke around and see everything that the Grinder Console GUI provides.Once you are done,
however,you also need to manually stop the test.
Stopping The Test
This load test is configured to run forever.This is nice because it gives you a lot of flexibility,but it also forces you to
remember to shut it down.Here’s how you do that:
1.In the Grinder Console choose Action -> Stop Processes.
2.You should see a dialog box asking is you’re sure.Please click on th OKbutton.
3.Select File -> Exit.
Note:It is a bad idea to forget to stop this test.Eventually,The Grinder will fill up your partition with log messages
if you let it run long enough.
4.3 Conclusion
Congratulations!You just finished executing a load test.However,there’s more to performance optimization that
running a test.You also need to actually look at the results and determine what (if anything) needs to change.In the
next chapter,we will performjust such an analysis on the test we just ran.
16 Chapter 4.First Test Run
Warning:This chapter is not yet complete.
5.1 Outline
• Overview
• Analyzing The Results
– Viewing The Report
– Viewing Graphite
5.2 Overviwe
5.3 Analyzing The Results
While your test was running,lots of statistics were being collected,both by The Grinder and by some applications
running in your VM.Let’s take a look at the results that were generated by these systems.
5.3.1 HTTP Results
First,let’s look at the results of the HTTP test that were recorded by The Grinder.By default,The Grinder stores its
results in the following location:
• $WIDELOAD_ROOT/loadgen/log
These log files contain lots of great information,but they’re a bit hard to decipher by themselves.We are therefore
going to use a different tool to generate a nice HTML report - GrinderAnalyzer.
To generate the report,execute these commands:
Wideload In Perspective Documentation,Release 1.0
cd $WIDEDLOAD_ROOT/loadgen
You should see a lot of output,and the final line should look like this:
Log file analysis completed successfully.
This script will publish the report in the following location:
• $WIDELOAD_ROOT/loadgen/GrinderAnalyzer.V2.b18/grinderReport/
To view the report,you can run the viewReport.sh script,which simply opens the report.html file in the
directory above using Firefox.
Your report should look something like this:
Figure 5.1:Section Of A GrinderAnalyzer Report
Explaining all of the different sections of this report goes beyond the scope of this book.However,here are some nice
features that you can use to get more information about your test:
• Each link will show you a graph if you hover over it
• You can sort on each column by clicking on it
• All of the response times are in milliseconds
18 Chapter 5.Analyzing The Results Of Our First Test Run
6.1 Overview
6.2 References
Wideload In Perspective Documentation,Release 1.0
20 Chapter 6.Appendix B - Enabling The Root User On Ubuntu
Wideload In Perspective by Tom Purl is licensed under a Creative Commons Attribution-NonCommercial-
ShareAlike 3.0 Unported License.