Stuff You Can Get For Free

cabbagewheatInternet and Web Development

Nov 13, 2013 (3 years and 7 months ago)

44 views

Java Monitoring


Stuff You Can Get For Free
(And Stuff You Can’t)


David Eastwood

Solutions Engineer

deastwood
@
appdynamics.com

A Bit About Me


Current: Solutions Engineer


Past: Programmer/Consultant/SE


Prime
Computer, Sybase, BMC, CA/Wily


Hobbies


Music, Guitar, Photography


2


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

Why should I Pay for a Monitoring Tool ?

The
c
hanging Application Landscape

Try out the Free Monitoring Tools

How people are benefitting from APM?

3

4

2

Today’s Agenda

5

3



Copyright © AppDynamics. A
ll rights reserved.

Let’s debug a
Slowwww

Application !

1

App is running
Slowwww

… AND …


It’s running in Production


Can’t restart the servers



Can’t reproduce in non prod environment


Data sets and load are not alike



Just released a new version and it includes


several refactoring, a new deployment architecture



Revenue critical app, can’t
make it any slower by


overloading
it with intrusive
monitors,
additional logging



Development team is at a different location and time zone



4


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

Home grown
-

Logs, JMX Dashboards etc.


When it works


Trusted Data, first level of defense


Often added based on learning from outages and critical infrastructure
components (pools, caches)


Means to aggregate across multiple log files


Systemic issues, e.g. recurring error / warn messages



When
it
doesn’t !


Sheer volume, don’t know where to start, after all logs are flat


The issue is triggered by specific data sets


Logging libraries are often synchronous, and heavy on string
operations, hence can lead to
perf

issue


What if you forgot to add “
isDebugEnabled
()”


Time and Server correlation is not easy


Server Time vs. Actual Outage time

5


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

Let’s try some free tools


JConsole



ships with all JVMs



Visual VM


Exclusive to Oracle (Sun) JVM



Eclipse Memory Analyzer


Heap Dump Analysis







The

Game

C
hanger



AppDynamics

Lite

2
.
0





6


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

Let’s go classic !
-

jconsole

JVM Health













7


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

Let’s go classic !
-

jconsole

Thread Dumps






Mbean

Browser




8


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

Let’s go classic !
-

jconsole

Mbean

Browser




9


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

JConsole



Summary


When it Works


JVM needs tuning


Get metrics for Memory, CPU, Threads and Garbage Collection


Experiment with Memory Pools and GC Algorithms


Systemic slowness, thread dumps point at areas


Server
Mbean

and Custom JMX provides immediate clues


When it doesn’t !


Metrics have no contextual information so no clue why thread is
stuck on a socket !


You don’t know / own the source code and
d
ev

is not next to you


It’s not a JVM tuning issue ?


Passive, always have to be on watch !


Overhead !



10


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

New Age
-

VisualVM

How are my threads doing ?










11


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

New Age
-

VisualVM

Where are my Threads spending their time ?









A Thread dump to show what’s behind the socket read









12


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

New Age
-

VisualVM

What are my Threads
m
unching on?








My App Classes



13


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

Visual VM


Summary


When it Works


JVM Metrics overview :
Memory,
CPU, Threads, GC etc.


Thread Summary View


Wait, Sleep, Monitor, Run Time


Sampling provides some initial clues to the problems


Thread Sampling


Where time is spent most often


Memory Sampling


What objects are created most often


When it doesn’t !


All the drawbacks of
jconsole


The profiler option (instruments all classes and methods) has way
too much overhead even for trivial apps, so not an option


Ships only for Sun (Oracle) JVM


JRockit

ships with a similar tool named JRA


Nothing notable for IBM JVMs for free





14


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

Out of Memory!
-

Eclipse Memory Analyzer

Leak Suspects






Retained Heap Table


15


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

Eclipse Memory Analyzer
-

Summary


When it works


You have a Heap Dump generated though

XX options


Open up Leak Suspect Report and


Check out the actual content of the Object (Map <Key, Value>)


Trace the outgoing and incoming references on the object (Who is
holding the map)


It happens often and you know why it happens (you have some
smart engineers working with you !)


When it doesn’t !


Knowing what’s big is less important, what you need is


How it got big ?


Who is adding / putting elements


Doesn’t happen all the time and you can’t associate with a specific
event, it grows in lock step, not linear !

16


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

AD LITE 2.0

The Game Changer

What’s different ?


Application flow map


Identifies key components


Servlets, Struts, EJBs


External calls


HTTP, RMI, JMS, JDBC, THRIFT etc.


Top Down View of Application


Recognizable artifacts e.g. URI vs. Class Name


Aggregated at the top, as experienced by real user


Dynamic Thresholds


Learns by itself what

s slow for Login vs. Checkout


Continuous visibility and Deep Dive when needed


Every click is monitored, capture outliers


Slow, Errors, Stalls


Your very own dashboard


App, BT, JMX Metrics


Always ON, alerts when things go wrong

Built for Production < 2% Overhead


18


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

What do you see 2 minutes after install?

19


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.


Home page to quickly assess application health


Visualize application dependencies and business transactions


Quickly drilldown into performance problems


1
st

Click


Business Transaction Dashboard

20


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.


Quickly identify which Business Transactions have problems


See summary performance metrics (load, response, errors, stalls)


Visual Cues (green/yellow/red) bar
-
pie accelerate troubleshooting


2
nd

Click: Troubleshoot Bad Requests

21


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.


Troubleshoot performance of a specific transaction (i.e. Add
-
to
-
Cart)


Drill into Bad Requests, Diagnostic Sessions and Sampled Snapshots


Start Diagnostic Sessions to get class/method level diagnostics


3
rd

Click: Diagnose Root Cause

22


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.


Deep Diagnostics for Slow Requests, Stalls and Errors


Everything you need to determine root cause


Slow Requests

Stalls

Errors


Code Level Diagnostics


Hot Spots


SQL Calls


External Calls




>30 seconds


Thread Stack


Exception Stack Trace


Logged Errors

What you see:

Architecture

AppServer

Agent

Viewer

UI

HTTP

Low footprint Agent
.


Same Agent


No configuration files


Single jar

Low footprint Viewer
.


Very small footprint

< 25 MB install


Jetty


No Database

Simple UI
.


Flex based


5 screens


Uses existing
data model

When does it make sense to Pay
for a Monitoring Tool?

If your app …


Deployed on more than 5 JVMs


Need to see data at one place, be able to navigate easily


Request processing is multi threaded, distributed


Need correlation across threads and services


Agile releases, Now vs. Then


Last 5
mins



Last Week


Last Quarter


Custom Data Capture, Slicing and dicing of the data


Capture Parameters, Select where
session_id

=‘’, All slow SQLs


Advanced Baselines and Policy Engine


Static / Dynamic / Seasonal Baselines


Making sense from the metrics deluge


Dashboards, Business Metrics


Pre built, custom widgets, live business metrics


Moving towards Cloud Architecture




25


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

The Commercial Landscape


The Big 5


IBM


CA


Compuware


HP


BMC



The New Age Players


AppDynamics


New Relic


26


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

AppDynamics Pro

Custom Data

Collection

API / Scripts

Custom Action API /
Scripts

Instrumentation Point
API

Business Metric
Collection API

Custom Header
Correlation API

Controller

Application
Server Agent
(JVM/CLR)

Machine
Agent

Alert
Integration
API / Scripts

Data Feed
API

Data
Collection
API

Cloud
Connector
API

Must haves for your Paid Tool


No professional
services


Plug and
Play, no experts needed to configure and tune the tool


You don’t tell the Tool, the Tools tells you what you got


Self adaptive learning


Always ON in production with very low overhead


Not a factor of how much traffic


Easy to learn the tool and extend too


Start with the basics and be an expert over time


Easy to share and collaborate across the Organization


Operation,
Dev
, QA, Performance, Architects


Total Cost of Ownership


include everything


Software, Hardware, Deployment, Learning, Maintenance


Continuous Innovation


Cloud, NO SQL,
Async

Transactions







28


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

How is APM Helping
Organizations ?


A nice problem to have...


http://
techblog.netflix.com
/2011/02/redesigning
-
netflix
-
api.html












A
“Blockbuster in the Cloud




World’s
largest Java public cloud
deployment

AppDynamics in Action: Netflix

37x

Growth

Jan

2010
-
Jan 2011

30


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

1ST CLICK
: THE APP TOPOLOGY

31


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

2ND

CLICK
:
REVIEW SLOW TRANSACTIONS

32


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

3RD CLICK
:
TRIAGE LOCATION

33


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

4TH CLICK
:
FIND THE LINE OF CODE

34


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.


E


Travel Site



Tomcat, Active MQ, My SQL



200 Business Transactions


AppDynamics in Action:
Karavel
, France

35


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

CUSTOM DASHBOARDS

36


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

DRILL DOWN ON SLOW TRANSACTIONS

37


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

FIND THE CODE AND THE BAD SQL


38


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

VALIDATE

THE FIX !

39


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.

FUTURE PROOF WITH ALERTS

40


November 13, 2013

|
Copyright © 2010 AppDynamics. A
ll rights reserved.