Memory issues in production systems

ninetimesdissemblingΛογισμικό & κατασκευή λογ/κού

10 Νοε 2012 (πριν από 4 χρόνια και 7 μήνες)

290 εμφανίσεις

Memory issues in
production systems

Production system


Restricted access


Application, DB, Application server, log files


Debugging, monitoring


Investigation should not affect users


Reproducing memory issue is not the best idea

Memory leaks


What is memory leak


java.lang.OutOfMemory: heap size


other symptoms
-

extensive garbage collection


Hard to discover


Users don't recognize it


Hard to investigate


Few information


Sometimes hard to fix


Design issues

Investigation of memory issues


Log files


application logs, access logs, gc logs


Monitoring behavior of running application


JConsole


Heap dump Analysis


JHAT, MAT, Netbeans

Log files


Motivation


find issues


find patterns leading to issues


Problems


combining different logs (access log, application
log)


bad visibility of root cause


not visible or hidden in
big amount of minor issues


memory problems trigger only few types of
messages

Investigating log files


Application logs


Log free memory once per minute


Access logs


Analysis of traffic and business use cases


GC logs


Show problems with GC (stop the world)

Monitoring via JMX


Tools


JConsole


Motivation


More information than in logs


Monitoring tests


Problems


Not always applicable on production system
(security reasons)


Test might not cover all scenarios

Heap dump analysis


Tools


JMAP, JHAT, MAT, Netbeans


Motivation


Offline investigation of behavior during specific time
frame


Comparison of different heap dumps


Problems


Access to heap dumps (security)

Creating Heap dump


jmap
-
dump:live,format=b,file=heap.dump <pid>


Part of JDK


Used to create heap dump on demand


JVM parameters


-
XX:+HeapDumpOnOutOfMemoryError
-
XX:HeapDumpPath=/directory/subdirectory


Produces file java_pid<pid>.hprof


Used for production system, creates heap dump when
OutOfMemory is thrown

Tools for heap dump analysis


JHAT


Part of JDK 6


Running as web application


IDE


Netbeans, MAT (Eclipse plugin)


Commercial tools

Heap dump analysis
-

JHAT


Heap Histogram


List of classes with number of instances and used
memory


Referrers vs Referees


Navigating in object graph


Example: Form → String → [c

Heap dump analysis


Netbeans,
MAT


Dominator tree


Object x dominates object y if every path in object
graph from root to y goes through x


Shallow vs Retained heap


Shallow heap is memory used by object


Retained set of X is set of objects which would be
removed by GC if X is removed by GC


Retained heap is sum of shallow heap in retained
set


Other queries


Leak suspects


Top consumers

Heap dump Analysis
-

OQL


Object query Language


Different dialect for each tool


JHAT


Simple SQL combined with javascript


callback functions


MAT


More SQL like (Subquery, union, distinct)


Predefined functions (dominator)

OQL Examples


JHAT


select
heap.findClass('org.apache.catalina.util.ServerInfo')
.statics.serverNumber.toString()


MAT


SELECT serverNumber.toString() FROM OBJECTS
org.apache.catalina.util.ServerInfo

Experiences


Bugs


Incorrect usage of external libraries or
frameworks


Design issues

Bugs

Map <String, String> redirectUrlCache = new LinkedHashMap<String, String>() {
@Override
protected boolean removeEldestEntry(Map.Entry eldest) {
return size() > 10000;
}
}

String getRedirectUrl(String url) {
if(redirectUrlCache.containsKey(url)) {
return redirectUrlCache.get(url);
}
String result = computeRedirectUrl(url);
redirectUrlCache.put(url, result);
return result;
}
Usage of frameworks


Hibernate


Short time usage of big amount of memory


Too big first level cache


Good understanding of framework is a must

Design Issues


Overused session scope


Big pages


Big buffer in page context


Page context referenced by tag handlers


Heavy load causes big memory usage


Fix might require changing use cases


Reusing functionality

Interesting links


http://blogs.sun.com/sundararajan/entry/querying_java_heap_with_oql


OQL help for JHAT


http://www.eclipse.org/mat/downloads.php


Standalone MAT


www.google.com