Exception Extractor Utility

greenbeansneedlesΛογισμικό & κατασκευή λογ/κού

13 Δεκ 2013 (πριν από 3 χρόνια και 5 μήνες)

98 εμφανίσεις

Exception Extractor Utility

(version 0.2)

A script to collect all Unique Java
Exception in logs.

Utility Feature


Scans all the logs at defined location and extracts ONLY the unique exceptions.


Can be set to auto
-
fire mode, where the script keeps on firing itself to collect more unique
exceptions in logs, after configurable time.


Can be set to collect certain exceptions only (e.g, NullPointerException, SQLException etc.)


Can be set to skip certain exceptions (e.g, MissingResourceException, BusinessException etc.)


Cab be configured to catch certain word
-
list only (having a common prefix). For example, if
CATCH_ONLY_WORD prefix is “STORE:” and list is set to “02819”, “00021”,”05607” then only
exceptions with string “STORE:02819”, “STORE: 00021”, “STORE: 05607” can be collected,
rest ‘STOREs’ will be ignored.


Can be set to scan a location at remote server or a location at local server itself.


Can be easily integrated with the server
-
startup script or can be deployed as an individual
utility.


Can be set for defined begin and end log tag. For example, begin log tag is set by default as
“BL:” and end log tag is set by default as “:EL”. Also an error Expression can be set like “LL:
ERROR” which is present in all error related logs.


Can be configured to delete temporary output files and save space.


Technical Specification


Utility is simple perl script (named exceptionExtractor.pl) wrapped under a bash based shell
script (named exceptionController.sh).


Utility (or perl script) runs as daemon process when executed.


Utility (or perl script) has a property file (named variables.txt). The normal text file is actually
a part of perl script, in which all the necessary initializations has been done (in perl way). Any
changes in configuration can be easily done in this file.


Utility require perl (>v5) at

/usr/bin/perl


Utility Controller require bash at
/bin/bash


Location of perl and bash can be replaced at 1
st

line of the script
. The script has been tested
with ksh also.

Modes of Utility


The script can be executed in following modes



Modes based on location of deployment of the script:


Local Mode
(preferred) : utility is deployed at local server


Server Mode
: utility is deployed at remote location.



Modes based on scheduling the script:


Auto
-
Firing mode
(preferred): utility keeps on firing itself after
configurable amount of time.


Fire
-
At
-
Will mode
: utility is fired, it collects unique exceptions and
stops.

Note: Utility does not support Server Mode and Auto
-
Firing Mode simultaneously.

Pre constraints and terminologies


Unique Exception
refers to unique stack trace in logs. It is possible that same exception at
class A is caught twice, if it has been called by Class B and Class C (which creates separate
stack traces).
It does not refer to catching Exception X only once, all depends on unique stack
-
trace.


A log is expected to contain a
BEGIN TAG
and

END TAG
(like BL: and :EL respectively)
otherwise the utility will not be able to separate two different logs. (EOL character will not be
helpful)


A path to log files has been broken into three parts:




$path_base $path_domain, $path_applog


Final path will be
$path_base/$path_domain/$path_applog


Example:

$path_base = “/usr/projects/allDomains”




$path_domain=“/Domain_Testing_A”




$path_applog=“/logs”


The above ‘variables’ implies logs are at path
usr/projects/allDomains/Domain_Testing_A/logs


The ‘variables’ related to path can be configured in
variables.txt
file along with deployments.


A variable file is a small text file with
perl
-
style variable initializations.
Attached XLS file below has all the variables and explanation.



The Variable file is loaded inside main utility
perl

script and read as it
is. Any
perl

syntax error in variable file will cause script to fail.

*Each variable defined, must end with semi
-
colon.

*Each String variable, must have “ surrounding the value.

*A variable start with $ while an array variable starts with @

Setting the variables.txt should ideally be ONE
-
TIME
-
ACTIVITY.


Understanding variable file

Local Deployment Mode


To deploy script locally (local
-
mode):


Adjust following variables (in variables.txt)


$deployedLocal=1;


$path_applog="/pathToApplicationLogs“;


$path_base and $path_domain is not used when deployed locally.


Place the utility at a desired local location ($path_applog must be path
wrt this location)


Server Deployment Mode


To deploy script at remote location (server
-
mode):


Adjust following variables (in variables.txt)


$
deployedLocal
=0;


$
path_applog
, $
path_base

and $
path_domain

SHOULD BE SET.


$server=“
xx.yy.zz.aa
”;


$user=“
userHavingAccessTo$server
”;


$
tempFolder
=“/
localTempFolderAtDeploymentLocation
”;


Place the utility at a desired remote location ($path must be path from /)



Auto
-
Firing Mode


This mode can only be used when
deployedLocal

is 1.


Adjust following variables:


$scheduler=1;


$timer=
IntegerValueOfNumberOfMinutes

(
n
);


The utility will process exceptions from log files after around
n

minutes. The $timer (
n

here) is approximate value/hint
to sleep. The daemon gets active based on kernel.

When $scheduler is 0, the Mode is Fire
-
At
-
Will, In this case
the utility executes, processes, collects unique exceptions
and STOPS.

Note
: DO NOT FORGET TO STOP/KILL THE UTILITY, IF SCHEDULER IS TURNED ON. THE PROCESS UNDER
EXECUTION IS CPU INCENTIVE.


(for LOCAL
-
MODE) Once the script is placed at desired
location and variables.txt is set.
exceptionController.sh

can be
used to START/STOP the script.







If SERVER MODE is used, please use following command to
execute after setting variables.txt

>
./extractException.pl

Do not use shell script in SERVER MODE, the script runs in nohup mode and
SERVER MODE is interactive.

Utility Execution

Sample Deployment

In the following image


*Consider
amanTest

to be a
Domain

where the script is also placed as shown.


*Deployment has 3 components pl, sh and txt file.


*$path_applog = “/applog” has been set (since path to application logs locally wrt to
pwd is /applog


*uniqueOutput.log is the output file created after execution of the utility (described
previously)

Important Points


STOP the process (using Controller) from same location where START was initiated. The
process kills
`pwd`/exceptionExtractor.pl
OR kill
-
9 the process ID.


Perl script runs in noHUP mode. Disconnecting session does not disconnect process. It has to
be killed/stopped.


The utility at each fire, creates a temporary output_xx
-
xx
-
xx_yy:yy:yy ($outFile)file, which is
a collection of ALL the exceptions (stacktraces) out of which only unique is appended to
uniqueFile ($outFile2). Deleting old output_xxxxxx ($outFile) files does not hamper the
process and saves the space
(This can be configured through $deleteOldOutputFiles=1;)


Perl is a loosely coupled language, the data type of variable is depended on its value. In order
to make an array (like @skipException) empty, assign it to empty list or list having a value
which is blank i.e,
@skipException = (“”);
In this way the skipException contains a value
which is BLANK Hence, utility will not skip any exceptions.


$use_filter = 0 does not take care of grepExceptions , skipException, catchOnlyWordPrefix
and catchOnlyWordList variables. But it has to be free of any syntax error.


DO NOT Start the script using Controller when SERVER
-
MODE is used. The script runs in
nohup/background and SERVER

MODE is interactive (asks for password to access
targetServer)

Attachments:

Revision History


0.1


Initial Beta version


0.2


Added feature to optionally delete old output files and save space.





Added additional variable to control ERROR Expression in all the

ERROR logs. (“” is an accepted value)



Problems in deployments.

Problem in execution.

Advice/Suggestion.




Please contact
avD.Zeek@gmail.com




THANKYOU