McFarmGraph a web based monitoring tool for McFarm jobs

UTA HEP/Computing/0026: McFarmGraph______________________________________________
1

McFarmGraph a web based monitoring
tool for McFarm jobs


Sankalp Jain
,
Aditya Nishandar
, Drew Meyer, Jae Yu, Mark Sosebee,

Prashant Bhamidipati, Heunu Kim, Karthik Gopalratnam,
Vijay

Murthi,

Parag

Mhashilkar







Abstract


McFarmGraph is a web based G
raphical User Interface used to monitor McFarm jobs on
Linux farms.
This document is intended to be a comprehensive description of the design
of McFarmGraph. For installation and administration please refer McFarmGraph
administration guide.




























UTA HEP/Computing/0026: McFarmGraph______________________________________________
2



1. INTRODUCTION:


McFarmGraph consists of two parts
–

the front
-
end CGI scripts that are used to display
the status of various jobs in a graphical format, and the back
-
end daemon that is used to
bring status files over to the web server from

the various farms.


Status Daemon

(
McFarmGraph_daemon
):

McFarm periodically (e.g. every four hours) outputs a status file on each farm
where it is running. This flat file summarizes the status of the various jobs
running in the farm. The update period can

be changed based on a parameter that
can be set by the farmer (see McFarm documentation). The purpose of the
daemon is to bring the status file from each remote farm that is doing productions
using McFarm. The daemon uses globus services (gsiftp) periodic
ally to transfer
the status files and stores them locally. The daemon then triggers the XML
generator, which converts the flat files to XML format. The flowchart illustrates
the control flow for McFarmGraph daemon.


Graphical User Interfaces:

This is a se
t of CGI scripts written in PERL, and some applets written in Java.
The scripts interpret the XML data representation of the status file transferred
from the remote farms and present them to the user in a graphical format that can
be accessed from the we
b.


Request Structure:

Each Mcfarm request consists of a bunch of jobs that are grouped together
according to the request id of type ReqXXXX. Individual job names consist of the
Request Id followed by a descriptor string and a number that is unique in the

group. For e.g. “
Req6279
-
zh
-
zmumu+hbb
-
03219094626”, “
Req6279
-
zh
-
zmumu+hbb
-
03219094710”,
Req6279
-
zh
-
zmumu+hbb
-
03219094921… all
belong to the group with request id
Req6279.
The “%Done” attribute displayed
on the webpage for a particular request id is the av
erage of %Done attribute of
individual jobs within that group. Also the figures displayed in the PieChart
represent the percentage of jobs that are in a particular phase.




2. STATUS DAEMON DESIGN:


Scalability and simplicity are the pivotal issues that
influence the design of the daemon as
well as the cgi scripts. Fig 1 shows the directory structure in which the job status files
from the remote farms are stored.





UTA HEP/Computing/0026: McFarmGraph______________________________________________
3





Fig 1. McFarmGraph Directory Structure on hepfm
007.uta.edu



The McFarmGraph job information as well as the configuration and log file is placed in
the directory structure as shown above. Whenever a new farm is added the daemon
automatically creates a directory corresponding to a farm (e.g.
SWIFT
-
HEP

f
or Swift
Farm). The mcpxx subdirectories are created according to the mcp versions on a
particular farm (e.g.
CSE
-
HEP

has mcp13 & mcp14, whereas
OU
-
HEP

has only mcp14).
The status files and their XML representations are stored in these (mcpxx) directories.

Each mcpxx will typically contain, mcpxx(flat file), mcpxx_arch (XML representation of
the archived job information) and mcpxx.xml(XML representation of queued and live
jobs).


The
conf
and the
log
directories contain the McFarmGraph _daemon configuratio
n and
the log files respectively. Addition of farms is done through the configuration file. The
tmp

directory is used as a scratch space when the daemon is running to store the process
id of a running daemon as well as some temporary files (e.g. ls.txt).


Fig 1a and 1b illustrate the control flow in the McFarmGraph_daemon.


//CSE
-
HEP

/SWIFT
-
HEP

/OU
-
HEP

/LTU
-
HE
P

/conf

/log

/tmp

README

/mcp10

/mcp11

/mcp14

daemon.conf

daemon.log

mcp10

mcp11

mcp14

/home/mcfarm/McFarmGraph_New/

UTA HEP/Computing/0026: McFarmGraph______________________________________________
4





Fig 1a. McFarmGraph Daemon Flowchart



Read the configuration file.

Start

Start/Stop/
check/no
args

Check if the daemon is up

Is
running?

Print
“daemon is
running”

Exit

Fork a child
process and
separate it from
the parent

Invoke the
main ( )
subroutine

Print
“daemon is
running”

Sleep for specified time
(UPDATE_INTERVAL)

Sleep
Interval
Over?

Print Usage
and Exit

Check if the daemon is up

Is
running?

Print

“daemon is
running”

Exit

Read the
daemon’s
process id

Flush the
logs

Issue a kill with
pid as the
daemons’ pid

Exit

Start

Stop

Yes

No

Yes

No

No

Yes

UTA HEP/Computing/0026: McFarmGraph______________________________________________
5







Fig 1b. main() subroutine in the McFarmGraph Daemon




3. XML GENERATOR:

Redirect the output stream
to the Log File.

Main Subroutine:
main ()

Exit

Check for
configuration
errors

Print “Configuration
Errors”


Create Farm objects
corresponding to
NUMBER_OF_FARMS

For each farm, call the
farm_mkdir () method to
create farm specific
directories if not already
present.

Read the configuration file
for

the farm variables by
invoking the initialize()
subroutine

Errors ?

For each Farm, call the
getFiles () method to retrieve
the job status files.

Yes

No

When all the files are
retrieved, start the XML
generator

UTA HEP/Computing/0026: McFarmGraph______________________________________________
6

In ear
lier version of McFarmGraph lot of computation was done while the client (browser)
was waiting. Although this processing wasn’t a bottleneck but would have increased as
the size of flat files increase. So in order to avoid this, bulk of the processing is n
ow
being done offline with the data stored in an XML file. While generating the page the
task is now simply reading the data from the XML file and generating the HTML code.


The task of generating the XML data is done by two scripts. A wrapper which for ea
ch
file pulled over from various farms calls a subroutine (in xmlgen.pm) which generates the
XML data for that status file. The diagram below shows the flow chart for the
subroutine.

START

Flow chart of XML generator

Read the file path of the file to operate on
and create file paths for both X
ML files

Create a temporary sorted file
from status file


EOF

No

Read a line from the
sorted file

RequestId
changes?

No

Accumulate job info


Calculate % Done

%Done =
100 ?

write info in
live jobs xml
file

No

Yes

Accumulate
archived job
info

Yes

Write archived job
info to arch XML
file

EXIT

Delete the sor
ted
file

UTA HEP/Computing/0026: McFarmGraph______________________________________________
7

4. CGI and PERL Scripts


The following script
s generate the various Web pages:

1.

filter.cgi

2.

applet.pm

3.

filemani.pm

4.

generalpage.pm

5.

html.pm

6.

jobpage.pm


All of these scripts are written in PERL and are located under /usr/local/apache2/cgi
-
bin
on hepfm000.uta.edu. Apart from these scripts there is the java
applet code which is in
the file PieChart.java under /usr/public_html/job_status/applet hepfm000.uta.edu. All the
images that are used in the web pages and the “style.css” file are also under
/usr/public_html/job_status on hepfm000.uta.edu.


Functions of v
arious scripts


filter.cgi:

All the requests from the browser are directed to filter.cgi along with a set of
parameters. This script then invokes subroutines in other files depending on the
parameters.

applet.pm:
This script generated all the ap
plet specific HTML code.

filemani.pm:

This script consist a single subroutine whose functions are explained below.


generalpage.pm:
This script generates bulk of the Req. Desc page.

jobpage.pm:
This script generates the Job Desc page

html.pm:
This scri
pt prints most of the HTML code for all scripts.


Generation of Web Pages

McFarmGraph generates most of the pages dynamically using CGI. The only static page
is the “index.html” page which is stored under /usr/public_html/job_status on
hepfm000.uta.edu. Fo
r adding a new farm this page has to be modified (refer the
installation guide for more details). For the other pages there are 3 cases:


1.

“Farm Request Ids” Page Request

UTA HEP/Computing/0026: McFarmGraph______________________________________________
8


In this case the parameters passed to the filter.cgi fil
e include “main” and the farm name.
The filter.cgi script calls the html.pm file function to print the header and then calls the
readDir function which read the directory for the requested farm and creates a link for
each mcp version available on that farm
.



Job Status
index.html
page

Browser


filter.cgi

main, farm name

Generation of “
Farm Request Ids” page


html.pm

printHTMLHeader,

printHTMLfooter


webpage


filemani.pm

readDir


printHTMLCell ,


printCellLink


HTML code

Req and reply between
browser and cgi script

Call to functions

Function Return

UTA HEP/Computing/0026: McFarmGraph______________________________________________
9

2.

“Farm Request Desc.” Page Request




UTA HEP/Computing/0026: McFarmGraph______________________________________________
10




The page generated here will either be one containing all the “
live jobs
” or all the
“
archived jobs
” on this farm for the requested mcp version depending on the presence of
last attr
ibute. filter.cgi script calls generalPage subroutine in generalpage.pm file which
does the rest of the processing. generalPage calls various subroutines in html.pm file and
also printApplet in applet.pm which embeds the applet into the HTML code generated
.
Farm request
ids page

Browser


filter.cgi

genpage, mcp ver.


farm name, arch?

Generation of “
Farm Request Desc.” page


html.pm


webpa
ge


generalpage.
pm

generalPage


printHTMLHeader ,


printHTMLCell ,


printCellLink


HTML code


applet.pm

printApplet


HTML code


HTML code

Req and reply between
browser and cgi script

Call to functions

Function Return

printHTMLfoo
ter

UTA HEP/Computing/0026: McFarmGraph______________________________________________
11

The PHASES column in archived page indicates all the phases this Request has gone
through.


3

“Job Desc.” Page Request





Farm request
desc. page

Browser


filter.cgi

jobpage, mcp ver.


Req desc.,farm name

,status

Generation of “
Job Desc.” page


html.pm


webpage


jobpage.pm

jobPage


printHTMLCell




HTML code

Req and reply between
browser and cgi s
cript

Call to functions

Function Return

printHTMLHeader,

printHTMLfooter

UTA HEP/Computing/0026: McFarmGraph______________________________________________
12

The page generated here lists all the jobs in a particular group identified by the Request
Id passed o
n by the browser. If a status parameter is present in the request (present in case
the applet link is clicked) then the page lists the details of all the jobs in the group whose
current status is the one requested. For e.g. it might contain all the jobs wh
ose status is
“D0GSTAR”.


3. FUTURE WORK:


The performance of the McFarmGraph tool can certainly be improved. Some of the
future work is highlighted below. We would like to reiterate that these are just some of
the suggestions; no study of their feasib
ility and success is done.




Exploring options to cgi
: When the number of farms being monitored increases
cgi scripts could be a potential performance bottleneck. Java servlets might be
able solve this issue.




Expiration of proxies
: During the course of dev
elopment of McFarmGraph, it was
observed that the proxies expire, thus disabling the retrieval of status files from
the remote site. Modification to the daemon could solve this problem.




Java applets load slowly
: For every row in the job status page a new
applet is
loaded and executes on the client side Java Virtual Machine. Mechanisms for
caching the byte code and having a single instance of the applet would speed up
the loading time.




McFarmGraph status updates
: Currently McFarmGraph pulls status files;
majority of the information that it contains has already been pulled over before.
Instead it would be more efficient to get information about those Requests that
either have jobs that are still running on the farm or have those requests that have
finished
since last update.