A.R.M.S. Active Resource Management Services

lessfrustratedBiotechnology

Oct 23, 2013 (3 years and 5 months ago)

70 views

A.R.M.S. Active Resource
Management Services


Presentation One

2/21/2013

1


Outline

Introductions

Societal Issue
E
xamined



Michael
Rajs

2/21/2013


2

Outline


Group Members and Roles:
s
lide 4



Introduce
Mentor: slide 5


Societal
I
ssue: slide 6


History: slides 7
-
11


Case
S
tudy: slides 12
-
16


Problem
Statement: slide 18


Computer
C
omponents
Identified: slides 19
-
21


Major Functional Component
Diagram: slide 22


Current Process
F
low: slide
23



Solution
S
tatement: slide 25


Objectives: slide 26


Improved
Process
F
low:



slide 27


Competition
Identified:
slides 28
-
30


Benefits
of
Solution: slide 32


Problems with Solution: slide
33


Conclusion: slide 34


References: slides 35
-
36



2/21/2013

3

Group Members and Roles


Michael
Rajs

(Group Manager)


Adam Willis


(Research Specialist)


Sybil
Acotanza

(Visualization
Engineer)


Scott
Pardue

(Team Leader)


Jordan
Heinrichs (Marketing Analyst)


David
Crook (Documentation
Specialist)






2/21/2013

4

Yaohang Li


Is an Associate Professor
in the Department of
Computer Science at Old
Dominion University.


His research interests are
in Computational
Biology, Markov Chain
Monte Carlo (MCMC)
methods and Parallel
Distributed Grid
Computing.



2/21/2013

5

What is the societal issue being faced?


How do researchers handle the
massive amounts of data they are
collecting?


2/21/2013

6

Historical Background

Adam Willis

2/21/2013

7

Collection of Data


1890 Census
R
ecorded
W
ith an Electric
M
achine
1


1935 Social Security Act
2


1974 Privacy Act
3


1989 World Wide Web
4


1997 Big Data
5


2011 IBM’s Watson
6


Now

“Every
day, we create 2.5 quintillion bytes of data


so much that 90% of the data in the world today has
been created in the last two years alone
.”
7

2/21/2013

8

Examples of Big Data


Large
Hadron Collider
8


150 million sensors report 40 million
times per second


Facebook
9


2.5 billion


content items shared


2.7 billion


“Likes”


300 million


photos uploaded


Walmart
8



1
million customer transactions



2.5
petabytes of data




2/21/2013

9

Big Data Analysis Hardware


Cluster Computing
10


A cluster consists of many nodes (computers).


Big data can be generated and analyzed
quicker by spreading the workload amongst
the nodes.



2/21/2013

10

Managing the Cluster


Distributed Resource
Management Systems (D
-
RMS
)


Job management subsystem


Physical resource management
subsystem


Scheduling and queuing
subsystem

2/21/2013

11

Case
Study


Sybil
Acotanza

2/21/2013

12

Dinosolve

Case Study


Bioinformatics


Disulfide bond prediction program




(
Cronk
, 2012)

2/21/2013

13

Dinosolve

Users


Who will use it?


Drug and antibody design


Bio
-
energy development


Genetic mapping
11


Why will they use it?


2% accuracy improvement
12

2/21/2013

14

Dinosolve

Web Site

(Li &
Yaseen
, http://hpcr.cs.odu.edu/dinosolve/)

2/21/2013

15

Dinosolve

Possible Problems


Hard resources for computation


CPU cycles


Memory


Disk space


Network bandwidth


Server crashes

2/21/2013

16

Problem statement

Components of
H
ardware and Software

Current Process
F
low


Scott
Pardue

2/21/2013

17

What is the problem?


Processing
time on big data sets is
computationally expensive
and as
the volume of queries grows the
system will
progressively drop in
performance until the system fails.

2/21/2013

18

What are the components of our current
system?



The current system uses the
following software and hardware.

2/21/2013

19

Software


Unix operating system installed on
the
dinosolve

cluster


Dinosolve

algorithm


Sun Grid Engine which will be our
Distributed Resource
M
anagement
S
ystem (D
-
RMS) installed on the
cluster.


MySQL (database software)


Web based user interface (website)

2/21/2013

20

Hardware



MySQL database server


A computer cluster to run the
dinosolve

algorithm


Web server for our web based
user interface




2/21/2013

21

Major Functional
C
omponent
D
iagram

2/21/2013

22

2/21/2013

23

Solution
Statement

Objectives

Improved
Process Flow

Competition
Identified



Jordan Heinrichs

2/21/2013

24

How will we correct the problem?


We aim to configure a distributed
resource management system

(D
-
RMS), in this case Sun Grid
Engine (SGE), to handle resource
allocation on the
dinosolve

cluster.

2/21/2013

25

Objectives


Interpret and visualize
current
usage
statistics


Configure, utilize, and optimize
the SGE


Aesthetically pleasing and
professional
user interface


2/21/2013

26

Process Flow with Solution

2/21/2013

27

Competing Distributed Resource
Management Systems



Sun Grid Engine (SGE)


Portable Batch System (PBS)


Load Sharing Facility (LSF)


2/21/2013

28

Competing Resource Management
Systems

Features

of
systems

PBS

LSF

SGE

Supported
platforms

Unix

Unix & NT

Unix

Multi
-
cluster

support

Yes

Yes

No

System level
checkpoint restart

Yes

Yes

Yes

User level
checkpoint restart

No

Yes

Yes

Large
computational grid
support

No

No

No

Massive Scalability

Yes

Yes

Yes

Parallel

job
support with Sun
HPC
ClusterTools

Loose Integration

Tight Integration

Loose Integration

Distribution

format
of end product

Source

Binary only

Binary and Source

Free?

Yes

No

Yes

Posix

1002.2d
compliance

Yes

No

Yes

2/21/2013

Reference 31

29

Competing Protein Prediction Servers

2/21/2013

Reference 19,20 and 21

30

Dinosolve

DiANNA

Scrath

Protein
Predictor

Accuracy

90.8%

81%

87%

Usability

X

X

X

508.22
compliance
percentage

67%

85%

67%

Professional

Benefits of solution

Problems with solution

Conclusion


David Crook

2/21/2013

31

What benefits will come from attaining
our goals?



Efficient utilization of available
resources


Increased throughput of the cluster


An intuitive and professional user
interface


Rise in popularity due to excellent
accuracy, efficiency, and professional
design


2/21/2013

32

Problems with solution



Improper
s
ynchronization of cluster
resources can lead to a deadlock in
the system


Race conditions between the HPCR
cluster and the MySQL database



2/21/2013

33

Conclusion


With the updated user interface
and correctly configured Sun Grid
Engine we hope to establish a
reputable
Disulfide Bonding
Prediction
Server.

2/21/2013

34

References for history

1.
http://
www.columbia.edu/cu/computinghistory/hh/index.html

2.
http://
query.nytimes.com/gst/abstract.html?res=F50C11FE385D13728DDDAE0A
94DA415B868FF1D3

3.
http://
www.census.gov/history/pdf/kraus
-
natdatacenter.pdf

4.
http://
www.bbc.co.uk/history/historic_figures/berners_lee_tim.shtml

5.
http://
dl.acm.org/citation.cfm?id=266989.267068&coll=DL&dl=GUIDE

6.
http://www.nytimes.com/2012/08/12/business/how
-
big
-
data
-
became
-
so
-
big
-
unboxed.html?_
r=1

7.
http://www
-
01.ibm.com/software/data/bigdata/

8.
http
://
en.wikipedia.org/wiki/Big_data

9.
http
://techcrunch.com/2012/08/22/how
-
big
-
is
-
facebooks
-
data
-
2
-
5
-
billion
-
pieces
-
of
-
content
-
and
-
500
-
terabytes
-
ingested
-
every
-
day
/

10.
http
://
en.wikipedia.org/wiki/Computer_cluster



2/21/2013

35

References for case study

11.

Li, Y. (2010, September 1). CAREER: Novel Sampling Approaches for Protein
Modeling Applications [Abstract].

National Science Foundation Award
Abstract #1066471
.


12.

Li, Y., &
Yaseen
, A. (2012). Enhancing Protein Disulfide Bonding
Prediction

Accuracy with Context
-
based Features.

Biotechnology and
Bioinformatics Symposium


13.

bioinformatics. 2011. In Merriam
-
Webster.com. Retrieved February 15, 2013,
from

http://www.merriam
-
webster.com/dictionary/bioinformatics


14.
Cronk
, J. D. (2012). Disulfide Bond. Retrieved February 15, 2013, from
Biochemistry Dictionary:



http://guweb2.gonzaga.edu/faculty/cronk/biochem/D
-
index.cfm?definition=disulfide_bond


15.

Yan, Y., & Chapman, B. (2008).

Comparative Study of Distributed Resource
Management Systems

SGE, LSF, PBS Pro, and
LoadLeveler
. Technical Report
-
Citeseerx
.


16. Li, Y., &
Yaseen
, A. (2012).
Dinosolve
. Retrieved from
http://hpcr.cs.odu.edu/dinosolve/

2/21/2013

36

References for competition

17.
Arvind

Krishna, “Why
Big Data?
Why Now?”, IBM , 2011

URL: http
://
almaden.ibm.com/colloquium/resources/Why%20Big%20Data%20Krishna.PDF

18.
Yonghong

Yan
, Barbara M. Chapman, Comparative Study of Distributed Resource Management
Systems
-

SGE, LSF, PBS Pro, and
LoadLeveler
, Department of Computer Science, University of Houston, May
2005 (
pdf
)

19.
Dr. Li’s site


http://hpcr.cs.odu.edu/dinosolve/

20. Scratch
Predictor


http
://scratch.proteomics.ics.uci.edu/

21.
DiANNA

server


http://clavius.bc.edu/~clotelab/DiANNA/

Portable
Batch System (
PBS)

22.
http
://
resources.altair.com/pbs/documentation/support/PBSProUserGuide12
-
2.pdf

23.
http
://
www.pbsworks.com/SupportDocuments.aspx?AspxAutoDetectCookieSupport=1

24. http
://
resources.altair.com/pbs/documentation/support/PBSProRefGuide12
-
2.pdf

25.http
://
resources.altair.com/pbs/documentation/support/PBSProAdminGuide12
-
2.pdf

26.http
://www.pbsworks.com/(S(tykrsyqbemmlf3o5zwrmjrgf))/
images/solutions
-
en
-
US/PBS
-
Pro_Datasheet
-
USA_WEB.pdf

27.http
://agendafisica.files.wordpress.com/2011/05/pbs.pdf

Moab HPC Suite

28.http
://
www.adaptivecomputing.com/publication/420/wppa_open/

IBM
Platform
LSF

29.http
://
public.dhe.ibm.com/common/ssi/ecm/en/dcd12354usen/DCD12354USEN.PDF

Apache
Hadoop

with Zookeeper

30. http
://
zookeeper.apache.org/doc/current/zookeeperOver.html

31. http://www.cloud
-
net.org/~swsellis/tech/solaris/performance/doc/blueprints/0102/jobsys.pdf




2/19/2013

References

37