Ruby (.rb) - thesisdev - Google Code

knifedamagingInternet και Εφαρμογές Web

2 Φεβ 2013 (πριν από 4 χρόνια και 2 μήνες)

513 εμφανίσεις







eNotify: Re
-
Engineering a Bug Tracking Application to
Meet
Enterprise
Requirements














Paul J
Russell










A Thesis in the Field of Information Technology


for the Degree of Master of Liberal Arts in Extension Studies





Harvard Uni
versity



June 2009

Abstract

This is a thesis for the degree of Master of Liberal Arts in Information
Technology (ALM in IT). The goal of this project is to remedy the scaling issues of a
widely used, server
-
bound,
bug

tracking
application named eNotif
y.

I designed and developed eNotify several years ago for a small group of teams. In
the intervening years it has been widely adopted by the engineering community
throughout the company. The scope of the current deployment far exceeds the original
design c
riteria. As a result, the application requires high maintenance costs and no longer
meets its end users’ requirements for data freshness. This project solves these issues and
positions the application for future enhancement by upgrading eNotify’s applicati
on
development framework and migrating to a distributed deployment model.

This thesis project comprises three major development tasks and a real world
deployment test. The three development tasks cover the major steps required to:

1.

Create a scalable, easi
ly
-
enhanced, web
-
based administration console;

2.

Convert the existing bug tracking processing engine to work with the new
administration console;

3.

Configure the application to work in a distributed model that enables
scalability and ease of maintenance.


I

demonstrate how a distributed design and use of a full
-
stack web application
development framework addresses the scaling issues of a mission critical application in a
real business environment while reducing the operating cost of running that system.


iv

Au
thor's Biographical Sketch

Paul Russell was born April 7, 1975 in New York, New York to parents James
and Connie Russell. His family moved to Connecticut in his early teens where he
discovered music. By his late teens he was an accomplished saxophonist an
d tuba player.
While he preferred to play alto saxophone in jazz ensembles, Paul rarely turned down the
opportunity to play with any group and he quickly become a regular face in nearly all of
the school bands and ensembles. In his sophomore year he brief
ly dated a girl named
Christine whom he had met on a school bus.

Paul’s father had a knack for both Physics and teaching. He tutored Paul, allowing
him to complete the school’s one
-
year Physics course in three
-
months, just in time for
him to take the Janu
ary SAT in Physics. Paul received a near perfect score on that test
and became enamored with science for the first time. Outside school and music, Paul
trained to become an Emergency Medical Technician and volunteered with his town’s
Emergency Medical Serv
ice.

He attended Rutgers University in New Jersey for two years where his primary
focus was studying Physics, playing Ultimate Frisbee, and volunteering at the local
Emergency Medical Service. During the summer he returned home to Connecticut where
he org
anized and performed in a big band concert series called Random Acts of Jazz,
which raised several thousand dollars through concert ticket sales, and donated all profits
to the elementary school music program.



v

The experience and lessons Paul learned from

running Random Acts of Jazz
provided the foundation for much of his future business success.

Paul found it hard to find a home at Rutgers and transferred to the University of
Connecticut (UConn) where he majored in Biology. At UConn he elected to live in

the
dormitory for Biology students, but on his arrival discovered that while the building was
labeled as a Biology dorm 95% of its residence were declared Computer Science majors.
His constant exposure to Computer Science seeded his interest in the field.

Paul completed his undergraduate degree in Biology and entered the job market at
the height of the tech bubble. A friend from his Random Acts of Jazz days introduced
him to a small web collaboration software startup in Boston, Massachusetts. Lacking any
formal experience, Paul was turned down for a job twice before finally being offered and
accepting a position in Quality Assurance. He took advantage of every opportunity to
learn and quickly built up a competence with basic programming concepts. In 2000
Paul
began a Masters in Information Technology at Harvard University.

That same year, Paul and Christine reunited. They married and now have three
children, Alex, Conner, and Allison. Paul’s strongest memories from his Master’s studies
is the sound of his
family playing outside on a beautiful summer day while he labored
inside his house on Algorithms and Data Structures assignments.


Paul is currently a Business development manager at a large technology
company. He lives in Acton, Massachusetts with his wif
e, three children, and three cats.
He has recently started teaching his son Alex to play the saxophone.



vi

Dedication


This thesis is dedicated to my grandmother, Mrs. James A. Russell and my father
James T. Russell, whose support, urging,
and frequent stat
us requests
, gave me the
encouragement I needed to complete this project.


vii

Acknowledgments


This work would not have been possible without the support and encouragement
of my thesis advisor Dr. John G. Norman. One of my greatest take always from this
proj
ect will be to have worked with a person with such a natural gift for teaching. His
perspective and guidance on every aspect of this project has been invaluable. Over the
course of this project I’ve come to think of him as a trusted mentor and friend.

I wo
uld like to thank Dr. Bill Robinson, under whose supervision I chose the topic
and developed the project proposal. The selection of this project provided an opportunity
to address the requirements of my degree while providing a valuable service to my
emplo
yer.

I am indebted to my thesis director
Dr. Jeff
Parker

who guided me through the
final stages of the thesis project.

It is a rare gift to work with someone that can be both
serious and funny at the same time. His communication style is one of the mos
t
motivating techniques for delivering constructive criticism I’ve experienced.

I would also like to thank my parents James and Connie Russell, and my
grandmother, Mrs. James A. Russell, who encouraged me to continue my studies after
receiving my Bachelor
s degree.

I am grateful to my brother John, for joining me on this journey through a Masters
Degree, putting up with me every day at work and providing much needed comic relief
during stressful times.

Finally, and most importantly I would like to thank my

wife Christine.
Throughout my Masters studies she has often been left alone to raise our three wonderful

viii

but often mischievous children while I was locked away in an office working. Her love
and support throughout the years has enabled me to complete thi
s program. This success
is as much hers as it is mine.



ix

Table of Contents

Table of Contents

................................
................................
................................
...............

ix

List of Figures

................................
................................
................................
.................

xvii

Chapter 1 Introduction

................................
................................
................................
........

1

Bu
g Tracking in a Large Company

................................
................................
.................

2

Legacy eNotify Application Overview

................................
................................
...........

4

What eNotify Does

................................
................................
................................
.........

6

eNotify Server Basics

................................
................................
................................
.

6

Website Generator Work

Flow

................................
................................
...................

7

Scaling and Server Limitations

................................
................................
...................

8

Legacy Application Shortcoming: eNotify Falls Victim to Its Own Success
...........

10

Chapter 2 Solution Requirements and Goals

................................
................................
....

17

Solutions Considered and Rejected

................................
................................
..............

20

Generate the Pages Dynamically

................................
................................
..............

20

Buy a Bigger Server

................................
................................
................................
..

21

Make the Site Generation More Efficient

................................
................................
.

22

Chapter 3 Project Description

................................
................................
...........................

24

Three Part Solution Strategy

................................
................................
.........................

24

1. Simplify the Website Generation Engine

................................
.............................

24

2. Implement a Central Control
ler

................................
................................
............

25

3. Implement a Distributed Deployment Model

................................
.......................

26


x

Solving the Graph Generation Problem

................................
................................
........

29

Languages, Environment and Straying From Standards

................................
..............

30

Ruby on Rails

................................
................................
................................
............

30

Perl

................................
................................
................................
............................

31

Web Services

................................
................................
................................
............

31

Development Environment

................................
................................
.......................

31

Hardware and Operating System Environment

................................
........................

32

Parting From a Conventional Rails Application

................................
.......................

32

Chapter 4 Project Implementation

................................
................................
....................

35

Controller Web Administration

................................
................................
....................

35

Teams List

................................
................................
................................
.................

38

Adding Teams to a Project

................................
................................
........................

38

Bug List

................................
................................
................................
.....................

39

Projections

................................
................................
................................
.................

40

Online Help

................................
................................
................................
...............

42

Navigation

................................
................................
................................
.................

43

Authen
tication and Authorization

................................
................................
.............

44

Definition of the Controller Schema and Migration of Legacy Data

...........................

45

Schema Creation

................................
................................
................................
.......

46

Switching From a Perl to a Ruby Based Import Scri
pt

................................
.............

50

Mechanics of Importing Legacy Data

................................
................................
.......

52

Generator Communication Mechanics

................................
................................
.........

52

Generator Communication Code

................................
................................
..............

55


xi

Error Management and De
aling With Rogue Generators

................................
.........

56

Graphing Implementation

................................
................................
.............................

58

Graphing Implementation

................................
................................
.........................

59

Generator Changes to Support Graphing

................................
................................
..

60

Applicatio
n Deployment

................................
................................
...............................

61

NFS Configuration

................................
................................
................................
........

61

Controller Configuration

................................
................................
...........................

62

Supporting Generated Site URL Pass
-
Through

................................
........................

62

Enabling Generators

................................
................................
................................
..

63

Chapter 5 Summary and Conclusions

................................
................................
...............

64

Comparison Metrics

................................
................................
................................
......

64

Key Accomplishments and Lessons Learned

................................
...............................

67

Convention Over Configuration with t
he Team Model

................................
............

67

ActiveScaffold
-

Record Level Policy & Permissions

................................
..............

69

Identifying the Current Logged In User In the Model

................................
..................

71

Messaging Between Controllers and Genera
tors

................................
......................

72

Addressing the Moving Projections Target

................................
..............................

73

Future Plans for eNoitfy

................................
................................
...............................

75

Disabling a Site From Generating

................................
................................
.............

75

Predicting the Next
Generation Time For a Site

................................
.......................

75

Partitioning eNotify Scale Based On Organizations

................................
.................

76

Augmenting security

................................
................................
................................
.

77

Glossary

................................
................................
................................
............................

78


xii

References

................................
................................
................................
.........................

80

Appendix 1 Application Code

................................
................................
..........................

82

Cascading Style Sheet (.css)

................................
................................
.........................

82

public/stylesheets/custom.css

................................
................................
....................

82

Embedded Ruby (.erb)

................................
................................
................................
..

83

app/views/administrators/edit.html.erb

................................
................................
.....

83

app/views/administrators/for_project.html.erb

................................
.........................

84

app/views/administrators/index.html.erb

................................
................................
..

85

app/views/administrato
rs/new.html.erb

................................
................................
....

86

app/views/administrators/show.html.erb

................................
................................
..

87

app/views/bug_lists/for_project.html.erb

................................
................................
.

88

app/views/graphs/show.html.erb

................................
................................
...............

89

app/views/layouts/admin_standard.html.erb

................................
.............................

97

app/views/login/index.html.erb

................................
................................
...............

100

app/views/owners/new.html.erb
................................
................................
..............

101

app/views/people/edit.html.erb

................................
................................
...............

102

app/views/people/show.html.erb

................................
................................
.............

103

app/views/projections/edit.html.erb

................................
................................
........

104

app/views/projections/edit_group.html.erb

................................
.............................

105

app/views/projections/index_f
or_bug_list.html.erb
................................
................

106

app/views/projections/index_for_team.html.erb

................................
.....................

107

app/views/projections/show.html.erb
................................
................................
......

108

app/views/projects/edit.html.erb

................................
................................
.............

109


xiii

app/views/projects/index.html.erb

................................
................................
..........

110

app/views/projects/show.html.erb

................................
................................
...........

111

app/views/teams/for_project.html.erb

................................
................................
.....

112

app/views/teams/new.html.erb

................................
................................
................

113

app/views/teams/show.html.erb

................................
................................
..............

114

Perl (.pm)

................................
................................
................................
....................

115

generator/Enotify/Generator/ControllerCommunicator.pm

................................
....

115

Ruby (.rb)

................................
................................
................................
....................

123

app/controllers/administrators_controller.rb

................................
...........................

123

app/controllers/application.rb

................................
................................
.................

126

app/controllers/bug_columns_controller.rb

................................
............................

127

app/controllers/bug_l
ists_controller.rb

................................
................................
...

129

app/controllers/generations_controller.rb

................................
...............................

133

app/controllers/graphs_controller.rb

................................
................................
.......

137

app/controllers/login_controller.rb

................................
................................
.........

138

app/controllers/owners_controller.rb

................................
................................
......

139

app/controllers/people_controller.rb

................................
................................
.......

142

app/controllers/projections_controller.rb

................................
................................

144

app/controllers/projects
_controller.rb

................................
................................
.....

147

app/controllers/teams_controller.rb

................................
................................
........

150

app/helpers/bug_columns_helper.rb

................................
................................
.......

153

app/helpers/bug_lists_helper.rb

................................
................................
..............

154

app
/helpers/owners_helper.rb

................................
................................
.................

155


xiv

app/helpers/projects_helper.rb

................................
................................
................

156

app/helpers/teams_helper.rb

................................
................................
...................

158

app/models/administrator.rb

................................
................................
...................

159

app/mod
els/bug_column.rb

................................
................................
.....................

160

app/models/bug_list.rb

................................
................................
............................

161

app/models/bug_list_downloads.rb

................................
................................
........

162

app/models/bug_ownership_change.rb

................................
................................
...

163

app/models/
bug_tool_url.rb

................................
................................
....................

164

app/models/generation.rb

................................
................................
........................

165

app/models/generator.rb

................................
................................
..........................

166

app/models/owner.rb

................................
................................
...............................

167

app/models/ownership_model.rb

................................
................................
............

168

app/models/person.rb

................................
................................
..............................

169

app/models/project.rb

................................
................................
..............................

171

app/models/projection.rb

................................
................................
........................

181

app/models/scheduler.rb

................................
................................
.........................

184

app/models/scope_filter.rb

................................
................................
......................

187

app/models/site_update_frequency.rb
................................
................................
.....

188

app/models/team.rb

................................
................................
................................
.

189

config/environment.rb

................................
................................
.............................

190

config/routes.r
b

................................
................................
................................
.......

192

db/migrate/002_create_projects.rb

................................
................................
..........

194

db/migrate/003_create_generations.rb

................................
................................
....

195


xv

db/migrate/004_create_generators.rb

................................
................................
......

196

db/migrate/
005_create_bug_list_downloads.rb

................................
......................

197

db/migrate/006_create_site_update_frequencies.rb

................................
................

198

db/migrate/007_create_people.rb

................................
................................
............

199

db/migrate/008_create_bug_tool_urls.rb

................................
................................

200

db/migrate/010_create_administrators.rb

................................
...............................

201

db/migrate/011_create_bug_lists.rb

................................
................................
........

202

db/migrate/012_create_ownership_models.rb

................................
........................

203

db/m
igrate/013_create_projections.rb

................................
................................
....

204

db/migrate/014_create_bug_columns.rb

................................
................................
.

205

db/migrate/015_create_owners.rb

................................
................................
...........

206

db/migrate/017_create_scope_filters.rb

................................
................................
..

207

db/migrate/018_create_bug_ownership_changes.rb

................................
...............

208

db/migrate/019_create_teams.rb

................................
................................
.............

209

lib/enotify_ldap.rb

................................
................................
................................
...

210

lib/thesis_importer.rb

................................
................................
..............................

213

lib/user_info.rb

................................
................................
................................
........

226

script/import/project.rb

................................
................................
...........................

227

script/thesisdoc/generate_doc.rb

................................
................................
.............

228

YAML Ain't Markup Language (.yml)
................................
................................
.......

233

config/database.yml

................................
................................
................................

233

db/constant_fixtures/bug_columns.yml

................................
................................
..

235

db/constant_fixtures/bug_ownership_changes.yml

................................
................

241


xvi

db/constant_fixtures/bug_tool_urls.yml

................................
................................
.

242

db/constant_fixtures/ownership_models.yml

................................
.........................

243

db/constant_fixtures/scope_filters.yml

................................
................................
...

244

db/constant_fixtures/site_update_frequencies.yml

................................
.................

245


xvii


List of Figures

Figure 1 eNotify Website Generator Work flow

................................
................................

8

Figure 2 Current Refresh Time for Sites on Legacy eNotify Servers

...............................

11

Figure 3 Example eNotify Gr
aph

................................
................................
......................

13

Figure 4 Time to Complete Tasks
-

Legacy System vs Goals

................................
..........

17

Figure 5 Proposed eNotify Deployment Model

................................
................................

27

Figure 6 eNotify Web Administration Project Summary

................................
.................

36

Figure 7 eNotify Web Administration Edit Project Settings

................................
............

37

Figure 8 Project Summary with Setup Instructions

................................
..........................

37

Figure 9 eNotify Web Administration Teams List

................................
...........................

38

Figure 10 eNotify Web Administration Project Bug Lists

................................
...............

40

Figure 11 eNotify Graph Showing Projection and Historical Bug Data

..........................

41

Figure 12 eNotify Web Administration Vi
ew and Edit Projections

................................
.

42

Figure 13 eNotify Web Administration Online Help

................................
.......................

43

Figure 14 eNoify Web Administration Navigation Hierarchy From Team Page

.............

44

Figure 15 eN
otify Web Administration: Authorization Restricted

................................
...

45

Figure 16 Controller Schema

................................
................................
............................

47

Figure 17 Generator and Controller Communication Flow

................................
..............

53

Figure 18 eNotify Graph
Showing Projection and Historical Bug Data

..........................

58

Figure 19 Time to Complete Tasks
-

Legacy System vs Goals vs Results

.......................

65


1

Chapter 1

Introduction

This project is an extension of a bug tracking

application ca
lled eNotify, also
referred to as the legacy application. It generates a website showing all of the teams and
individuals involved in a software project and which bugs each person own. eNotify is
used by managers to track and drive bug management processes
, and by individuals to
determine exactly what bug
-
related work they have to complete.

eNotify was developed as part of Harvard Extension School course CSCI E
-
13,
“Practical Perl,” and deployed at my place of business.


The original launch of eNotify
met
with substantial success and the demand for the application’s services has far
exceeded the capabilities of the original design resulting in substantial support workload
for the legacy operation’s team and frustration for the end users.

The aim of this pr
oject is to develop and implement a solution to the legacy
shortcomings and deliver the following three benefits:


1.

The application is able to meet end users demands for current
data
;

2.

The support demand on the operations team is significantly reduced;

3.

The

application has been re
-
engineered such
that the existing team with
minimal effort can enhance it.

This document is broken into 5 chapters. Chapter 1 (this introduction) contains an
explanation of the need for this bug tracking

system, an explanation of

the legacy

2

application, and a review of its shortcomings. Chapter 2 details the goals of this project,
criteria for success and solutions that were investigated and not selected. Chapter 3
provides the description of this thesis project, detailing the sol
ution that was chosen, the
high level design of the new application, and discussion of the reasoning for decisions
made in the design. Chapter 4 covers the details of the implementation including specifics
on each of the five phases of the project developm
ent and deployment. Chapter 5
describes the validation of successful requirements implementation. It also covers the
major achievements and lessons learned during this project as well as future plans for the
application.

Bug Tracking in a Large Company

La
rge engineering organizations with thousands of projects and tens of thousands
of engineers often have unique bug tracking processes when compared with the majority
of smaller engineering organizations in the world. The process a large company uses to
gove
rn bugs is a core part of its DNA and can be a differentiator amongst competitors.
Where some small organizations work well with the concept of assigning a bug to a
single person and having that person see the bug through the bug’s life cycle, in a large
c
ompany ownership of a bug usually spans more than one person. In a large engineering
organization, bug ownership is distributed to a number of people each specializing in a
specific part of the bug resolution.


For example:



New bugs need to be triaged and
assigned. The development
-
engineering
manager of the software component to which the bug belongs owns this
task.


3



Assigned bugs need to be fixed. The engineer to whom the bug is assigned
owns this task.



Resolved bugs need to be verified by someone other t
han the engineers
who fixed them.



Bugs require customer facing documentation that describes the symptoms,
steps to reproduce, and any known workarounds.

There are many other bug states and ownership scenarios. Even so, defining
ownership rules can be a s
imple task. The challenge comes in getting a large group of
individuals to understand the ownership rules, interpret them the same way, and realize
they need to take action on a bug without being reminded.


Without an effective bug
tracking application app
lying ownership rules that explicitly display the owner of bugs
for specific tasks, one of two scenarios occurs. Either bugs are ignored or the
organization invests a substantial amount of effort in manually tracking ownership of
bugs and notifying people
of their open tasks such that bugs are not ignored.


In my place of business,
the corporate bug tracking
system, from which eNotify
gets its bug data, does not provide the concept of bug ownership. Changing the corporate
bug tracking system to support ow
nership concepts is not an option due to cost. While
specific details of the corporate bug tracking system are confidential it is possible to share
the following details. At present the corporate bug tracking system contains tens of
million of records and
is tightly integrated with other company functions including but
not limited to software patching, manufacturing, development operations, customer
support and employee compensation. Additionally, over 10,000 people use the system
daily. The cost of retrain
ing those individuals to compensate for the addition of

4

ownership concepts to the workflow of the bug tracking user
-
interface was estimated at 8
million dollars, by the training organization. Figures in this range are typically non
-
starters if the person r
ecommending a software change does not own or operate the
system in question. Thus, at the start of this project replacing or changing this bug
tracking application to support ownership was
not a
viable

business decision.


Legacy eNotify Application Over
view

As a final project for CSCI E
-
13 “Practical Perl” I designed and developed an
application called eNotify that automated and streamlined my company’s internal bug
management processes. The pre
-
existing process was manual and required consistent
communi
cation and action from all involved/stake
-
holding employees in order to
maintain an accurate view of the current bug status. The bug management process often
broke down as a result of inconsistent communications resulting in missed delivery dates
and softw
are release delays.

An example of this occurred 6 years ago during a large product integration
project. Four products were being combined to produce a single feature
-
rich application.
The project personnel consisted of over 300 engineers making up 22 team
s. This was
their first time working together and most of the work consisted of making the four
products communicate with each other. As a result it was often unclear which of the
products was responsible when a bug was found.

Bruce (not his real name), th
e project manager, held a weekly project status
meeting where the managers of the 22 teams would provide status reports. Bruce made it
clear that the project had to resolve over 800 bugs before the product could ship in six

5

months. Given the 300 engineers

available this did not seem like an overwhelming task.
A massive spreadsheet with a list of the bugs was manually maintained. Each Monday
managers would report progress on reducing their bug count, but the overall 800
outstanding bug count was not going
down. Due to the complexity of the integration
task, the managers were mistakenly underestimating the number of bugs they needed to
fix by assuming it was another team’s responsibility.

One month before the expected ship date the outstanding bug count s
tood at just
under 800 bugs. Bruce called an open
-
ended meeting and forced the managers to go
through each bug and agree on its ownership. Thirteen hours, and a substantial amount of
arguing later, the spreadsheet was up to date. With a clear view of the
work in front of
them the managers agreed they would need an additional year to complete the project.

The following year Bruce implemented a strict set of rules governing spreadsheet
updating. Each team had a spreadsheet and was expected to keep it up to d
ate with the list
of bugs they owned. Weekly meetings were held, and the majority of the engineering
organization’s time was spent keeping the bug spreadsheets up to date and
communicating changes to engineers so they would know which bugs to fix. The proc
ess
was extremely error prone and at the end of the one
-
year project extension, the bug count
stood at 700. The project was extended for a third year. The leadership of the
organization began to realize that their bug management process had failed. With
a new
deadline looming and no reason to believe they would achieve different results with the
same process, they were looking for a new way to manage their project.


6

What eNotify Does

eNotify provides a mechanism for implementing a consistent ownership mod
el
across

bugs in a project. It eliminates the manual overhead of tracking bugs and provides
an automated mechanism to consistently communicate the specific work owned by each
individual to that individual and the rest of the team. A project leader uses th
e eNotify
web administration to identify the people working on a project and the lists of bugs that
should be tracked. Using this information and a predefined set of ownership models
eNotify generates a static website of simple action
-
oriented pages provid
ing the project
managers, teams and individual contributors with a clear view of the bugs they have and
what they should do with them.

eNotify was introduced during the third year of Bruce’s integration project. It
immediately cleared up all questions of b
ug ownership, eliminated the need for bug
spreadsheets, gave each engineer a clear list of bugs they needed to fix, and allowed the
engineering organization to shift its focus from bug tracking mechanics to the fixing of
bugs. When Bruce’s product shipped,

on time per the reschedule, it had 12 known, low
severity bugs and everyone knew exactly who owned those bugs.

eNotify Server Basics

The original eNotify was designed to reside on a single server. It is capable of
supporting bug tracking for 1 to ~10 pr
ojects in the form of a project website henceforth
referred to as a site. Configuration for each project’s site was managed through a web
portal and the configuration settings were stored on disk in flat files in a number of
formats (XML, property=value an
d CSV). The goal with this storage mechanism was to

7

keep the application simple and to allow for manual manipulation of the configuration
files if/when the administration console had a bug.

Website Generator Work Flow

The eNotify site generation process
is written in Perl (Perl Foundation, 2002) and
is launched by a recurring cron program (Stevens, 2002) in Linux or Scheduled Task
(Morimoto, Noel, Droubi, Mistry, & Amaris 2008) in Windows. The task checks for a
running version of eNotify and if one does n
ot exist it runs the eNotify website generator.
Websites are generated following the workflow diagrammed below. Per Figure 1, the
eNotify process reads the list of projects that need to be updated and updates them in
series. Once a site is generated, the H
TML (WC3, 1995) files that make up that site are
accessible via a web server that serves the static pages from the server’s disk.




8


Figure
1

eNotify Website Generator Work flow

Scaling and Server Limitations

Given the serial na
ture of project site rendering, theoretically the number of
projects supported on a server is limited by the availability of disk space required for the
generated project websites. In reality the time it takes to generate the series of websites
combined wi
th the end user’s demand for updates is the practical limiting factor. Each
project website contains information that is critical to the day
-
to
-
day operations of a
project. As a result, teams want their project websites updated as frequently as possible.



9

The time it takes to generate a project website is based on the number of objects (web
pages and graphs) that need to be generated. Thus generation time is proportional to:

<Constant rendering time> + ( <#users> * ( <#bug lists> + <#bugs
per list> ) )


As
the number of users, bug lists and bugs increase, so does generation time.
When run in series, the duration between site updates is equal to the sum of the
generation of all sites on that server. Since project website generation and page hosting is
tied to

a single server, and the variables (users, bug lists and bugs per list) are controlled
by the users, it is difficult to keep any given group of sites from exceeding the capacity of
their hosting server. When a server runs out of capacity, project website
generation
frequency drops and project teams are unable to get their bug ownership data in a timely
fashion. In extreme cases sites take more than 24 hours to refresh.

An example of eNotify capacity overload occurred in the months following the
successful
completion of Bruce’s integration project. During the project review, eNotify
was identified as a best practice and received significant publicity. As a result, I received
numerous requests from project managers to create eNotify websites for their projec
ts. At
the time, the eNotify server capacity and site refresh rate had never been tested. Because
Bruce’s project website was the only one on the server it refreshed without delay. Bruce’s
website took approximately one hour to generate, and that frequency

was acceptable to
Bruce and his engineers. As new project websites were added to the eNotify server and
project managers began adding teams and bug lists with large numbers of bugs,
individual project managers began to see delays in their site updating. T
he problem
became clear when a project manager named George added 60 teams with over 600
engineers and more than 40 bug lists to an eNotify project. George’s site took 6 hours to

10

generate and other project administrators immediately started complaining tha
t their
eNotify websites were not refreshing fast enough.

As more projects were added to the eNotify server it was not uncommon for a site
to go more than 24 hours without a refresh. The engineering organization had come to
rely on eNotify to provide task
lists to engineers. Delays in eNotify website refreshes
translated into lost engineering cycles as new bugs were entered into the bug tracking
system but were not identified by eNotify websites that had not yet refreshed.

Legacy Application Shortcoming: e
Notify Falls Victim to Its Own Success

The legacy eNotify application succeeded in streamlining our bug
management
process and within a year of Bruce’s project completion eNotify was adopted across the
business, requiring the deployment of 6 eNotify serve
rs. The addition of new servers was
driven by end
-
user’s requirements that sites must be refreshed in less than 2 hours.


11



Figure
2

Current Refresh Time for Sites on Legacy eNotify Servers

Figure 2 shows time between site refreshe
s on each eNoitfy server. Servers 1, 4
and 6 each meet users requirement for refresh times of less than 2 hours, and project
managers using those servers are quite happy. Server 2 has recently exceeded the two
-
hour mark and servers 3 and 5 are severely ov
erloaded with large projects that result in
long delays for websites to refresh. From the perspective of the operations team and the
end users, these servers are over capacity and the operations team is rejecting requests to
add new projects until this th
esis project is put into production.

Limitations of the legacy application result from the fact that eNotify sites are
bound to a server running an instance of eNotify that is responsible for generating sites

12

for many projects. All other sites see the time

it takes eNotify to generate one site as lost
refresh time. The task of load distribution, currently achieved by manually copying a
site’s metadata from one server to another server falls to the operations team. The process
is time consuming and error pro
ne.

Due to inconsistencies between the installations on two eNotify servers, the last
manual load distribution required a 9 day effort by an operations team member, during
which time one eNotify website was offline for 70% of the time. While it is difficu
lt to
quantify the productivity impact on the end users it was clear that the project manager
was extremely unhappy.

The major issues faced are the result of the basic design assumption that an
instance of the application is bound to a single server. Those

issues are:

Shar
ing Configuration Restrictions

There is no capability to share site configuration, user lists or bug lists across
servers. Project administrators want the ability to share or copy configuration from one
project to another but are unable to

do so

when projects reside on separate servers. In
order to share configuration, organizations must co
-
locate pr
ojects on a single server
which, as we saw in the case of George’s project, can
results in over utilization of that
server’s resources
,
reducti
on in
project

website update frequency
, and unhappy
customers.

Inability To Scale With Demand

Because the generation of a project website is tied to a specific server and a
project website’s generation time is proportional to <Constant rendering time> + (

13

<#users> * ( <#bug lists> + <#bugs per list> ) ) it is possible that the configuration of
additional bug lists in a project, or a spike in the number of bugs being tracked for a
project, will result in significant changes in the time it takes for a project
’s website to be
generated. This problem is exacerbated by the operation team’s lack of control over the
eNotify site metadata. Project managers are able to add bug lists and teams to projects on
demand, which can dramatically impact the generation time o
f a site and the delay for
other sites to refresh as was the case with George’s addition of 60 teams, 600 engineers,
and 40 bug lists to his project.


Graph Rendering Consumes Disproportionally Large Resources

eNotify graphs show three pieces of informatio
n:



The projections set for this bug list



The bug counts as recorded by the generator for past dates



The current bug count



Figure
3

Example eNotify Graph



14

Legacy graph generation uses the Perl package GD:Graph (
Verbruggen

, 199
9)
and generates one graph per bug list for each individual, each team and the cumulative
project bug data. Users insist that graphs always need to be available, but according to
the web server logs, views of the graphs are infrequent as compared to the n
umber of
times the graphs are generated.


Generating these graphs is processor intensive and represented a large portion of
the generation time and processing load. In order to improve the site refresh frequency a
trade off was made in the legacy applicati
on.


Graphs are only generated once a day. This
often reduces generation time by more than half, but it also results in out
-
of
-
date
information in cases where the projections are changed.

Projections are the predicted bug counts that a team expects to have

at any point
in time. Projections are used as a tool for setting a bug count reduction goal and then
tracking to that goal. Each team’s manager is responsible for entering their team’s
projected bug counts into eNotify though a web interface.

When a team

manager changes their projections the changes are not immediately
reflected on the graph. The manager must wait until the graphs generate the following
day in order to see the change.


This is often problematic for project planning purposes
and system adm
inistrators frequently receive requests from managers to manually trigger
a regeneration of graphs. These events increase system load, reduce site update frequency
and create manual work for the administrators.


15

Expensive Feature Development

The administra
tion console is intended to provide an interface for users to manage
their project’s website configuration. The prototype administration evolved from an
infrastructure using Perl CGI (Waterman, 1999) and Template Toolkit 2 (Wardley, 2007)
in a non Model Vi
ew Controller (Sun Microsystems, 2002) like framework. In order to
make a single change to functionality or display, code level edits are required in multiple
places: the objects responsible for reading and writing to the flat file storage, accesses
and mu
tates on the objects used in the admin, and in 2 layers of the admin display. A
change in one part of the application could easily require changes in three other places to
keep the application from breaking. As a result, the legacy generator remained unto
uched
for years and due to the high cost of change in this environment few updates or feature
enhancements are affordable.


New Site Management & Scaling Optimization

In order to manage new site addition to the servers, a dedicated administrator
from the
operations team fields each request for a new site and manually balances the
server load by guessing how much load the site will eventually cause and choosing a
server that will likely be able to support this load. Because most of the variables
determining

load are unknown at the time of site creation, this is an inefficient and
ineffective way of managing load. In addition to the challenges of guessing load, the
manual effort required from the operations team to create sites often results in a 3 to 4
day d
elay when a project manager requests a new eNotify site.



16

Project Administrator Change Requests

Each legacy eNotify project supports a single administrator. This administrator is
typically the project manager and is responsible for configuring their projec
t’s teams and
bug lists. Only the server administrator can change the project administrator, as it
requires manual changes to the project’s configuration files. Requests for project
administrator changes are frequent as projects evolve and project administ
rators change
roles or go on vacation.

Overall, the solution has reached a support tipping point. The six servers require
constant care to manage their environment and the cost of configuring a new server is on
the order of four engineering days. While the

business can’t afford to take eNotify
offline, it would clearly benefit from an architectural renewal that would reduce the
support costs, simplify the mechanism for enabling sub 2
-
hour site refresh rates and allow
for future innovation of the business pr
ocess.



17

Chapter 2

Solution Requirements and Goals

The requirements for this project were identified through conversations with the
operations team and the end users. The requirements are focused on reducing or
eliminating time spent waiting for eNotify project webs
ite generation or costs associated
with manual load distribution.


Figure
4

Time to Complete Tasks
-

Legacy System vs Goals






18

There are six requirements designed to drive the savings as shown in Figure 4:

1.

Provide the ability
to increase a site refresh frequency with zero end user
down time. As described in chapter 1 this process varies, but the last
migration resulted in approximately 9 days of down time.

2.

Eliminate the site migration operations cost. This is related to require
ment
1, but focuses on manual labor cost to the operations team, which is also
variable and most recently resulted in a 9 day effort.

3.

Reduce the operations labor cost for adding capacity. The current cost of
adding and installing an eNotify server is 2 bus
iness days for one
engineer.

4.

Eliminate the 3
-
4 day delay for new site creation after a project manager
requests a site.

5.

Eliminate the manual labor cost for the operations team associated with
responding to new eNotify site requests and existing site mainte
nance
requests. Cumulatively these tasks cost the operations team approximately
4 person days of effort a month.

6.

Eliminate the 24
-
hour delay for graph updates. Graphs, including
projections, should update at least as frequently as eNotify websites
without
incurring additional rendering time at the generator.



19

Several solutions have been considered for resolving the issues with the existing
prototype. The following are the criteria for success that were used in choosing a solution
for this problem. Ties were

broken by the impacted criteria with the lowest number.

1) Requirements
:
Address the six requirements identified above.

2) Cost
: The solution must align with existing business return on investment
standards. In our current business climate there is a
push to leverage existing
infrastructure as opposed to investing engineering resources to achieve a goal without
that internal infrastructure. The cost of low
-
end hardware and virtual machine based
systems has dropped significantly. When attempting to achi
eve faster processing and
scalability it is cheaper and more politically palatable to leverage low
-
end hardware or
virtual machines then to invent a newer faster processing algorithm.

3) Simplicity
: The solution will be handed off to a team that has littl
e time to
manage it. The application needs to be relatively self sufficient and conceptually simple
to understand.

4) The right technology
: The initial prototype was designed using Perl because
there was an expectation that the complicated part of the app
lication would be in parsing
the configuration and bug data. While this turned out to be true, using Perl for the web
administration became excessively complicated. The skill set of the support team is not
well suited to maintaining a complex Perl based we
b application. For that reason, if there
is a need to rewrite the web administration, the technology choice for the re
-
implementation needs to be designed for web application development and must fit the
skill set of the team that will support the applicat
ion.


20

Solutions Considered and Rejected

The following solutions were considered and rejected after failing to meet one or
more of the above criteria.

Generate the Pages Dynamically

From a design perspective it makes sense to question the a priori nature
of the
page generation. It’s likely that a large majority of the generated pages are never viewed.
In the original design I decided to implement
a
priori page generation due to the
underlying data source and eNotify’s original business goal
s
.

The under
lying bug data source is a flat file storage mechanism accessed via CGI.
Response times on queries range from 30 seconds to several minutes and are returned in
text format. The meta
-
data required to apply an ownership model, the cornerstone of
eNotify’s fu
nctionality, is not available via the bug data source. Thus, in order to create a
combined data model the bug data would need to be imported into a new database and
merged with the ownership model meta
-
data.

While on demand page generation is possible the
re are several issues that prevent
its implantation in this project.

1.

At the time the legacy application was written, scalability was not a design
concern. Rather than adding the complexity of replicating the bug data in a
relational database, generating pa
ges a priori was a simple solution given the
toolset I had available.

2.

Leveraging the existing data source allows eNotify users to utilize their
knowledge of the existing proprietary bug query language for defining bug lists.

21

Thus no training or mapping is
required and bug queries work across corporate
tools.

3.

There is a multi
-
year corporate effort under way to migrate the existing bug
database to a model that would support on demand page generation. In this
context eNotify is seen as a 4 to 5 year stopgap to
ol.

4.

The existing mechanism for moving bug data from the source through the
ownership model and into the UI is extremely stable. It has been recognized as a
best practice for bug data sourcing.

5.

In light of the previous points the business would like to limi
t its development
investment and instead focus on reusing the existing code
-
base wherever possible.


On demand page generation is clearly the purest technical solution. In this case
that solution is sub
-
optimal when combined with the known business factors
.


Buy a Bigger Server

A 64 bit quad processor server with a 500GB RAM disk and 64GB of memory is
capable of running all of our existing eNotify websites in parallel without minimal code
changes to accommodate parallel processing. This solution allows us

to meet our
immediate scaling needs and centralize all sites onto a single server, thus eliminating the
multiple URLs and inability to share configurations. There are four major issues that
eliminate this strategy as a viable option:


22

1)

The business is unwi
lling to foot the $24,000 bill to purchase one of these
servers.

2)

One server would represent a hard to replace single point of failure.

3)

At our current pace, we would surpass the capacity of such a machine
within 6 months.

4)

This solution does not address t
he high cost of adding features via the
existing administration framework.

Make the Site Generation More Efficient

The existing system could be optimized to make better use of the existing
resources. Smarter algorithms for applying ownership and render
ing pages could make it
possible to extend the scalability of the existing servers. It would be possible to move to a
dynamic page generation model where pages were rendered only when a user requested
them. More efficient or dynamic generation algorithms w
ere considered during the
original prototype phase as well as during this latter scaling resolution phase. This avenue
is appealing for two reasons. First it eliminates computer cycles being wasted on pages
that are never viewed and second it provides for
an interesting thesis topic. From the
business perspective it has some major issues.

1.

The high engineering cost of rebuilding the infrastructure for a working
system from the ground up is hard to justify in the presence of other
options.

2.

The solution comp
lexity would increase dramatically which typically
translates into higher support and maintenance costs.


23

3.

As load increases, processing power for page generation will continue to
be an issue. This could be addressed through distributed web server load
bala
ncing, but that only increases the complexity of the solution.

4.

There is a simpler and cheaper solution.

Each of these solutions was rejected due to its inability to meet one or more of the
project requirements. The following chapter details the selected
solution.




24

Chapter 3

Project Description

The solution chosen to resolve the shortcomings of the legacy
application

involves refactoring both the application and its deployment model. This
approach requires re
-
architecture of the administration web application, ret
aining the site
generation logic and leveraging existing corporate infrastructure to enable a more flexible
deployment model. There are three distinct steps that when viewed as a whole provide a
cheap, scalable, and flexible solution.

Three Part Solution S
trategy

1. Simplify the Website Generation Engine

The goal of this step is to take the site generation capability of the existing system
and decouple it from the configuration and business logic. The legacy site generator
consumes project configuration da
ta from a variety of flat files saved by the
administration web server, to its local file system, and then renders generated pages to
that same file system. The site generator will be altered such that it polls a web service
for site update instructions an
d configuration data and renders pages to a NFS (Network
File System (Red Hat, 2005)). This will allow the site generator to be hosted on a
separate server from the administration server. Site generator servers are configured to
mount an additional NFS dri
ve containing the generator code such that the complete
setup of a generator consists of mounting a network drive and configuring a cron job to

25

call the generator code. Multiple site generation servers can be brought online to expand
the scaling capacity o
f the system.

This solution essentially creates a cloud of computing power that is applied to all
of the eNotify sites. By decoupling site generation from a specific server the concept of
server specific refresh rates, as seen in Figure 2, are eliminated.
Load distribution is
automatic across multiple generators. This eliminates the need for manual load
distribution, the resulting site down time, and operation labor costs associated with site
migrations across servers.

2. Implement a Central Controller

The
existing administration console will be retired and re
-
implemented as part of
a centralized administration and configuration management system. The intent is to use
the Ruby on Rails (Hansson, 2005), full
-
stack (Bennett, 2006) MVC (Model View
Controller) w
eb framework as it allows for rapid feature development, and is a
technology with which the support team is familiar.


The central controller will provide
all of the features of the previous administration console as well as:



A stable and flexible web int
erface for creating and editing project configuration
data



A single repository for all site data



The business logic that determines when sites update and in which order



A set of web services (World Wide Web Consortium, 2007) that provide:

o

Access to the

business logic in the form of a poll
-
able service that gives
generators instruction to generate a project website

o

All the configuration information needed by a website generator to build a
project website

o

A mechanism to report acceptance of a site gener
ation task


26

o

A mechanism to report website generation progress as well as error and
success data



A mechanism to manage and track numerous generators



Support for multiple site administrators per project



A pass through mechanism for generated page requests

such that requests for both
the web administration, and the generated project pages, can be served up from
the same domain.


This creates a central point of
administrator and end user
control for the system
housed on a single server.

3. Implement a Distr
ibuted Deployment Model

The deployment model is the linchpin to addressing the requirements while
making this solution operationally scalable. The diagram below shows a high
level
view
of the intended eNotify system deployment
.



27


Figure
5

Proposed eNotify Deployment Model


Site generators are hosted on a set of identical virtual machines that are easily
replicated. The website generators are configured with a repeating cron job to call out to
the central controller web service re
questing the name of the site to update. Each
generator has an NFS mount to a shared network drive, which it uses for accessing the

28

eNotify generator application and for storing generated web pages and downloaded bug
data.

The central controller web appli
cation is hosted on a dedicated server. The central
controller manages all the business logic determining the order of sites updating. When
queried by a generator, the central controller determines which site should be updated and
returns that site’s confi
guration data to the generator. Site update order is based on a
number of criteria including desired refresh frequency, last update time, user defined
priority of each site, and administration override priority.

Users access both administration as well as

generated pages through the central
controller. Project site administrators use the central controller web application UI to
manage their projects. The central controller also has a web server configured to serve up
generated pages from the mounted NFS dr
ive. This provides a single URL for accessing
both the administration UI and generated project websites.

From a cost perspective this solution benefits from the ability to leverage existing
hardware resources as well as a smart use of engineering investme
nt. The NFS is already
in place at my company and is a cheap storage solution easily capable of scaling to our
future needs. By simplifying the site generators to be processing drones, we are able to
leverage any cheap hardware or virtual machine by simply

mounting the network drive
containing the generator application, and letting the central controller manage its
behavior. The development of a new administration web application in a Ruby on Rails
MVC framework allows for reduced cost of future development

while not overburdening
engineering with an excessive development task. Ruby on Rails (Hansson, 2005) is
specifically designed for development of web applications and its use of intelligent

29

defaults and standards significantly reduces development time ove
r other languages and
frameworks such as PHP (PHP Group, 2005), Java/Hibernate (Red Hat Middleware,
2006) and Perl/Mod CGI.

While the solution leverages a number of technologies, each one is used in a
relatively straightforward way. The decentralized natu
re of the solution allows us to
easily scale the application in the future simply by cloning one of the existing site
generation drones. The central controller provides
a common point of entry
for access to
generated eNotify websites

as well as system m
onitoring and maintenance control. The
solution benefits from component level simplicity that should translate into reduced
maintenance costs.

Solving the Graph Generation Problem

While the majority of the legacy shortcomings are addressed through the thr
ee
-
stage re
-
architecture and deployment change, the graphing issue remains. The two issues
with graphs are the:

1.

Extremely high generation to use ratio.

2.

Out of date data when projections are changed.


These issues are addressed by eliminating the graph
-
ren
dering step from the generator
and rendering graphs on demand via the controller. The controller has the projections in
its database, thus projections are updated each time the graph page loads. This has the
benefit of eliminating all generator processing
time for graph rendering, and eliminating
the class of requests for system administrators to manually override graph generation.


3
0

Languages, Environment and Straying From Standards

Ruby on Rails

The central controller is implemented in Ruby on Rails using
a MySQL database
(Sun Microsystems, 2009). Ruby on Rails is a combination of the Ruby (Thomas, 2005)
language coupled with a full stack MVC framework called Rails. Ruby on Rails
simplifies the development task by nearly eliminating the need to repeat confi
guration
steps in the different parts of the application stack. While the designers of Ruby on Rails
favor convention over configuration they still give full control to the developer to
override convention when necessary. The conventions used in Ruby on R
ails are
extremely well thought out and will save time and effort in future development.

The decision to use the Ruby on Rails framework with MySQL was based on two
main factors.

1.

I’ve developed a preference for using Ruby on Rails based on experience
dev
eloping web applications in C, Perl, Java and Ruby on Rails.

2.

The operations team has a strong skill
-
set and existing application
experience with Ruby on Rails and MySQL. Their ability to successfully
support and enhance the application in the future withou
t adding
substantial cost is enabled by use of a programming language and tools
with which they are already familiar.

These factors make it an excellent technology choice for the development of a
web application such as the central controller.



31

Perl

Per
l development is necessary for the conversion of the existing prototype from a
full web app to a drone generator as well as to migrate the existing configuration data into
the central controller database. During the proposal stage this decision was based o
n the
assumption that it would be easier to carve a drone generator out of the existing eNotify
code, and build a web service based configuration data fetcher in Perl, than it would be to
rewrite the generator in Ruby. The basis for this assumption was the

existence of a
functioning generator and proved to be true during development prototyping.

Web Services

All remote programmatic communication is implemented using standard web
services. REST (Hass, 2005) was chosen as the communication protocol as it wa
s
essentially made available for free by the Ruby on Rails stack.

Development Environment

The original controller development was done on Windows using RadRails
(Aptana, 2006).


I experienced numerous problems with the Aptana IDE and logged two
bugs that
the Aptana developers are in the process of fixing.


Mid way through
development I switched the controller development environment to OS X and used
TextMate (MacroMates, 2009). The minor updates to the generator code were done in
jEdit (Pestov, 2004) as I
was already familiar with Perl development in that IDE.




32

Hardware and Operating System Environment

Both the central controller and the generators are deployed on a farm of Red Hat
Linux Server images running on VMware ESX servers (VMware, 2007) in our c
orporate
data center.


The standard Red Hat (
R
ed Hat, 2005) virtual machine has NFS access to
our corporate file system enabled by default.

Due to the simplicity of the generator deployment, nearly any Linux based
machine can be used as a generator. While
virtual machine images are the easiest to
manage, the generator’s simplicity allows us the option to use any unused

low
-
end
servers. This provides flexibility to use any machines that are available to scale the
system in the future. The collection of avai
lable generators act as a cloud of computing
resources available to the application and easily scaled up and down to meet the needs of
the system.


Parting From a Conventional Rails Application

The conventional Ruby on Rails approach to solving the design

problem is to put
all of the bug data into the database and build a fully dynamic web application. This was
not the approach taken. In this solution, only the eNotify site metadata is stored in the
controller’s relational database while the bug and genera
ted data are stored in flat files,
per the legacy generator design. The decision to separate the data storage across
paradigms, in clear opposition to the conventional Ruby on Rails approach, was a critical
design decision and warrants explanation.


33

Challe
nges With a Relational Approach to Generator Data

Storing eNotify site data in a MySQL relational database means not only
replicating the corporate bug data, but also storing all of the historical data needed to
generate the graphs. The historical data is
a snapshot of the state of the bug counts,
through the lens of eNotify ownership models, at given points in time. Due to a limitation
in the corporate bug tracking system, historical data cannot be reproduced and must be
recorded in real time by eNotify. T
his puts a large data storage burden on eNotify. Based
on the existing graphing requirements the number of rows required for storing historical
bug count data for 1 year is roughly 11 billion rows.

(~100 active projects) * (~300 owners per project) * (~100
0
active bug lists ) * (1 year of active data ~365 days) = ~11
billion rows


Excluding the need to replicate the bug database, the requirement to store 11
billion additional rows of data each year in a new format represents a significant technical
challeng
e for a single MySQL database which has a limit of approximately 4 billion rows
per table (Zawodny, 2009). Sharding (Hibernate, 2009) provides a technical solution to
the problem by allowing a single table’s data to span multiple MySQL databases, but
brin
gs with it complexity that the operations team is not prepared to manage.

Investigation of other solutions, such as storing each site’s data in separate
databases or leveraging the capabilities of a mass storage non
-
relational database such as
Apache Couch
DB or Apache Hadoop all fail to meet the goals, described in chapter 2, in
two key ways:

1.

The Right Technology:
It requires a database and application scaling skill set
that is not part of the existing operations team.


34

2.

Simplicity:

It requires substantial ch
anges to, or even replacement of, the
generator code, which increases both the complexity, and the scope of the solution
without providing value in terms of the projects requirements or goals.


Additionally a standard Ruby on Rails implementation with all
of the data in the
database dramatically increases the scope of the development effort on this project. That
increase in scope includes both the application and database scaling considerations as
well as the development effort associated with alternating t
he generator.


Given that this
is a mission critical application and the generator has been generating flawless pages for
over 5 years, there is a substantial aversion to adding risk to the system by re
-
writing its
core functionality. In this case, re
-
imp
lementing the generator solution using the
conventional Ruby on Rails relational database approach would have resulted in a case of
over engineering a solution to a problem that did not exist.

The solution using a server farm of cloud based generators and

an NFS data store
provides a solution that leverages the robust parts of the existing application, the physical
resources we have available, and the core competencies of the support team that will own
this application in the future.






35

Chapter 4

Project Implementa
tion

The project implementation effort was made up of six major tasks:

1.

Controller web administration design and implementation

2.

Definition of the controller schema and migration of legacy data

3.

Communication between the generators and the controller

4.

Gener
ator support for the controller

5.

Dynamic graphing implementation

6.

Deployment


The project’s progress was tracked on a blog at http://thesisdev.blogspot.com/.

Controller Web Administration

The controller web administration user interface, henceforth web adm
inistration,
provides an interface for project managers to create new eNotify projects and administer
the ones they own. The logical starting page for the web administration is the project list
page which contains lists of the projects currently supported

by eNotify, a summary of
each project’s configuration and links to view more detail.


36


Figure
6

eNotify Web Administration Project Summary

From this page users also have the ability to create a new eNotify project, edit a
projec
t’s global settings (shown in figure 8), and view a project’s bug lists, teams and
administrators. For example Bruce, the project manager, would come to this page to set
up a new project. He would click on “Create New” at the top right of the projects tab
le,
and would enter project metadata such as the name of the project, the frequency with
which the site should update, and the start date that should appear on eNotify graphs for
this project.


37


Figure
7

eNotify Web Administration
Edit Project Settings


Once Bruce saves the project it is created in the database, but additional data is
required before a generator can generate this project’s website. The status for the project
displays in red as “Setup Required” on the Project list as

show in Figure 8. Clicking on
the “Setup Required” status brings Bruce to the project’s summary page where
instructions are provided to walk him through the remaining setup steps.


Figure
8

Project Summary with Setup Instructions


38

Teams List

One of the steps Bruce must complete to set up a new eNotify site is the definition
of the people that can be bug owners on the project. eNotify supports the concept of a
single level team hierarchy for owners on a project. Thus, an eNotify pr
oject has a set of
teams and each owner belongs to a team.
Teams are identified by the team manager’s
name. A manager can only manage one team per project
.


Bruce accesses the Team List page by clicking on the “Team Count” column in
the server summary. T
he Team List provides a simple list of the teams in this project.
From the Team List page Bruce can click on a manager name to see the list of owners on
a team. As this project’s administrator Bruce has the unique ability to add a new team or
delete an exi
sting team using the links on the right side of the Teams List table


Figure
9

eNotify Web Administration Teams List

Adding Teams to a Project

To add a new team to his project Bruce clicks on the “
Create new team


link. He
is pro
mpted for the username of the manager of the team and has the option to have that
manager’s employees, as defined in the corporate LDAP (IETF, 2006) server, added to
the team. The eNotify controller is connected to the corporate LDAP system to provide

39

auto
mated checking for username validity and to enable the auto
-
population of manager’s
employees into an eNotify team.

Usernames are the unique identifier used by the corporate bug tracking system,
thus it is critical that all managers and team members are ex
isting users in LDAP. For
that reason, the controller automatically looks up the manager user name Bruce provides
in the corporate LDAP system to make sure that person exists. A team is only created if
the user name provided exists in the corporate LDAP se
rver.

Bug List

Each eNotify project has a set of bug lists. A bug list is a logical construct used to
define a group of bugs. Bugs for a given list are identified with a proprietary SQL