an Enterprise Environment

Using
Atlassian

Products in

an Enterprise Environment


Mark McCormack

British Telecom

Inlife Support Lead

Development Services Organisation

© British Telecommunications plc


Outline

•
Using
Atlassian

products in an Enterprise

•
Atlassian

Product adoption

•
Size of Confluence and
Jira

in our Enterprise

•
Some
Atlassian

Use Cases

•
Customisations

•
Application Monitoring

•
Platform

•
What’s Next

•
Product
Wishlist

•
Some Lessons Learnt

•
Q & A

© British Telecommunications plc


About

•
Network is available in 173 countries.

•
Around 3,000 new customer sites every month

•
Carry 80% of all credit card transactions in the UK.

•
More than 90% of the UK’s financial institutions use our secure data networks

•
Rolling out super
-
fast broadband to 40% of the UK by 2012.

•
FTTC downloads of 40Mb/s and uploads of 2 Mb/s.

•
FTTP downloads of 100 Mb/s

London 2012 Olympics

© British Telecommunications plc


Development Services Organisation

•
We advise on best practices for all tools

•
Inlife team looks after all hosted applications:

–
We are an Atlassian shop

–
Subversion
,
Jive Forums
,
Nexus

–
Other in
-
house tools

•
Authz
,
Teams
,
IPPI
,
Provisioning
,
Static Data

& PALMS

•
SDK Catalogue

(will soon become BT Enterprise Repository)

© British Telecommunications plc


Product History

Dec 2003

•
5 people created a team (called code red) but there was no way of coordinating their efforts

•
Bought Confluence and Jira and installed it on a PC and put it in a meeting room


June 2004

•
Usage spread to a few development teams (200 users)

•
Bought new hardware and installed it in a server room (engineering.intra.bt.com)


April 2006

•
Another group installed a separate instance of Jira and Confluence to manage service
improvements


February 2007

•
Management Services and Engineering decided joining forces would be a good idea

•
Userbase reached 300 and 2000 respectively

•
http://collaborate.intra.bt.com was born and hosted by VS Group on VMWare.


July 2007

•
A Virtual Administrator crew formed out of 35 volunteers from the userbase to perform end
user support and administer the platform

•
Two users in particular did most of the Admin work. Userbase grows to over 9000 people.

© British Telecommunications plc


Product Maturity

August 2007

•
Sponsors found to pay for new hardware & uplift plan created

•
Team split started to disrupt progress to achieve this and ownership was contested



September 2007

•
Userbase grows to 10,000


October 2007

•
I/O Capacity of the disks in the VSP environment started to be exceeded.

•
Platform started to have outages with huge traffic increase


January 2009:

•
Moved to an enterprise hardware infrastructure, Oracle 10G RAC & 24x7 support

•
Collaborate Wiki

was formed, Confluence 2.10 and integrated with Crowd


February 2009:

•
Introduced
Storm

&
Jira Issues


July 2009:

•
Started to assimilate other wikis (like twiki, rdrwiki, ProfComms)

© British Telecommunications plc


Confluence Numbers

•
800,000 pages

•
102,000 attachments

•
3000 spaces

•
25,000 display requests per day

•
30,000 actions per day

•
(
eg

login, edit, comments,
rss

feeds etc)

44,000

13,000

Users
Contributers
Old,
700,000

Current,
100,000

Pages

Current,
75,000

Old,
27,000

Attachments

© British Telecommunications plc


Numbers

•
3
flavours

of
Jira

•
ACF
–

Architecture Conformance Framework

•
Issue Tracking (100 projects)

•
Story Management
Operating Model

•
BT is adding > 20,000 stories every month



4,500

This
Week
1,000

5,500
Users

2,000

8,250

8,000

7,800

2,000

83,000

110,000 Stories

20

20

60

300

1

400 Spaces

© British Telecommunications plc


Confluence Use Cases

•
Tool Guides on how to get started

•
Tool Access Request Forms

–
We have a teams application we encourage before requesting a Space

–
Early stages of a Provision Application

•
Application Deployment Schedule

•
Collaboration between Dev and Inlife Teams (geographically dispersed)

•
Don’t send emails but rather send Confluence links

•
Trying to have an FAQ before raising an issue for help

•
Application Usage Statistics

•
MyBT has a dashboard interface and makes use of the Confluence remote API and
render as html inside a portlet.

•
Most of the Ajax related actions don’t work


© British Telecommunications plc


Confluence Customisations

•
Implemented the functionality of security classification requirements in the scaffolding
plugin.

•
To implement this functionality the edit page / do edit page actions of the confluence
needs to be overridden. Scaffolding plugin was already overriding the edit actions of
confluence.

•
Confluence doesn't allow two plugins to override a particular action.

•
Extended the edit actions to include classification

© British Telecommunications plc


More Confluence Customisations

•
https://collaborate.bt.com/staticdata

plugin to provide a LOB picker from LDAP

•
Static Data Macro that retrieves platforms

•
BT Space Admin Confluence plugin makes use of our eorg DB in order to get the Line
of Business functionality associated with a Space

•
Modified the Excel Plugin as it was hanging Confluence a lot

–
We had one user with a page that had 48 references to the excel macro

–
Every reference was an API call that forced a GC, so we disabled explicit GC

–
Page went from 14 minutes to load to about 4 seconds


© British Telecommunications plc


Customisations

•
Hierarchy plugin

•
Static Data plugin to strip LOB info out from LDAP

•
Loads of constraints on Story creation and workflow

–
All Stories must have at least one child and at least on Acceptance Criteria

–
Organogram

•
Added Active User Report under Users, Groups and Roles

•
Roles for Storm

•
Loads of different workflows (put issues On Hold whilst with Dev team or Vendor)

© British Telecommunications plc


More Customisations

•
https://collaborate.bt.com/fisheye

plugin to allow the authorisation of SVN repositories
(handled by hasPermissionToAccess method which is overrided to authorise against
our Authz).

•
Use
https://collaborate.bt.com/crowd

client libraries for apps like
https://collaborate.bt.com/forum

•
We have a Jira Project that helps us keeps track of Subversion Projects ensuring
everyone has a Salsa ID.

•
Unified Rolling Calendar Plugin

© British Telecommunications plc


Application Monitoring

•
ZXTM

•
Nagios alerts via email for slow (or dead) apps.

•
JMX is enabled for most of our java apps

•
Disk space (e.g shell script to alert > 80%)

•
Backup successes or failures uploaded to our wiki

•
Access logs with IP tracking

•
Usage statistics to figure out what our users are doing

•
Perl script we run for snapshots of ugly apps

•
YourKit Profiling from within IntelliJ

•
kSar

© British Telecommunications plc


Platform

•
Currently 18 HP Blades mostly BL460s with 2 Quad cores and 32GB of RAM for our
existing hosted application servers

•
RHEL 5 (soon OEL)

•
Were using
Weblogic

10.3 but now Tomcat 6

•
Java 6.0.18

•
ZXTM 5.0

•
Oracle 10g in a Cluster hosted by another team on HP Blade 680s with 2 Quad cores
and 64GB of RAM

•
Crowd is our only clustered application (currently)

•
We use Capistrano to help automate our deployments and allows easy rollback

© British Telecommunications plc


What’s Next?

•
Clustering of Confluence,
Jira

and SVN

•
Triple our hardware for Disaster Recovery

•
Archiving & stats analysis
plugins

(
customised
)

•
Improve provisioning

•
All
the
Atlassian

2s (Bamboo, Crowd, Fisheye, Crucible) although upgrading to
Fisheye 2.0 will be expensive on disk space.

•
Start charting long
-
term trends with better monitoring

•
Assimilate all wikis (like
LimsWiki

for one)

© British Telecommunications plc


Wishlist

Confluence Wishlist

•
Tighter integration with Jira

•
Allow Jira Issues to be raised via Confluence and vice versa

•
Allow profiles to have attributes (like team, group or project)

•
Confluence attachment storage API. We want to store attachments away from
Confluence and write our own adaptors to do this.

•
Improve the edit permissions cascading security

•
Customisations of actions (eg. Overrides to allow > 1 plugin)


Jira Wishlist

•
Allow two columns for attributes

•
Allow picklists to show more values without scrolling

•
Smart forms that can include wiki macros

•
How can we hide Administrators from assignee list?

•
Can Greenhopper scale? H
ow compatible is it with existing 3.13.2
plugins/customisations?

© British Telecommunications plc


More Wishes

Jira …

•
Would like to use our Issues as a source for FAQ before users raise an issue?

•
Ought to be Confluence tightly coupled with Jira Issue labels

•
Timebooking to allow resource Management?


Others

•
Fisheye 2 IE6 browser support

•
Make Crucible a viable app for the enterprise (we have some open Atlassian issues for
10k+ users)

•
Can we turn on “remember my login on this computer” checkbox during login?

© British Telecommunications plc


Lessons Learnt

•
Be wary of any
plugin


–
Excel Macro
plugin

–
We turned off the usage tracking
plugin

–
We trialled the stats analysis
plugin

but its currently off since we got hammered
two lunchtimes in a row shortly after going live

•
WebLogic

seems to be a lot slower than tomcat

•
Confluence 3.x is much better than 2.x

•
We had problems with Crowd 1.6 client and currently use Crowd 1.3 client

•
Tune the JVM heap

–
Confluence
: we use java
–
server
–
Xms:8G
–
Xmx:8G
PermSize
=512m
MaxPermSize
=512m +
UseParallelGC

+
AggressiveHeap

LargePageSizeInBytes
=4M
-
DisableExplicitGC


–
Jira
: we use
–
server
–
Xms4G
–
Xmx4G
PermSize
=1G
MaxPermSize
=1G
+
UseParallelOldGC

+
AggressiveOpts


–
Crowd
: we use java
–
server
–
Xms:4G
–
Xmx:4G
PermSize
=512m
MaxPermSize
=512m


–
Fisheye
: not really tuned yet
–
Xms:4G
–
Xmx:4G
MaxPermSize
=512m

Questions

mark.j.mccormack@bt.com

British Telecom

Inlife Support Lead

Dev elopment Serv ices Organisation