Change Management Procedures
Document No:
ITS
-
NOC 300
Title:
Change Management Procedures
Month Year
May 2011
Doc. Type
MS Word
File name:
ITS
-
NOC 300: Change
Management Procedures V4.1
_.doc
Change Management Procedures
-
2
-
Classify by
urgency
,
impact
, and
risk
.
Pre
-
RFC Checklist
Start
The Change Process
How to make changes to
document
Urgency
High ?
Get
Service Manager/
Customer Approval
Schedule
using
maintenance window
N
o
Needs
CAB
approval?
?
Determine appropriate
approver
Alert Change
Manager to
emergency change
N
o
Communicate
with
Customers
Log
change request.
Send
RFC
to CAB
Perform
Change
Test change
User OK?
CAB OK?
Y
e
s
Y
e
s
N
o
Yes
Analyze/resub
mit
2
Yes
N
o
Analyze/resub
mit
Change Management Procedures
-
3
The Change Process Part 2
No
Back Out
Successful?
No
Yes
Yes
Change
Successful?
Stop
Close RFC
Perform
Critical Incident
Review (
Post Mortem
)
2
Analyze Failed Change
-
Complete
Documentation
updates
-
Schedule Training
(User, Staff)
Change Management Procedures
-
4
-
Tabl
e
of Contents
Contents
Summary
................................
................................
................................
..........................
6
Goals of Change Management
................................
................................
.........................
7
Change Classifications
................................
................................
................................
.....
8
1) Urgency
................................
................................
................................
...........
8
ChangeLog
................................
................................
................................
.......................
9
2) Impact
................................
................................
................................
..............
10
3) Risk
................................
................................
................................
..................
10
Example of Change Types
................................
................................
...................
11
Forward Calendar of Changes
................................
................................
.........................
11
Maintenance Window
................................
................................
................................
......
12
By Service
................................
................................
................................
............
12
Guidelines for Selecting Upgrade Windows
................................
........................
13
Change Approval Matrix
................................
................................
................................
.
14
Approvers Table (low and medium risk changes only)
................................
.......
14
Activities and Deliverables
................................
................................
..............................
15
Approval Procedures
................................
................................
............................
15
The Requ
est for Change (RFC)
................................
................................
.......................
16
How to
................................
................................
................................
..................
16
Lead Time for Approval
................................
................................
................................
..
17
RFC Submission Fields
................................
................................
................................
....
18
Customer Groups Communication/Service Bulletins
................................
......................
20
Forward Schedule of Changes (FSC)
................................
................................
..............
21
Emergency Change Process
................................
................................
.............................
22
Closing the RFC
................................
................................
................................
...............
22
Change Assessment
(Post
-
Implementation Review)
................................
.......................
24
Glossary
................................
................................
................................
...........................
25
Appendix 1 CAB Contact List
................................
................................
.........................
27
Appendix 2 Pre
-
RFC Initiation Process
................................
................................
..........
28
Appendix 3 Critical Incident Reviews (aka Post
-
Mortem”) Guidelines
.........................
33
Appendix 4 Examples of changes
................................
................................
....................
34
Appendix 5 Examples: Communications Strategy
................................
.........................
36
Change Management Procedures
-
5
Version
4.0
4.0 November 28, 2009
Review of initial documentation
Change Management Procedures
-
6
-
Summary
Change Management applies to all changes to
Applications,
Databases, Networks, Infrastructure and Documentation.
All changes are recorded and classified by their:
service
impact
risk
urgency
Who can initiate changes:
Each change is sponsored by a Manager who is responsible for the
planning and implementation of the change, and any recovery.
Certain changes are reviewed and approved by a Change Advisory
Board (CAB). Those changes
include all High Risk changes and
changes that meet the criteria illustrated in the
Change Approval
Matrix.
The CAB meets weekly with the sponsoring Manager(s)
and approves changes for the following week, and beyond.
Highly Urg
ent (Emergency) changes require separate approval.
Unsuccessful changes
that are successfully backed out within the
change window are followed up with an Analysis of Failed Change
by the sponsoring managers.
Unsuccessful ch
anges that are not successfully backed
out within
the change window are followed up with a CIR (Critical Incident
Review( aka Post Mortem by the Director, IT Operations
or
designate.
Frequency
of CIR Review
Reviews are conducted by the Operations
Committee at the
bi
-
weekly Tuesday morning meetings at 09:00am
in Rm443.
Meeting is
chaired by the Director of Operations
or his/her
designate.
.
Change Management Procedures
-
7
Goals
of Change Management
The goals of the Change
Management Procedures are to:
Minimize disruption caused by changes to the production
environment through effective risk management.
Assign ownership and responsibility for changes to the relevant
manager.
Optimize the change process through active m
anagement of
reliable data on changes.
Manage all changes to the production environment including
hardware, software, environmental equipment, networks, and
procedures.
Keep users informed of changes to the production environment and
manage their
expectations.
(This is also
an opportunity to promote
and strengthen client communications.)
Keep
UBC IT
staff aware of changes to the production
environment and increase the number of
UBC IT
staff reviewing
changes.
Develop a more pro
-
active approach
to changes through
encouraging better planning of changes, back outs and
contingencies.
To measure success and success trends over time.
Key Success Factors
A CM process used by staff who see it as an enabler
Control over urgent changes
Correct
scoping and staffing of CM
Knowledge of changes required for good decisions
A CM process aligned with the project lifecycle
Visible management commitment and support to enforce process.
Change Management Procedures
-
8
-
Change Classifications
Changes are classified according to:
1.
Urgency
2.
Impact
3.
Risk
4.
Nature of Application.
These changes are classified by the change initiator, and checked
by the CAB.
1) Urgency
Describes how quickly the change must be implemented.
High
(Emergency Changes).
The
two changes
that fall under this
category are: Break/fix:
required
immediately
–
a system or service has failed and is not available
and
cannot wait for the
normal change approval process and/or next
scheduled maintenance period
.
The other
type of emergency change
is
urgent and immed
iate action is
required
to a
vert a system or service outage.
High Urgency changes are considered as emergencies and are
subject to a separate Emergency Change process
. Approval is
granted as noted in the Emergency Change Process section.
For Break/fix
situations and the Sponsor deems a Change is
required, an Emergency Change can be done without approval
received prior to the change but should submit an RFC as soon as
possible after the change.
Medium
A normal change which will be processed by the CAB a
nd installed in
the next maintenance window.
Standard
aka Change Log
A standard change is one that is effectively pre
-
approved and can readily
reference pre
-
defined workflows
–
these can be entered as a Change Log.
A standard change is a change to th
e infrastructure that has the following
characteristics:
recurrent, well known and proven
pre
-
defined, relatively risk free
is the accepted solution to a specific requirement or set of
requirements.
Approval is given in advance.
Does not require CAB appro
val but is subject to Post
Implementation Review by the CAB.(PIR)
Change Management Procedures
-
9
Low
A change that can be deferred and/or grouped with other changes in a
subsequent release.
ChangeLog
A “change log” is defined as a standard change and does not require CAB
approval.
These standard changes ae well understood, recurring, low risk change,
redundant, accepted response to a specific requirement of set of
circumstances, the approval for which has been delegated to the group
making the change.
Examples:
Installation of S
SL certificates / patches
Scheduled workload to new TWS server
Cycling of LDAP instances
Ongoing alumni production db maintaince
Symposium call pilot integration
Change Management Procedures
-
10
-
2) Impact
The change impact is determined by the number of users who
could be affected b
y a reduction in their normal service level. This
takes into account the worst case where a change does not
succeed.
High
Change effect covers the campus.
Medium
Change effects roughly a department or building.
Standard Change
There is no anticipated
user impa
ct, for example, a change to a redundant
system.
A standard change is one that is effectively pre
-
approved and can
be entered as a Change Log. See examples:
Low
Change effects a single user or small number of users.
3) Risk
Describes the ris
k
*
that the change will not be made successfully.
High
There is a higher than normal risk that the change will have to be backed
out, or that a system failure will occur after the change is installed.
Medium
A change where success is the expected outcome
, but the change is not
routine, and could require a back
-
out.
Low
A routine, proceduralized change, with little chance of failure.
*Risks are managed through back
-
out and contingency planning, and all changes with high risk require
approval of the CA
B. The Change Owner of a high risk change is expected to demonstrate back
-
out &
contingency plans consistent with the level of risk.
Change Management Procedures
-
11
Example of Change Type
s
Admin
FMIS, HR, Alumni, Student systems
Elearning
WebCT
Enterprise Middleware
CWL, LDAP,
DNS servers
Enterprise Applications
Email, Mercury, uPortal, UBC main web page
Forward Calendar of Changes
Check the forward calendar of changes for conflicts with your
proposed change. This calendar is found on outlook.
See example below.
Change Management Procedures
-
12
-
Maintenance Window
By Service
The maintenance window is determined from the following table. Note that the maintenance
windows in the table are meant to cover backout if required, i.e. in a 6:00 am to 8:00 am
window, the change would normally be done from 6:00 am to 7:00 am.
If the change required
exceeds the allotted time, then the change should begin earlier, rather than impinge on the
contingency time.
Please note that at this time
–
November 2009
–
this p
rocess is being revised and a draft
document is being proposed.
Service
Upgrade Time
Contingency Time
WebCT
(Effective July 1,
2005)
Saturday with an expanded
time frame between 08:00
and 12:00.
Saturday
11:00am to 12:00am
HRMS
Weekend 06:00 to 9:00
Weekend 07:30 to 09:00
PeopleSoft
Saturday, 06:00 to 09:00
Saturday, 06:00 to 09:00
Change Management Procedures
-
13
Guidelines for Selecting Upgrade Windows
If no maintenance window has been established for a service, use the following table to select a
potential window. (
the
default time is 06:00
–
08:00am Monday to Friday when not stated)
Change Type
Impact
Upgrade Time
Contingency Time
Admin
applications
All
Outside business hours and
with agreement of customer
Outside business hours and
with agreement of customer
Cable system
All
Set by customer
Set by customer
Elearning
All
06:00
–
〸㨰ち洠⁓a瑵牤ty
07:00
–
08:00 Saturday
Enterprise
middleware
Medium
or Low
06:00 to 07:00 weekdays
07:00 to 08:00 weekdays
Enterprise
middleware
High
06:00 to 07:00
weekends or
holidays
07:00 to 08:00 weekends or
holidays
Enterprise
applications
Medium
or Low
06:00 to 07:00 weekdays
07:00 to 08:00 weekdays
Enterprise
applications
High
06:00 to 07:00 weekends or
holidays
07:00 to 08:00 weekends or
holidays
Network
Medium
or Low
06:00 to 07:00 weekdays
07:00 to 08:00 weekdays
Network
High
06:00 to 07:00 weekends or
holidays
07:00 to 08:00 weekends or
holidays
Security
(firewalls)
Medium
or Low
06:00 to 07:00 weekdays
07:00 to 08:00 weekdays
Security
(firewalls)
High
06:00 to 07:00 weekends or
holidays
07:00 to 08:00 weekends or
holidays
Voice
All
05:00 to 07:00
07:00 to 08:00
Change Management Procedures
-
14
-
Change Approval Matrix
Approvers Table (low and medium risk changes only)
I
MPACT
Service Type
None
No Impact
Low
Single Users
Medium
Department
High
Campus
Admin applications
Group
Group
Group
CAB
Cable
Group
Group
Group
CAB
eLearning
Manager
CAB
CAB
CAB
Enterprise Operating
Systems
CAB
CAB
CAB
CAB
Enterprise middleware
CAB
CAB
CAB
CAB
Enterprise applications
CAB
CAB
CAB
CAB
Network
Manager
Manager
Manager
CAB
Security (firewalls)
CAB
CAB
CAB
CAB
Voice
Manager
Manager
Manager
CAB
Infrastructure
CAB
Firewall Rule Changes
CAB
Note: all High Risk Changes require CAB approval.
Change Management Procedures
-
15
Activities and
Deliverables
A
pproval Procedures
Who can approve:
Determine from the
Approvers table
whether the change can be
approved by your group, manager or theCAB.
Group or manager level?
If group or manager level approval is ind
icated, then follow
approval procedures for your own group.
If CAB approval is required
Complete the RFC web form at
http://www.
UBC
IT
.ubc.ca/change.html
This webform
will send a message to NOC staff, who will open an
RFC in Magic. The newly created Trouble Ticket number
generated will be sent to you for further referencing. (for example
to relay success of change.
Change Management Procedures
-
16
-
The Request for Change (RFC)
How to
Get approval
by CAB
Enter the RFC on the webform at:
http://www.
UBC IT
.ubc.ca/change.html
This webform will send a message to NOC staff, who
will open an RFC in Magic
trouble ticketing system.
.
The newly c
reated Trouble Ticket number generated
will be sent to you for further referencing. (for example
to relay
status
of change
or other explanations.
CAB meetings
schedule
The CAB meets Tuesday afternoon at 1:30 pm in Rm443.
Deadlines
RFCs to be considered
by the CAB need to be entered by 23:59 on
Monday evening.
Change to be implemented on the
subsequent
weekend must be
sent to CAB by Monday (up to 23:59)
Decisions wil be sent out by Tuesday afternoon following the CAB
meeting.
Change Owner
responsibility
All changes must be preent to the CAB by the change owner or
designate in the event that the
re is a need for clarification
regarding the proposed change.
RFC change “logs”
It is possible that you may want to
“log” but not require a change
to be approved by
the CAB.
a.
Ensure that the
criteria
for the log has been met.
b.
Then “Send RFC to Log” instead of sending “RFC to
CAB” when completing the webform.
How to comment on/change RFC
Do a “reply to all” from
the original message that the NOC staff
had sent to you notifiying you of RFC reference #.(which in fact is
a Magic trouble ticket)
Closing the change
Change Management Procedures
-
17
All ITS staff have access to Magic and have IDS. Find the RFC
number in Magic and add an action item i
ndicating the outcome of
the change: Successful, Cancelled, Backed Out, Failed.
If you do not have a Magic ID (i.e.: you are a new employee or a
consultant, for example)
–
send a message to
ops5@exchange.ubc.ca
containing the above status information of
the RFC and they will enter the completion status on your behalf.
Lead Time for Approval
Lead time
The lead time required to make a change depends on the SLA in place
for the service(s) interrupted.
In the absence of an SLA, the service manager or customer can set the
lead time for approval. Without an SLA or Service Manager or customer
input, the CAB requires the following minimum notice: An RFC
submitted on Monday can be scheduled for the subsequent
Saturday
morning or after.
There is a 72 hour window that is required for customer
notification once change has been approved.
A change that cannot wait for the next CAB meeting can be marked
Urgent, and may be approved through the emergency change
proc
ess.
Change Management Procedures
-
18
-
RFC Submission Fields
Subject
A short description of the change which will appear on the email
subject line.
Contact Name
The name of the RFC submitter.
Change Owner
The name of the
UBC IT
manager responsible for the change.
Contact email
Email
of the RFC submitter.
CC
Email addresses to which the change will be sent (in addition to the
CAB).
Sponsor Name
The name of the service manager or group manager who accepts
responsibility for the change and its outcome. The change sponsor
is normally a
member of CAB, represents the change to CAB
during its discussion, and is the point person if the change fails.
Note that the sponsor must not be
away at the time of the change or
as required, have a designate in place.
Routing
This specifies whether the
RFC will be circulated to CAB (the
usual case), or just logged in Magic.
Logging straight to Magic is appropriate when the RFC has been
approved at group manager level.
Proposed date, start time, end time
The date and time for the change will normall
y fall into one of the
maintenance windows listed under Maintenance Windows.
Reason for Change
Overall description of the change. (Is there a need to use specific
terminology that should appear on website?)
Urgency
High, medium, or low.
Impact
High
, medium, low, or none.
Risk
High medium, low.
Effects
The services that could be affected by the change. Give a clear
description of the effect, and potential effect, on the customers.
Change Management Procedures
-
19
Communications
The person responsible for communications.
Commun
ications and Web Site Updates
Who is responsible for communicating this change to key
stakeholders? This may be the same person as the person
requesting the change, or the service manager, or
other
person.
Please note that the
UBC IT
main website is t
he default.
Web Site Content
Suggested wording for any potential service bulletin, including a
concise headline. All websites should be consistent in message
content and layout where possible.
Components
From the
UBC IT
perspective, what platform components will
have been modified.
QA approval
Name of the person(s) responsible for assuring the quality of the
change. This could be the person who tested the change, or the
installer.
Implementers
Who will make the change.
(Will change require training or
documentation updates?)
Backout personnel
List of people to contact if the change must be backed out.
QA Checklist
A series of tic boxes that provide an overall view of the level of
quality assurance.
Change Management Procedures
-
20
-
Customer Groups Co
mmunication
/Service Bulletins
Customer groups are contacted in sufficient time to amend or reschedule the proposed change.
The benefit of good communication with the customer is that it not only allows a better
understanding by the customer of the effor
t that is required to sustain a service but shows that
any downtime has been carefully considered and reviewed to minimize impact on the customer’s
business cycle.
The normal means of contact consists of email lists and web postings, although personal c
ontact
is also used as appropriate.
Documentation on who to contact and how, together with
sample scripts for se
rvice outages
(located in section 4:
Guidelines for describing the problem
OR
can be found in the document
Service Outage Notificatio
ns located here:.
Saturn
\
users
\
UBC IT
|public|Systeminfo|Service Outage Notification Procedures.doc
(
The contact is done
by
the person designated in
Communication and Web Site Updates
on the RFC.)
If customer objects to the change time after the notification goes out, the customer is referred to the
change sponsor for resolution.
Change Management Procedures
-
21
Forward Schedule of Changes (FSC)
The NOC operators will add new approved changes to the forward calendar of changes (FSC).
The forward FSC is available on Microsoft Outlook.
–
See
example below.
Change Management Procedures
-
22
-
Emergency Change Process
An Emergency Change is a change whose Urgency is high, and
where a deliberate decision is made to
shortcut the normal change process.
Examples:
Service outages (major)
HW/Application failure
Security
threat
The steps for getting an Emergency Change approved are:
Requestor:
Obtain approval of the customer and
Service Manager
Alert the Change Manager to the forthcoming change
Issue the RFC with Urgency = High
The following conditions must be met for an Emergency Change to be approved:
An Emergency Change requires 3 approvers from
the UBC IT Management (Directors and
Managers).
One of the approvers should be a Director, preferably the assigned Code 3 Director.
The Change Sponsor cannot be an approver.
A Director can veto the approvers.
The normal change control process notifies CAB
of the RFC through email. CAB members can
communicate through reply email or telephone Operations to participate in the approval process.
In the event that Operations needs to escalate and telephone Management to obtain approvers, then they
will start w
ith the assigned Code 3 team.
Ideally the approvers are ones who are familar with the subject area as they can best represent the
impacts, urgency, and risks of the Change.
With completion of the above, the normal change control process is adapted as app
ropriate.
*Note that changes that are not genuinely urgent
will not be approved
through the emergency process,
and will have to wait for the next meeting of the CAB.
Closing the RFC
Following the change, its implementation is reviewed and closed. The outcome is recorded as a
Magic
action
based on one of the following:
Successful
Change accomplished and system back up on time.
Cancelled/Denied
Change called off before the change st
art time.
Backed out
Change was removed from service during the maintenance window.
Change Management Procedures
-
23
Failed
The change interrupted normal service.
All
UBC IT
staff have a Magic ID, the action can be entered directly online. If for some reason
you do not have a Magic
ID, a message can be sent to the NOC staff at
ops5@interchange.ubc.ca
, quoting the RFC number and the outcome, and the staff member will
update the record.
Change Management Procedures
-
24
-
Change Assessment (Post
-
Implementation Review)
For all completed changes, the following steps are taken:
The NOC staff produce KPIs which summarize the changes performed in each period by
outcome
Changes from the previous week are examined at the weekly CAB meeting, for example:
change objective
achieved,
feedback (positive/negative) from users and customers,
were there unexpected side effects from implementing change,
effective resource planning,
implementation was executed as planned,
change was on time,
backout was successful if/when appl
ied)
“lessons learned” incorporated into “pre
-
RFC” checklist
report and followup with problems (e.g.: vendor, customer related, CAB)
information is absorbed into improvement process.
The statistics are reviewed by the change committee.
For any changes
which was backed out or failed:
a problem report is created by the operators and sent to the change sponsor, or the
service manager if different from the change sponsor.
For any change which
failed,
a critical incident review is conducted at the next Oper
ations Committee meeting
following the change, and the results of the review is published along with the
Operations Committee minutes.
Change Management Procedures
-
25
Glossary
Analysis of Failed Change
An analysis of a failed change is held after a change fails and is
successfully backed
-
out. The analysis is managed by the Change Owner
Manager or designate and results reported to the Director, IT Operations,
and the Operations Committee. It confirms th
e sequence of events,
establishes the cause(s) of the failure(s) and makes recommendations to
improve future performance.
CAB
The Change Advisory Board.
The CAB consists of all members of the Operations Committee, all
service managers, and others as req
uired to represent the various
customer groups.
Change Advisory Board Changes are the responsibility of the Change
Advisory Board
.
Change Manager
The Change Committee convener. The Change Manager performs the
following tasks:
Convenes the CAB
Circu
lates new changes to CAB
Issues approvals for changes
There is always a duty change Manager. If you do not know who it is,
you can find out by calling the Operations Centre.
Change Owner
The service manager or group manager who accepts responsibility for
the
change and its outcome. The Sponsor performs the following tasks:
Represents the change to CAB during its discussion
Takes responsibility for recovery if the change fails.
Deals with customer objections to the change timing.
Note that the Change
Owner must not be away at the time of the change.
If absence is unavoidable, please identify designate prior to meeting.
Critical Incident Review
(aka)Post Mortem
A Critical Incident Review (aka “post mortem”) is held after
both
a
change and the back
-
out plan fail. The Critical Incident Review is
managed by the Director IT Operations or designate and is reported to
the Operations Committee and
UBC IT
’ Senior Management. It
confirms the sequence of events, establishes the cause(
s) of the failure(s)
and makes recommendations to improve future performance.
A Critical Incident Review may also be called on other non
-
change
failures in the production environment at the discretion of the Director IT
Operations.
Change Management Procedures
-
26
-
Emergency Change
A cha
nge whose Urgency is High, which cannot wait for the next CAB
meeting. An emergency change is a case where a deliberate decision is
made to shortcut the change process.
Enterprise Middleware
Examples of middleware at
UBC IT
include, webservers (Apache,
T
omcat, ColdFusion), LDAP services,
Enterprise Applications
Examples of applications which are extended to the UBC community
include: UBC Mail (Exchange, SunOne), CWL, MyUBC, UBC
Directory.
FSC
Forward Schedule of Changes
This shows when approved ch
anges are scheduled for installation. The
FSC is available using Microsoft Outlook. (open ChangeLog calendar)
Normal Changes
Normal changes are changes to the live environment that can be
scheduled in advance and installed in a predictable manner.
Pr
oblems
Problems are unscheduled failures of the live environment. Problems are
handled through a process of problem management and escalation.
Changes to the live environment made as part of a problem fix are
recorded in the trouble ticket system, not the
change management log.
Problem management is not covered by this document.
Standard Changes
A
standard change
is a well understood, recurring, low risk change,
accepted response to a specific requirement of set of circumst
ances, the
approval for which has been delegated to the group making the change.
These can be entered as change log where the outcome is expected to be
successful with no impact to the user community.
Change Management Procedures
-
27
Appendix 1
CAB
Contact List
Emergency CAB Members
Listed alphabetically
Aksentsev, Felix
(IBA)
Belsito, Mark
(Apps PM)
Burns, Jennifer
(Passive)
Cooper, Lynda (Parental Leave)
Craven, William (Passive)
Cumming, Lois
(Systems)
Fong, Kent
(Access)
Frazer, Dave (Passive)
Haeusser, Jens
–
(Passive)
Hay, Marilyn
–
(NMC)
Huang, Amy (Passive)
IT
-
Systems Operations
(Passive)
Johnson, Patrick
Kita, Stan
Lay, Sean
Lee, Jeanne
Lim, Michael
Loewen, Doug
Macdonald, Bob
McKelvie, Evelyn
Miladinovic, Jovan
Ng, Susan
Operations (ITServices)
Quinville, Doug
Rosco, Steve
Sayer, Margaret
Shaw, Wes (Passive)
Smith, Brock
Thompson, Don (Passive)
Thorson, Michael
Twining, Neil
IT Resource: Klinck Rm.443
Bourdon, Eric
Razi, Sam
Change Management Procedures
-
28
-
Appendix 2 Pre
-
RFC Initiation Process
(Draft: Please
comment
(improvements or additions to list
)
Note 1: This is not meant to be a comprehensive list
, but a starting point
in determining what needs to be
considered
, your input is greatly appreciated.
1)
Reason for change:
-
Response to a business need
-
new service rollout
-
end of service/systems lifecycle
-
security, audit, legislation
-
customer driven
-
current service
-
identified problem/hardware failure
-
technical
–
software upgrades
2)
Perform risk benefits/risk an
alysis for each change prior to submission
What is risk of failure, what might fail, what would be estimated time to restore and recove
r
.
a) Customer Impact on Business
-
What services would be affected by unplanned outage
-
What is effect of not implementing change?
-
Is timing right for cycle of business?
-
What are the customers critical dates (see osmium calendar)
-
What is impact to an identified SLA or other agreement if there is one
in existence?
Customer Notifications
-
Notification of outage / Does it meet customer notification timelines?
Are alternate avenues (websites, systems status line) provided for
customer/
Are customers notified of resolution.
b) Environmental Impact
-
What is impact on other current processes/projects
–
What is the
priority attached to this?
-
What are other services that run on same infrastructure
-
What other changes are occurring that day (se
e outlook calendar
calendar)
-
Evaluate the impact on the following:
o
System
Hardware
OS Release
Memory
Disk space
CPU type
Remote access
Change Management Procedures
-
29
Password issues
o
Network
Bandwidth, routing, SNMP passwords, interference,
firewall, name resolution, security
o
Application
Detailed list of applications and dependencies,
release and patch level, critical applications
o
Processes
Description of the existing business processes and
dependencies, critical processes
o
Organization
Information on all units, alarm plan, li
st o
f
phone numbers,
logistic issues, etc. health
and well
-
being of staff who may be working
extra long
hours. There
may
also
be
physical
security issues such as after hours access.
o
Security Office
Are existing security measures in place met?
Is the
security office familiar with change
c) Resources requirements
-
Identify internal and external resources/ skillsets required for successful
change
-
Availability of an
experienced architect resident to ensure the
infrastructure and changes are plann
ed
and vetted before production
-
Verify that all equipment, software, hardware, and updates are available
-
Verify that backup tapes are available in the event of a back
-
out or
restore.
-
Research the requirements/ other supporting resources to achieve a
succe
ssful change (required patches and stability of upgrade, licenses,
security issues identified).
-
Identify who
needs to be aware and/or on standby when changes are
planned.
A rep from each group in
department
change m
anagement team to be ‘on duty’
Change Management Procedures
-
30
-
-
Syste
ms and services should be
operationalized to the extent that the
NOC can turn services on and off
-
Must communicate full
y to
NOC
who provide updates via status
messages/website updates:
Problem process loop, open/close, ongoing
d) Testing
This should ad
dress issues of performance, security, maintainability, supportability, reliability,
and functionality:
-
Develop a detailed plan of action to reduce the risk to an acceptable
level. (Comprehensive testing plan/signoff on tests carried out)
(implementation
details)
This should explain the steps that must be
taken to restore access in the event that the change has a negative
impact. Provide online link or file locati
on
.
-
Develop a plan of action to lessen the affects on the customer
if the
change should cause an outag
e. What is workaround? Is there
automatic failover in place?
-
Always develop a pre
-
change check
-
list (template should be
developed)
When did we last do a shut
-
down?
A stop/start should be tested without changes
Checkpoints need to be identified as problems arise
–
stop or go?
E.g. no shutdown script
–
stop?
Go live checklist
Walkthrough all dependencies first
Ensure everything is tested with the current version of the
operation system
Test cards prior to installi
ng into production (done in this case).
Test the drivers supplied under the correct release of the O/S
–
test
every configuration parameter in advance
Additional or spare hardware requirements?
Create an outage ‘timetable’ for planned outage window
-
e.g.
Sunday 5
-
7am
a.
Flag go/no go steps
b.
Identify when to back out if necessary
c.
Create for every change
-
A duplicate environment is required (parallel to try out)
–
need for all
applications
-
many development environments are not adequate
Change Management Procedures
-
31
-
Com
municate to sen
ior management the importance of appropriate
equipment
need functionality
–
not necessarily the exact same boxes (save
$$$)
-
IF vendor resources are required, check availability
a.
Is the support line adequately ‘manned’ during off
-
hours
(weekends?)
b.
Pre
-
arrange contacts to
‘stick handle’ the situation with
internal re
sources
24x7?
c.
Notify vendor (s)
of any impending changes/outages to
ensure we are on their radar if any issues arise
e)
Escalation process:
-
Clarify
Technical and Mgmt stakeholders
:
-
C
ontact lists
available
: cell
-
phone / wireless
-
phone / pager
, all
can be
automated
-
Refer to p
olicies as to who and how the go/no go decision is made
–
management?
f)
After a successful change
:
Review the following:
was
the planning correctly carried out
?
Report on completion/ Update RFC record or log as to status. Reply to
email that was sent to you by NOC informing you of RFC reference
number
has the impact been correctly estimated?
has
the usage of resources been correctly estimated?
Was it necessary
to deploy the resources of the standby team during the rollout or
change?
has the right effect of the change been reached?
are the users satisfied with the result?
# incidents on the change
were / are there unexpected side effects?
have detect
ed anomalies been communicated
to the CAB for future
changes?
Are there necessary documentation / training (customer, staff) updates
required after change. Identify resources: who will do what
Change Management Procedures
-
32
-
Finally
:
Obtain approval to proceed from your immediate or appropriate responsible manager for
requesting the change.
Submit a complete, concise, and descriptive Change Request no later than Monday (up to 23:59
-
midnight.)
Change Management Procedures
-
33
Appendix
3
Critical Incident Re
views
(aka Post
-
Mortem”)
Guidelines
When to hold
A
“CIR” (Critial Incident Review)
is held after both a change and the
back
-
out plan fail.
Th
is is
managed by the Director IT Operations or designate
and is
reported to the Operations Committee and
UBC IT
’
Senior Management
.
It confirms the sequence of events, establishes the cause(s) of the
failure(s) and makes recommendations to improve future performance.
A “CIR” (Critial Incident Review
)
may also be called on other non
-
change failures in the production environment at the discretion of
the Director IT Operations.
Example:
“CIR” (Criti
c
al Incident Review)
Description of Incident
Date
Attendees:
Absent:
Preparation:
Always bring docu
mentation/ticket info to
CIR meeting
Agenda:
-
Ground rules for critical incident review.
-
What took place? Chronology of events
-
Why did it take place?
-
What to do prevent this in the future (processes, environment, etc)
1. Action Items
Action required
Area of responsibility
Date due
Status
Review
-
Other
Lessons Learned
Things that worked well
Further recommendations.
PROBLEM RESOLUTION PROCESS
Change Management Procedures
-
34
-
Appendix
4
Examples of changes
Applications
new application rollout
new systems
releases/conversions/functional enhancements/fixes
maintenance
Databases
changing the name of a db table, view, or column
modifying a stored procedure, trigger, or user
-
defined function
changing or adding relationships using referential integrity features
changing or adding db partitioning.
moving a table from one db, dbspace, or table space to another.
changing the uniqueness specification of an index.
clustering the table data by a
different index.
changing the order of an index (ascending or descending).
Netwo
rks
installs, upgrades, disconnects, router configs.
Telecommunications
facilities
-
communication rooms
Infrastructure
Hardware (moves, add, changes, hw relocation, emerge
ncy
replacements, OS, installations
,
)
SW releases, enhancements
Documentation
Periodic maintenance,
User requests,
Hardware and/or software upgrades,
Acquisition of new hardware and/or software,
Changes or modifications to the infrastructure,
Environmental changes,
Operations schedule changes,
Changes in hours of availability, and
Unforeseen events.
Facilities
Major electrical upgrades, installs
Air Conditioner upgrades
Fire suppression systems
Change Management Procedures
-
35
UPS systems
Security panels
Desktops
HW
configurations
OS
Applications, releases, patches
Change Management Procedures
-
36
-
Appendix 5
Examples:
Communications
Strategy
Purpose
This document serves as a GUIDE FOR COMMUNICATIONS
REQUIRED for the Change Management process.
Include all stakeholders in the information flow to manage expectations
and invite participation prior to, during and after rolling out new features
or service.
Issues:
o
What will be communicated
o
What media mixes are most effective
o
Frequency
o
Urgent m
essages
o
Audiences
Change Management Procedures
-
37
Example guidelines for describing the problem
Concise Description
Describe concisely and specifically what the planned outage affects
in terms of what the
customer uses the service for.
Indicate what this outage affects or what error messages a user might see if they try to access the
service while it is out.
Examples
“This service or system will be unavailable…. Faculty and staff will be unable to log into..…”
“The main UBC web server
will be unavailable…. You will be unable to access any web pages
beginning with, …..may return ‘no such domain name’ messages.”
“E
-
mail services for faculty and staff will be unavailable ….”
Specify the duration of the outage.
Example
“…unavailable on Sa
turday, August 7, between 5AM and 7AM.”
Indicate if there is an alternative available to users while a service is unavailable.
Examples
“The __________web server will be unavailable… In the meantime you can access your email
by using…..”
“The ________
_server at www._______________will be unavailable … Please use the
following ……”
Identify if there is a web page that has useful information for the users.
Example
“For updates and a complete list of services affected by the outage, please visit
http://www.itservices.ubc.ca/support/
.”
Include suggestions where the user can go for additional information
Examples
“If you have questions please contact the Help Desk at 822
-
2008
“Please call
822
-
4115 for recorded updates.”
Please refer to the "Systems Alerts" section found at
http://www.itservices.ubc.ca/support/
for further updates.
Please contact the Network Operations Centre at 822
-
5438
(option 4) when network problems
are experienced.
Web page updates, examples
Example 1: Oracle3 outage notices for websites.
Change Management Procedures
-
38
-
“System Alert” section at
http://www.itservices.ubc.ca/support/bul
letins
Oracle3 Service Outage
-
A service outage is scheduled on August 27, 200x from 6:00am to
7:30am for maintenance on the oracle3 server. It will affect the following services and
applications.
Asbestos Tracking System
Campus
-
Wide Login (CWL)
Facul
ty and Staff Pension
Magic TSD Call Tracking System
Paradigm Call Tracking System
myPress Copyright Management
myUBC
ORSIL
Tracc
-
II
UBC Faculty and Administrative
Directory
UTAW (Utility Tool for
Administrators of WebCT)
We apologize for any inconvenience.
Please contact the Network Operations Centre at 822
-
5438 (option 4} if problems are
experienced.
Users of Windows 2000 and XP
-
If you encounter access problems following the Oracle3
Service Outage please try the follo
wing:
From the Start menu, select "Run", then type in 'cmd' (without the quotes) and press the "Enter"
key.
A DOS command will appear. Enter the following information.
ipconfig /displaydns
Then press the "Enter" key to view the local DNS cache.
ipconfig
/flushdns
The press the "Enter" key to flush the local DNS cache.
or you can simply reboot your PS to pick up the new DNS for Oracle3. Please contact the
ITServices Help Desk at 822
-
2008 if you continue to experience difficulties.
CWL Site
Service Bulle
tin
A service outage is scheduled on August 27, 200x from 6:00am to 7:30am for maintenance on
the oracle3 server. As a result, myUBC, CWL, WebCT administration tools and other online
services may not be available during this time.
F
or a full list of services affected, visit
www.itservices.ubc.ca/support/bulletins
NETINFO/ INTERCHANGE
Service Bulletin
A service outage is scheduled on August 27, 200x from 6:00am to 7:30am for maintenance on
the oracle3 server. As a result, myUBC, CWL, WebCT administration tools and other online
services may not be available during this time.
For a full list of services
affected, visit
www.itservices.ubc.ca/support/bulletins
Change Management Procedures
-
39
my.ubc.ca
Service Bulletin
A service outage is scheduled on August 27, 200x from 6:00am to 7:30am
for maintenance on
the oracle3 server. As a result, myUBC, CWL, WebCT administration tools and other online
services may not be available during this time.
For a full list of services affected, visit
www.itservices.ubc.ca/support/bulletins
Change Management Procedures
-
40
-
Example 2: Enrollment Services Database outage notices for websites.
Student Centre Service web site
Service Bulletin
The Enrolment Services Database will be unavailable Saturday, October 19, 200x (a
nd Sunday
Oct 20 if required). The outage is expected to start at 3:00AM Saturday morning and last all day.
The purpose of this outage is to perform an Oracle Database upgrade.
We apologize for any
inconvenience. For a list of all services affected by the
outage, please visit www.itservices.ubc.ca/support.
“System Alert” section at
http://www.itservices.ubc.ca/support/bulletins
Service Bulletin
The Enrolment Services Database will be unavail
able from
3:00 AM, Saturday, October 19,
200x and Sunday, October 20, if required
. The purpose of this outage is to perform an Oracle
Database upgrade.
All applications/services that require access to the Enrolment Services Database will be affected.
The f
ollowing is a list of known services that will be impacted:
Enrolment Services
-
Admissions
-
Awards
-
Course Scheduling/Catalog (Ad Astra)
-
Elections
-
Degree Navigator (DAG)
-
Faculty Service Centre (FSC)
-
Student Authentication Service
-
Student
Information Service Centre (SISC)
-
Student Information System (Old Green Screens/Unikix)
-
Student Service Centre (SSC)
Others
-
TracII/Netinfo access requiring student number validation
-
myUBC authentication using student number
-
CWL registration using
student number
-
Housing authentication using student number
-
WebCT administration
We apologize for any inconvenience.
my.ubc.ca
Service Bulletin
Due to upgrades to the Enrolment Services server, students will not
be able to log in using student numbers on
Saturday, October 19 and Sunday, October 20. We apologize for any inconvenience. For a list of all services
affected by the outage, please visit www.itservices.ubc.ca/support/bulletins
CWL Site
Change Management Procedures
-
41
Service Bulletin
Student CWL registration will not be available on
Saturday, October 19 and Sunday, October
20
, if required, due to upgrades to the Enrolment Services database server. For more information,
visit
www.itservices.ubc.ca/support/bulletins
. We apologize for any inconvenience.
NETINFO
Service Bulletin
Netinfo registration and password resets will not be available on
Saturday, October 19 an
d Sunday,
October 20
, if required, due to upgrades to the Enrolment Services database server. For more
information, visit
www.itservices.ubc.ca/support/bulletins
. We apologize for any inconveni
ence.
www.webct.ubc.ca
Service Bulletin
Enrolment Services interruption: Saturday, October 19/02
-
all day
Due to the downtime of the Enrolment Services database on Saturday, October 19/02, the
following WebCT ser
vices will be affected:
Users will not be able to login to WebCT using student number and PIN.
* Please use your interchange/netinfo account to avoid interruption
For more information, visit
www.itservices.ubc.ca/support/bulletins
. We apologize for the
inconvenience.
Sincerely,
ITServices WebCT Admin Team
Change Management Procedures
-
42
-
Example 3: Phase 2 of the admin cluster firewall installation
.
“System Alert” section at
http://www.itservices.ubc.ca/support/bulletins
Service Bulletin
Notification of ITServices Outage
The following is advance notification of a major outage within ITServices network and the
ser
vices it provides. This outage has been scheduled in the early morning to minimize the impact
on students, faculty and staff services:
From
06:00am November 13, 200x
To
:
08:00am November 13, 200x
Reason:
Scheduled network configuration maintenance
Impact:
The following services will be unavailable
during the above period
Enrolment Services
Admissions
Awards
Course Scheduling/Catalog (Ad Astra)
Degree Navigator (DAG)
Elections
Faculty Service Centre (FSC)
Student Authentication Service
Student In
formation Service Centre (SISC)
Student Information System (Old Green Screens/Unikix)
Student Service Centre (SSC)
Others
TracII/Netinfo access requiring student number validation
myUBC authentication using student number
CWL authentication using student
number
Housing authentication using student number
WebCT Administration
We apologize for this disruption in service and thank you for your patience.
Please refer to the "Systems Alerts" section found at
http://www.itservices.ubc.ca/support/bulletins
for further updates.
Change Management Procedures
-
43
Please contact the Network Operations Centre at 822
-
5438 (option 4) when network
problems are experienced.
Enter the password to open this PDF file:
File name:
-
File size:
-
Title:
-
Author:
-
Subject:
-
Keywords:
-
Creation Date:
-
Modification Date:
-
Creator:
-
PDF Producer:
-
PDF Version:
-
Page Count:
-
Preparing document for printing…
0%
Comments 0
Log in to post a comment