HealthPartners: Disaster Recovery Planning - Digital Concourse

separatesnottySoftware and s/w Development

Nov 25, 2013 (3 years and 9 months ago)

176 views

G2_6:
HealthPartners: Disaster
Recovery Planning


Something you
Work on and Hope You’ll Never Need

August 23, 2012



Jan Grossman, Senior Project Manager

Tom Machacek, IS & T Manager



HealthPartners: DR Planning

Disclaimer regarding this presentation










This presentation contains NO confidential
information and may be copied and distributed to
anyone who really finds this stuff interesting!

2

HealthPartners: DR Planning

Agenda



Introduction to HealthPartners


HealthPartners’ Disaster Recovery Plans Today


Enterprise Systems


Disaster Recovery Project


Disaster Recovery Planning


Challenges


Wrap
-
up/Q&A

3

HealthPartners: DR Planning

Introduction


About HealthPartners


Founded in 1957, the HealthPartners
(
www.healthpartners.com
) family of healthcare companies
serve more than one million medical and dental health plan
members nationwide. It is the largest consumer
-
governed,
nonprofit healthcare organization in the nation, providing
care, coverage, research and education to improve the
health of members, patients and the community.
HealthPartners family of care includes Regions Hospital
(
www.regionshospital.com

), a full service hospital certified
as a Level I Trauma Center serving Minnesota and western
Wisconsin for more than 130 years. It also includes over 50
medical and dental clinics that serve Minnesota and western
Wisconsin.



Our mission: Why we are here


Our mission is to improve the health of our members, our
patients and the community
.




4

HealthPartners: DR Planning

Introduction

Lawson Install


LSF 9.0.1/9.0.1 Apps


HR/Payroll Suite, E/MSS (Heavily modified for benefits
enrollment, pay statement, code of conduct etc.)


Financial Suite


Procurement Suite, SMRC, MSCM, RQC 10


LBI


Oracle 11; IBM AIX Unix 6; Websphere 7


Landmark/Contract Management


Lawson Smart Office


Dashboard Gear


Cognos
/TM1 Budgeting

5

HealthPartners: DR Planning

Introduction


2000:
Financial Suite


2003:
SCM Suite, RSS


2004:
LBI (Beta site)


2006:
HR/Payroll Suite


2007:
MSCM, ESS and benefits enrollment, Smart Recon


2008:
MyInfo
(external ESS access), MSS, LSF9


2009: 9.0 Apps upgrade, LBI upgrade, MSCM
upgrade, Dashboard Gear
-
FN, LBI
-
RS


2010: 9.01/9.01, retire
Essbase


2011: RQC, LSO, Landmark, Absence Mgmt,
Dashboard Gear
-
HR


6

Introduction


2012:


Absence Mgmt (live) Lawson Smart Office (limited live),
Landmark/Contract Management (live)


SCM “patch bundle”/RQC 10/MSCM upgrades (live)


Projects in process:


Lawson Security 9 (just beginning)


Mobile Supply Chain Management 10


Landmark 10 and Contract Management 9.1


2013:


Lawson Security 9


LSF9.0.1.9 or higher, Lawson Business Intelligence 10 and AIX
7.1


Lawson Process Automation 10


2014:


LSF10/Apps10/Lawson Security 10/Workspace


HealthPartners: DR Planning

7

HealthPartners: DR Planning

Introduction


Lawson Statistics


10,500 ESS users working toward 12,000 (all employees)


450 MSS users


252 Benefit plans for 7 entities


2100 GL Accts, 3300 GL AUs, 30 Corporations, 3.7 million GL
transactions/year (many are summarized)


32,000 AP “Trade” Vendors, 200,000 “Health Plan” vendors,
11,000 Employee Vendors, 250,000 Invoices/year


35,000 Item master file items,


93,000 POs; 550,000 lines; $150 million spend/year


800 RQC Users

8


DISASTER


Data Center Gone !!!

HealthPartners: DR Planning

9

HealthPartners


Disaster Recovery Policy


Purpose: Standardize procedures for system
recoverability that support the priorities of
HP’s business operations and services



Supports Risk Management policies:


risk mitigation


Ensures continuity of business operations and
recovery of resources following a disruption.



HealthPartners: DR Planning

10

Disaster Recovery Planning


Business Continuity Plans drive the Tier designation
= the what and when of recovery


What gets planned for


Over 800 applications supporting the HP family of
businesses


Over 1700 servers


AIX



184


Linux



165


Unix



142


Mainframes



8


SAN




17


Telecom




12


Windows



1292







HealthPartners: DR Planning

11

HealthPartners


Disaster Recovery Planning


Existing Plans
in place for Tier 0, Tier 1
systems that are supported at the West Disaster
Recovery Center in St. Paul


DR Center:


Dedicated disaster recovery center with Tier 0
and Tier 1 recovery systems in place


Plans assume an infrastructure of power,
PC’s, Network, Telephone and Internet
access



HealthPartners: DR Planning

12

HealthPartners


DR Priority Process


Infrastruct
ure
Systems



Disaster
Recovery
Center


Tier 0

(0 days)


Epic


HealthPlan


Lawson


ImageNow


LaborWorkx

Tier 1

(0 to 3
days)



Hot Standby
with
Restored
Data


Quick
Shipment

Tier 2

(4 to 7
days)


No Standby
Systems


Quick
Shipment
List

Tier 3

(8 to 14
days)



No
Standby
Systems


Low
Recovery
Status

Tier 4

(15 to 21
days)


As
identified


No
specific
plan

Tier 5

(22+
days)

HealthPartners: DR Planning

13

HealthPartners


Tier 0 and Tier 1

Tier 0
-

Infrastructure


Tier 1


Business


HealthPartners: DR Planning


Networks


Telecom


Unix Servers


Windows Servers


Databases


TSM Servers (Tape
Backups)


Epic


Patient Care


HealthPlan



Medical Plans


Lawson
--

Supplies


ImageNow



Patient Care
Information


Laborworkx

--

Staffing and
Scheduling of Medical Staff

14

HealthPartners


It’s a Project


Enterprise systems DR Planning Scope


over 17 plans


Lawson and its Ancillary Systems


ImageNow


LaborWorkx
: time and attendance, staffing
and scheduling


Filesite

Document Management


HealthPlan

applications


Tax and Audit systems

HealthPartners: DR Planning

15

Disaster Recovery Plan Development

1.
Pre
-
Planning

Requirements
Gathering

2.
Compiling
& Drafting



3.
Plan Finalization


Walkthroughs/Tests

Approvals, Submission

HealthPartners: DR Planning

16

Disaster Recovery Plan Development

1. Pre
-
Planning


Systems


Current Configurations


Disaster Recovery Configurations


Gap Assessments


To Be Assessments


DR Plan Requirements Gathering


Understand the Business Continuity Plan


Establish a project plan (for each system)


Project Team


Schedule


Kickoff Meeting



HealthPartners: DR Planning

17

Disaster Recovery Plan Development

2. Compiling and Drafting


DR Plan Template (HP Tool)


Development:


DRP Scope and Objective


Recovery Strategy


Team Composition


Recovery Resources


Communications Plan


Executable Recovery Procedure(s)




HealthPartners: DR Planning

18

Disaster Recovery Plan Development

2. Compiling and Drafting (continued)


Appendixes


Team Contacts


DR Team Schedule


Configuration Diagrams/components


Interfaces (diagram/list)


Recovery Sequences


Service Level Agreements


IS&T Recovery Scripts


Business Validation Checkout Scripts (if
available)


HealthPartners: DR Planning

19

Disaster Recovery Plan Development

3. Plan Finalization


Conduct Walkthroughs


Confirms the plan’s expectations with the multi department
team


Gain acceptance of the plan, process and expectations for
discovery recovery


Disaster Recovery System Test


Tier 1 systems, annually


Tier 2 systems, to be scheduled based on DR system
availability


Finalization of Plan (make those corrections based on
feedback from Walkthroughs and DR System Tests.


Management Signatures


Registration of the Plan


HealthPartners: DR Planning

20

HealthPartners’ Lawson System
--

Disaster
Recovery Sequence


Lawson

(1)

Landmark
(2)

Contract
Mgmt

(2)

MSCM

(2)

LSO

(3)

LBI

(3)

MHC

(3)

Document
Server

(3)

Dashboard
Gear

(4)

Cognos

(4)

HealthPartners: DR Planning

21

Disaster Recovery Planning Challenges


Affirmation of the system components, versions, relationships


CMDB Service Now updates


Consensus of the multi department resources


Were they on board with the Plan expectations


Disaster Recovery Center accommodations for additional hardware


Is there space and is the infrastructure in place


Business Continuity expectations versus reality of DR system
availability


If there is no DR hardware, delivery lead time needs to be
factored in


Budget responsibility and availability to purchase equipment


Who owns the budget, can it be acquired in the current budget
cycle

HealthPartners: DR Planning

22


DISASTER


Data Center Gone !!!

HealthPartners: DR Planning

23

DISASTER


Data Center
--

Gone


Are we ready and prepared?


The plan is written, we’ve done the walkthrough and it is
filed with the Disaster Recovery Center.


Future Projects


During project planning, review existing DR Plans, update
it with changes from the project


If a new system,
Write a DR Plan


Next year?


Review the plan, make changes with current information


Perform a Walkthrough and/or


Complete a DR System Test


Then: Let’s Hope we Don’t Need It!!!

HealthPartners: DR Planning

24

HealthPartners: DR Planning

Wrap
-
up/Q&A

Thanks for attending!



Jan Grossman, IS&T Senior Project Manager


Janice.j.grossman@healthpartners.com


952
-
967
-
7351


Tom Machacek, IS & T Manager


Thomas.J.Machacek@healthpartners.com


952.883
-
6382


25