Data Center Slides April 2011

lovingbangInternet και Εφαρμογές Web

3 Νοε 2013 (πριν από 3 χρόνια και 9 μήνες)

88 εμφανίσεις


Drivers for New Data Center


Out of power and cooling in
Komas

data
center


Increased demand for computing and data
storage capacity


Inadequate existing facilities on campus to
accommodate expansion


Buying co
-
location space too expensive


Diverse data center needs (enterprise and
high performance computing)


MCI Building Purchase


Purchased MCI building (875
S.
West
Temple)


Prime facility for a data center build
-
out


74,000 square feet to allow campus data
center consolidation and future phased build
-
out


Seismic retrofit


Space to support paper medical record
storage


HP/EYP Analysis


Scope Change


7 main data centers all have capacity issues


Needed strategy for consolidation and growth
capacity


Needed strategy to reduce data center operating
costs


Cloud computing and cloud storage
strategy


Colocation


UHEA


Weber State


Others


Budget


Design and construction
-

$21 M


Network and racks
-

$2M


1/3 of the cost as similar data centers with
less capability

Southwest Corner View

Arial View

Master Plan

EQUIPMENT

YARD

Project Team


Earl Lewis


Project Manager


Jim Livingston
-

Sponsor


Brent
Elieson



UIT


Steve
Corbato



UIT/
Cyberinfrstructure


Joe Breen


CHPC


Bryan Petersen


UEN


Glen Cameron


UIT


Mike
Ekstrom



UIT


Tim Richardson


UIT/ACS


Caprice Post


UIT/Architecture


Dave
Huth



UIT/Architecture


Andrew Reich


UIT Architecture


Lisa Kuhn


UIT/Finance


Steve Scott


UIT/
Secuity


William Holt


Hospital Financial
Management Analysis


Bill
Billingsly



CD&C PM


Jim Russell


State DFCM


Department of Sustainability


VCBO


SmithGroup


AlphaTech


Oakland Construction


Heery



Schedule


Design complete


Construction documents complete


Bids due first week in March


Construction done by mid January


Commissioning complete in March


Move
-
in to start in April, to be completed by
end of August

West Pavilion


DR and hot site for tier 1 applications


Part of move strategy to new data center


Slowly populate with network gear and other
infrastructure over the coming year


Populate with tier 1 applications as we move
to new DC

Move Planning


RFP for project management firm


completed


Planning to start first of April ‘11


Configuration Management Database
foundation to move planning

West Temple Data Center

You’re Cooler Than You Knew Tour

Table of Contents


Reason for project / Facility


Project scope


Building Structure


Administration Workflow


Fiber Entrances


Metro Optical Ring


MPOE/Meet Me rooms


White Space Cable Plant, MDA & Disk


Mechanical Design


Electrical Design


PUE & Savings Implications


Tiers


POD /Rack Design, ISO base


Procedures


Physical Security


Power Densities


Resiliency Strategy: Tier3, WPDC & Richfield


Co
-
location Capabilities

Reason for the Data Center…

Supply & Demand, no power on campus

Time

KWatts

Project Scope


10,000 KW facility in 3 phases


$4.5M bought 74,000 square feet


$23M will build out 4,000KW(2,400 pluggable)


High efficiency (low PUE)


Innovation


Low operating cost


Better workflows and more usable center


LEED Gold


875 S. West Temple


74,000 square feet, 1 city block!


Hospital seismic importance factor 1.5 x 2/3 maximum credible
earthquake force = “Essential Facility”


Compared to typical retrofitted buildings, this structure will perform better


Lateral design criteria meets IBC definition of “Essential Facility”


18” walls: 8” steel reinforced
shotcrete

and 10” Masonry


New roof, foundations and steel frames




Henry, J.R. (2006). “Section 1604.5: Importance Factors.” Structural Engineering and Design,
Stagnito

Media, Deerfield, IL,
http://www.gostructural.com/magazine
-
article
-
gostructural_com
-
september
-
2006
-
section_1604_5__importance_factors
-
4714.html

, last
accessed 5
-
14
-
10. 2006 IBC, FEMA 427


Workflow

Building Entrances

Metro Optical Ring Project


Connection Sites: Park, EBC, WTDC MPOE1 & MPOE2,
Level3 POP, BYU, USU, NOAA, Salt Palace, Kearns Bldg and others


Funding Sources: BTOP/NTIA, trade agreements, UU


Optronics
: DWDM vendor TBD.


Dark Fiber IRU’s, Conduit IRU’s & UU assets


Speed (80 x 10Gig)


Redundancy: circuits, pathways, chassis, CPU, Provider


Self Provisioning of circuits (
ethernet

& fiber channel
or even PTP, FR, etc.)


Cost savings 1/20
th

x (# of circuits ~20) = 1/400
th

DWDM: D
ense
W
avelength
D
ivision
M
ultiplexing works by combining and transmitting multiple signals simultaneously at different
wavelengths on the same fiber.

IRU
:
I
ndefeasible
R
ight of
U
se is the effective long
-
term lease (temporary ownership) of a portion of the capacity of an international
cable.

Metro Optical Ring


Data Center Path

MPOE / Meet Me Rooms

Purpose:


Building entrance aggregation


Carrier rooms


Inter
-
Carrier IP peering


Metro Optical Ring (self provisioned circuits)


Data Center collocation inter
-
occupant connectivity


MDA(MDF) distribution

Features:


>66 feet apart (tier 3 compliance)


Fire zone separation


2
nd

story suspended mezzanine


Sized for all 3 data center phases


Loading bay for forklift deliveries

White Space Cable Plant


7 server cabinets cabled back to 1 network rack


17 network racks trunked back to 2 MDA’s


SAN and
eNet

Core located in MDA


32 Disk cabinets located next to MDA


Ladder rack and fiber guide


100G capable fiber plant


6A copper plant


Equipment specific fan out fiber arrays & MTP’s


Fiber patch cord slack accommodated!


All cabinets are 32” wide!


60A PDU’s will fit inside cabinet, no air clogging



Mechanicals


2008 ASHRE TC 9.9 recommendation: 64.4
-

80.6F, 40
-
60% RH @ C.1&2



Ambient Air side economizer 85% of year including hot air remixing


Indirect evaporative cooling towers 14% of year (
w.s
. economizer)


DX chiller cooling ~1% of year and backup for water source


Atomizer humidification provides direct evaporative cooling benefit


Chilled water loop/heat exchangers for water cooled racks & equipment


FAN wall, filters, coils, service corridor, security corridor, 14’ plenum,
equipment driven air flows, Hot/Cold pressure differential


VFD near
-
unity PF
, follows the affinity laws, which means that flow is proportional to speed,
pressure is proportional to the square of speed, and horsepower is proportional to the cube of speed.
That means if an application only needs 80 percent flow, the fan or pump will run at 80 percent of rated
speed and only requires 50 percent of rated power. In other words, reducing speed by 20 percent
requires only 50 percent of the power.



Racks on concrete


Hot isle containment



Electrical Design


Meter power 4,000KW


Available white space power 2,400KW


Common Data Center Power Topologies:

1.
Block Redundant: Catcher System

2.
Distributed Redundant

3.
2N Redundant

4.
Parallel Redundant

5.
Isolated Redundant


N+1 at the block level for:


Transformer


Generator


ATS


UPS


STS


PDU


2N Power distribution system (OHPB, CPDU)


All CPDU’s will fit in cabinets properly!


220V +


All devices and 1
-
lines
PQM’d


PUE and Savings


PUE = power usage effectiveness. (meter load / equipment load) category 3 metering


PUE Benchmarks


Industry Average 2


Komas DC 1.65


NREL Golden Colorado 1.06


Yahoo Lake Michigan 1.08


Google DC’s 1.2 **


UU West Temple Data Center ~1.2


T3 Electrical Bill $ savings anticipated


$250K to $700K annual RMP savings depending on load, anticipated rate hikes & realized PUE


RMP efficiency incentive sever hundred thousand dollars, one time opportunity


Owner maintenance and change savings


CRAC’s (dozens of maintenance contracts eliminated)


Branch circuit builds (>400 avoided)


Recent Data Center Benchmarks


Codename Sequoia: Tier 2 @ 3x our cost (power normalized)
4/1/
2011 substantial completion ***


Yahoo Lake Michigan: Tier2 @ 1.5X our costs (power normalized) *

PUE Savings Model

Load*wast rate

wasted power

hours/yr

FY10 Rate

FY10 innefeciencies

FY16 Rate

FY16 innefeciencies

1,000*.2=

200

8760

0.06

$105,120

0.12

$210,240

1,000*.7=

700

8760

0.06

$367,920

0.12

$735,840

$262,800

$525,600

1,400*.2=

280

8760

0.06

$147,168

0.12

$294,336

1,400*.7=

980

8760

0.06

$515,088

0.12

$1,030,176

$367,920

$735,840

Parameters

PUE

PUE

KW

KW

$/KWH

$/KWH

1.2

1.7

1,000

1,400

0.06

0.12

Tier Level Definitions

Tier Level

Requirements

1


卩湧l攠湯n
-
r敤e湤ant⁤ st物r畴i潮 灡p栠獥s癩湧 t桥⁉T 敱ei灭敮p


乯N
-
r敤e湤ant ca灡捩c礠c潭o潮敮ts


Basic site infrastructure
guaranteeing 99.671% availability

2


䙵lfill猠all Ti敲eㄠr敱eir敭敮ts


Redundant site infrastructure
capacity components guaranteeing 99.741% availability

3


䙵lfill猠all Ti敲eㄠ& Ti敲e㈠r敱eir敭敮ts


M畬ti灬攠i湤数敮摥湴⁤ist物r畴i潮 灡p桳 獥s癩湧 t桥⁉T⁥煵i灭敮p


All IT equipment must be dual
-
powered and fully compatible with the topology of a
site's architecture


Concurrently maintainable
site infrastructure guaranteeing 99.982% availability

4


䙵lfill猠all Ti敲eㄬ Ti敲e㈠慮搠di敲″ r敱eir敭敮ts


䅬l c潯oi湧 敱ei灭敮琠p猠s湤数敮摥湴d礠摵al
-
灯w敲敤Ⱐi湣n畤i湧 捨cll敲猠a湤⁈敡瑩湧Ⱐ
Ventilating and Air Conditioning (HVAC) systems


Fault tolerant
site infrastructure with electrical power storage and distribution
facilities guaranteeing 99.995% availability

*Uptime Institute

UIT POD/Rack Design


Disk on seismic ISO Base platform


Subzero hot isle containment


Uniform Cabinets: ~48U, 32”x42”


7 to 1 center of row network racks


FTU’s in middle in network rack


Console Server & KVM whole DC solution


24 port 6A patch panels


0U CPDU’s on just one side


Cabling on other side


Cable fingers on rear


Cabinet top cable seals


Mount on concrete floor


Unistrut

bracing




Physical Security


Barricades, cameras, card readers, mantraps


DHS standards and consulting


24 x 7 presence


72 hours of generator fuel


Water independence


Physical access standards

Data Center Procedures



Effective Data Centers have ~200 documented procedures


Maintenance


Repair


3
rd

party access


Change Control


Testing


Equipment installations & standards


Receiving


Build Room use


System Documentation


Training


Standards


DCIM/BMS





Consistent outcomes require consistent behaviors

Infrastructure and operations (I&O) leaders must now go beyond performance management of IT equipment and begin to manage the

en
tire
data center infrastructure. Tools for data center infrastructure management
(DCIM)

will provide detailed monitoring and measurement of data
center performance, utilization and energy consumption, supporting more
-
efficient, cost
-
effective and greener environments.

David J.
Cappuccio
, Gartner

Power Densities

~240 Rack locations in 1,400KW T3 portion of DC



1,400KW / 6KW/rack= 233 racks (DL380 x 20)


1,400KW / 8KW/rack = 176 racks


1,400KW / 15KW/rack = 95 racks


1,400KW / 20KW/rack = 70 racks (HP C
-
class x 4)





UIT Data Center Resiliency Strategy

1.
Tier 3 & essential facility data center

2.
West Pavilion essential facility & survivable
data center

3.
Richfield remote DR data center

4.
PSI nuclear resilient data storage facility

Potential Co
-
Location Tenants


UIT


UEN


CHPC


LAW?


Development?


Other Dept’s


Weber State


CENIC.org


UHEA



DC Products


Cloud File Storage


Cloud Network Attached Storage


Cloud Server Hosting


Co
-
location


DC Centralization


Inter
-
Carrier IP Peering

Contact Anita Sjoblom for more info or visit orderit.utah.edu

Questions??

AHSRAE: http://tc99.ashraetcs.org/documents/ASHRAE_Extended_Environmental_Envelope_Final_Aug_1_2008.pdf