IP Storage: Best Practices

Hosted by

IP Storage: Best Practices

Marc Staimer, President & CDS

Dragon Slayer Consulting

marcstaimer@earthlink.net

503
-
579
-
3763

Hosted by

Agenda


IP Storage Level Setting


File vs. Block Storage


FCIP, iFCP & iSCSI


Fiction & Facts about iSCSI


Storage Replication over WANs


Considerations for Designing IP Storage Networks


Questions

Hosted by

IP Storage Level Setting


There are 3 types:

•
Block Storage


iSCSI

•
File Storage


NAS

•
Storage over WAN for business continuity = Block


iSCSI


FCIP


iFCP

Hosted by

IP Storage Level Setting


iSCSI Storage

•
Block
-
based external storage on Ethernet


Vs. SCSI, USB, 1394, or FC


NAS or Network Attached Storage

•
File
-
based storage = NFS & CIFS

•
No different than any other file server

•
Requires block storage behind it

Hosted by

Hosted by

By show of hands, what is
acceptable packet loss for IP
Block Storage on a LAN/WAN?

1)
1%

2)
5%

3)
10%

4)
0%

Hosted by

Acceptable Packet Loss for

IP Block Storage on a LAN/WAN


The Answer is:


Hosted by

Block vs. File Storage


IP block data is unlike any other IP data

•
Overwhelms most current LAN/WAN environments

•
Incredible amounts of traffic

•
Tolerates
“ZERO”

p慣k整 loss

•
Very low latency


File storage = specialized file server

•
NFS & CIFS

•
Higher prioritization is required depending on app

•
Volume of data may overwhelm untuned LAN/WAN

Hosted by

FCIP, iFCP, iSCSI


FCIP

•
Fibre Channel tunneled in TCP/IP

•
IP transport between FC switches


iFCP

•
IP header put on Fibre Channel frames for routing

•
IP connection services for FC devices


iSCSI

•
SCSI
-
3 mapped to TCP

Hosted by

FCIP


Pt
-
to
-
Pt: Becomes “One” FC SAN

•
Disruptions pass SAN
-
to
-
SAN

•
Large FSPF database

•
PSS between SAN sites

•
Gateways between fabrics (blades or boxes)

Ethernet LAN/WAN Switches

Cisco MDS 9216

CNT UESR 3000

Brocade 3xxx

SAN 1

SAN 2

SAN 3

SAN 4

TCP/IP

WAN

Hosted by

iFCP


Pt
-
to
-
Multi
-
Pt

•
Device specific passing only the data that is required

•
Devices can appear in multiple individual SANs


The SANs themselves remain independent

Ethernet LAN/WAN Switches

SAN 1

SAN 2

SAN 3

SAN 4

TCP/IP

WAN

McDATA/Nishan
3300

Hosted by

Parable

Hosted by

Hosted by

iSCSI: Ethernet
-
Based SAN


The Hype

•
Block storage on Ethernet

•
Leverage current infrastructure investment &
knowledge base

•
Lowers cost

•
Eliminates headaches

•
Ubiquitous

•
Makes FC another Ethernet Road Kill

Hosted by

iSCSI Defined


IETF standard protocol


•
Establishes & manages
connections

•
Carries storage (SCSI) blocks


From initiators to storage
targets


Encapsulates SCSI blocks in TCP/IP

•
Then tunneled in Ethernet

•
iSCSI is to Ethernet as FCP is to
Fibre Channel


Network application

•
One infrastructure for LANs,
NAS, & SANs


Ethernet





Frame



TCP/IP




Packet


SCSI
-
3

TCP/IP

iSCSI

NAS: File
Storage

GbE Switch

Mission Critical

IA App Servers

Hosted by

iSCSI Applications


NAS/SAN combined storage units


Entry level SANs


Limited budget SANs

Hosted by

SAN Benefit Assertions of iSCSI


Reduced costs

•
Professional services, implementation, management
& IT Staff time


Reduced complexity


Reduced Management


Increased Interoperability


Elimination of Multiple Networks


Unlimited SAN Distance


Equal or Better Performance

Hosted by

Fiction & Facts about iSCSI


Fiction

•
Cost is lots < Fibre Channel
(FC)

•
Complexity is much < than
FC

•
Uses current infrastructure

•
Requires no storage
knowledge

•
Is as fast as FC

•
Will replace FC in Enterprise

•
Easier to manage than FC

•
Eliminates SAN distance
limits


Latency (delay) is not an
issue


Facts

•
Known technology

•
Costs are relatively < FC


Cycles or hardware

•
Doesn’t require special HW


But benefits from it

•
Latency (Delay) matters


Can’t be > 1 millisecond

•
Deterministic routing

•
Doesn’t require any
-
to
-
any

Hosted by

Rating the iSCSI Value Props


Reduces or Eliminates SAN Professional Services


Lowers SAN Hardware Costs


Simplifies SAN Management


Eliminates Interoperability Issues


Converges SAN/LAN/MAN/WAN Fabric Infrastructure


Extends SANs over unlimited distances


Equal or better performance than FC SANs

Hosted by

iSCSI Reality Check



There are some real cost benefits for:

•
NAS/SAN on the same fabric infrastructure

•
SANs that don’t need the performance of FC

•
Entry SANs that may not even need GigE and TOEs


Hype overshadows reality:

•
GigE NICs with iSCSI and TOEs cost ~ same as FC HBAs

•
FC ports & GigE ports on server motherboards


Makes port cost differences higher for GigE w/TOEs

•
Very low cost simplified FC switches


Have erased much of the infrastructure HW differences

Hosted by

Sample iSCSI Vendors


Switches

•
Cisco

•
Extreme

•
Foundry

•
Enterasys

•
Nortel

•
Lucent

•
3Com


Gateways

•
Cisco

•
McDATA


Silicon

•
Adaptec

•
Alacritech

•
Intel

•
Siliquent

•
QLogic


Storage
NICs

•
QLogic

•
Intel

•
Alacrite
ch

•
Adaptec

•
Emulex

Hosted by

Hosted by

By show of hands, is latency
(delay) important to iSCSI block
storage?

Hosted by

Is Latency Important to IP Block
Storage?


The Answer is:


•
Yes, for the most part

•
It also depends on application

Hosted by

Hosted by

By show of hands, who
believes that TOEs & iSOEs
are an iSCSI block storage
requirement?

Hosted by

Are TOEs & iSOEs an iSCSI block
storage requirement?


The Answer is:

Not necessarily

Hosted by

Spectrum of iSCSI Adapter Solutions

Host

Adapter

Adapter Driver

TCP/IP

iSCSI

SCSI Port to OS

Software iSCSI

“NIC + Driver”

Media Interface

Ethernet

Media Interface

Ethernet


Fast Path TCP/IP

Software iSCSI

With Partial

TCP Off
-
load

TCP/IP

iSCSI

SCSI Port to OS

Media Interface

Ethernet

TCP/IP

iSCSI

Firmware TCP

and iSCSI Off
-
load

SCSI Port to OS

= SW or FW

= Hardware

Media Interface

Ethernet

TCP/IP

iSCSI

Hardware TCP

and Firmware iSCSI

Off
-
load

SCSI Port to OS

Hosted by

iSCSI: No TOE


Definition

•
Std Ethernet NIC


TCP/IP & iSCSI


Host
-
based in drivers


Who

•
Microsoft & Cisco


Advantages

•
Lowest cost

•
NICs available today

•
Easy integration with OS


Disadvantages

•
Lowest performance

•
High CPU load

•
High interrupts


Once/packet


Many/ TCP segment

Adapter Driver

TCP/IP

iSCSI

SCSI Port to OS

Software iSCSI

“NIC + Driver”

Media Interface

Ethernet

= SW or FW

= Hardware

Host

Adapter

Hosted by

iSCSI: Little TOE


Definition

•
NIC w/limited TOE


Packets in order & no frags


Out of order etc. go to OS


Who

•
Alacritech


Advantages

•
Relatively low cost

•
Small layout (low profile card)

•
Good throughput w/pristine
Ethernet


Disadvantages

•
Out
-
of
-
order & frags <
performance

•
Interrupts Once/TCP segment


Many/IO

•
OS interface challenges

Media Interface

Ethernet


Fast Path TCP/IP

Software iSCSI

W/Partial TCP Off
-
load

TCP/IP

iSCSI

SCSI Port to OS

Host

Adapter

= SW or FW

= Hardware

Hosted by

iSCSI: Firmware TOE + iSOE


Definition

•
TCP/IP & iSCSI firmware


On
-
board processors


Who

•
Adaptec, Intel, Emulex,
QLogic


Advantages

•
Flexibility to change code

•
Low CPU load

•
Low interrupt load: < 1/IO


Disadvantages

•
No 10Gb scaling

•
Performance

•
Power, size

Media Interface

Ethernet

TCP/IP

iSCSI

Firmware TCP

and iSCSI Off
-
load

SCSI Port to OS

Host

Adapter

= SW or FW

= Hardware

Hosted by

iSCSI: Hardware TOE + iSOE


Definition

•
Hardware ASIC


TCP/IP bulk data path


iSCSI digest (CRC)

•
iSCSI in processors


Who

•
QLogic, iReady


Advantages

•
Flexibility to change iSCSI code

•
Low CPU load

•
Low interrupt load < 1/IO

•
Performance, scaling to 10G


Disadvantages

•
Complex chip

•
Lack flexibility to change TCP
code

Media Interface

Ethernet

TCP/IP

iSCSI

Hardware TCP & Firmware
iSCSI Off
-
load

SCSI Port to OS

= SW or FW

= Hardware

Host

Adapter

Hosted by

A TOEs Impact on iSCSI

Hosted by

Hosted by

By show of hands, who believes
that iSCSI allows block storage
to go unlimited distance?

Hosted by

Will iSCSI allow block storage to
go unlimited distance?


The Answer is:

Yes & No

•
Latency is the limiting factor





•
Application dependent


Transactions cannot exceed 1ms one way (100
miles)


Asynch replication is not distance dependent

“The speed of light, is not

just a limit, it’s a law.”

Hosted by

Key Block IP Storage Issues


Distance

•
Latency

•
WAN bandwidth utilization of IP


Security

•
Encryption

•
Access


Performance

•
Must be = to, or > than current expectations

Hosted by

Storage Replication over WANs


Issues

•
Good Citizen on Shared TCP/IP WANs

•
Filling the pipe > 50%

•
End
-
to
-
end throughput


Compression


TCP latency

Hosted by

Changing Paradigm for Asynch
Storage Replication


Native Storage GigE interfaces emerging

•
EMC Symm5 and DMX are available today

•
EMC CLARiiON in development

•
Hitachi developing GigE for Lightning and Thunder


Software Replication Apps over native IP

•
Leverages IP WAN already in place

•
Eliminates SAN gateway requirement (FCIP or iFCP)


Significant < cost Mirror/Replication apps

Hosted by

High Speed TCP/IP

Data Transport Challenges


Optimized for

•
Small payloads & relatively short distances


Employs inefficient

•
Error recovery & session management techniques


Delivers poor bandwidth utilization

•
For most high performance applications

•
Usually < 30% efficiency at extended distances

•
Even less as distance and bit errors increase

Hosted by

Cost of Inefficiency


Higher Bandwidth Cost

•
Despite < costs, high speed (DS3, OC3, etc) circuits =
expensive


DSC survey of 200 end
-
users

•
BW = 50
-
70 % of storage replication costs


Operational Inefficiencies

•
Can’t complete within time window delaying production
ops

•
Explosion in data exacerbates the problem

•
Current = specialized equipment & separate networks


Can’t fully leveraging IP infrastructure = > costs

Hosted by

Native GigE Replication: SRDF


Adaptive Copy & SNAP/Asynch

•
Performance degrades starting at ~ 300 miles


At 500 miles performance degradation is noticeable
& significant

Ethernet LAN/WAN Switches

TCP/IP

WAN

EMC DMX

SAN

Hosted by

NetEx HyperIP

Native GigE Replication: SRDF +
RFC 3135


Adaptive Copy & SNAP/Asynch

•
RFC 3135 = TCP/IP Performance Enhancing Proxy

•
Up to satelite distances (46K miles roundtrip)

•
90% + bandwidth utilization (T1/E1, DS3, OC3, OC12)

•
Plus 2 to 4 to 1 compression


Who

•
NetEx (HyperIP), Expand, NetCera, Digital Fountain

Ethernet LAN/WAN Switches

TCP/IP

WAN

EMC DMX

SAN

Hosted by

EMC SRDF Replication over WANs

Replication Methodologies Illustrated

Ethernet LAN/WAN Switches

Cisco MDS 9216

CNT UESR 3000

Brocade 3xxx

TCP/IP

WAN

McDATA/Nishan 3300

EMC DMX

NetEx HyperIP

= GigE

= FC

Hosted by

Considerations for Designing IP
Storage Networks


Separate LAN fabric

•
Minimally, separate VLAN


Layer 2 switching

•
Best latency for Ethernet switching

•
Nothing less than GigE


Understand LUNs

•
Mapping and Masking

Hosted by

Questions?