IP Storage: Best Practices

fullgorgedcutNetworking and Communications

Oct 24, 2013 (3 years and 9 months ago)

149 views

Hosted by

IP Storage: Best Practices

Marc Staimer, President & CDS

Dragon Slayer Consulting

marcstaimer@earthlink.net

503
-
579
-
3763

Hosted by

Agenda


IP Storage Level Setting


File vs. Block Storage


FCIP, iFCP & iSCSI


Fiction & Facts about iSCSI


Storage Replication over WANs


Considerations for Designing IP Storage Networks


Questions

Hosted by

IP Storage Level Setting


There are 3 types:


Block Storage


iSCSI


File Storage


NAS


Storage over WAN for business continuity = Block


iSCSI


FCIP


iFCP

Hosted by

IP Storage Level Setting


iSCSI Storage


Block
-
based external storage on Ethernet


Vs. SCSI, USB, 1394, or FC


NAS or Network Attached Storage


File
-
based storage = NFS & CIFS


No different than any other file server


Requires block storage behind it

Hosted by

Hosted by

By show of hands, what is
acceptable packet loss for IP
Block Storage on a LAN/WAN?

1)
1%

2)
5%

3)
10%

4)
0%

Hosted by

Acceptable Packet Loss for

IP Block Storage on a LAN/WAN


The Answer is:


Hosted by

Block vs. File Storage


IP block data is unlike any other IP data


Overwhelms most current LAN/WAN environments


Incredible amounts of traffic


Tolerates
“ZERO”

p慣k整 loss


Very low latency


File storage = specialized file server


NFS & CIFS


Higher prioritization is required depending on app


Volume of data may overwhelm untuned LAN/WAN

Hosted by

FCIP, iFCP, iSCSI


FCIP


Fibre Channel tunneled in TCP/IP


IP transport between FC switches


iFCP


IP header put on Fibre Channel frames for routing


IP connection services for FC devices


iSCSI


SCSI
-
3 mapped to TCP

Hosted by

FCIP


Pt
-
to
-
Pt: Becomes “One” FC SAN


Disruptions pass SAN
-
to
-
SAN


Large FSPF database


PSS between SAN sites


Gateways between fabrics (blades or boxes)

Ethernet LAN/WAN Switches

Cisco MDS 9216

CNT UESR 3000

Brocade 3xxx

SAN 1

SAN 2

SAN 3

SAN 4

TCP/IP

WAN

Hosted by

iFCP


Pt
-
to
-
Multi
-
Pt


Device specific passing only the data that is required


Devices can appear in multiple individual SANs


The SANs themselves remain independent

Ethernet LAN/WAN Switches

SAN 1

SAN 2

SAN 3

SAN 4

TCP/IP

WAN

McDATA/Nishan
3300

Hosted by

Parable

Hosted by

Hosted by

iSCSI: Ethernet
-
Based SAN


The Hype


Block storage on Ethernet


Leverage current infrastructure investment &
knowledge base


Lowers cost


Eliminates headaches


Ubiquitous


Makes FC another Ethernet Road Kill

Hosted by

iSCSI Defined


IETF standard protocol



Establishes & manages
connections


Carries storage (SCSI) blocks


From initiators to storage
targets


Encapsulates SCSI blocks in TCP/IP


Then tunneled in Ethernet


iSCSI is to Ethernet as FCP is to
Fibre Channel


Network application


One infrastructure for LANs,
NAS, & SANs


Ethernet





Frame



TCP/IP




Packet


SCSI
-
3

TCP/IP

iSCSI

NAS: File
Storage

GbE Switch

Mission Critical

IA App Servers

Hosted by

iSCSI Applications


NAS/SAN combined storage units


Entry level SANs


Limited budget SANs

Hosted by

SAN Benefit Assertions of iSCSI


Reduced costs


Professional services, implementation, management
& IT Staff time


Reduced complexity


Reduced Management


Increased Interoperability


Elimination of Multiple Networks


Unlimited SAN Distance


Equal or Better Performance

Hosted by

Fiction & Facts about iSCSI


Fiction


Cost is lots < Fibre Channel
(FC)


Complexity is much < than
FC


Uses current infrastructure


Requires no storage
knowledge


Is as fast as FC


Will replace FC in Enterprise


Easier to manage than FC


Eliminates SAN distance
limits


Latency (delay) is not an
issue


Facts


Known technology


Costs are relatively < FC


Cycles or hardware


Doesn’t require special HW


But benefits from it


Latency (Delay) matters


Can’t be > 1 millisecond


Deterministic routing


Doesn’t require any
-
to
-
any

Hosted by

Rating the iSCSI Value Props


Reduces or Eliminates SAN Professional Services


Lowers SAN Hardware Costs


Simplifies SAN Management


Eliminates Interoperability Issues


Converges SAN/LAN/MAN/WAN Fabric Infrastructure


Extends SANs over unlimited distances


Equal or better performance than FC SANs

Hosted by

iSCSI Reality Check



There are some real cost benefits for:


NAS/SAN on the same fabric infrastructure


SANs that don’t need the performance of FC


Entry SANs that may not even need GigE and TOEs


Hype overshadows reality:


GigE NICs with iSCSI and TOEs cost ~ same as FC HBAs


FC ports & GigE ports on server motherboards


Makes port cost differences higher for GigE w/TOEs


Very low cost simplified FC switches


Have erased much of the infrastructure HW differences

Hosted by

Sample iSCSI Vendors


Switches


Cisco


Extreme


Foundry


Enterasys


Nortel


Lucent


3Com


Gateways


Cisco


McDATA


Silicon


Adaptec


Alacritech


Intel


Siliquent


QLogic


Storage
NICs


QLogic


Intel


Alacrite
ch


Adaptec


Emulex

Hosted by

Hosted by

By show of hands, is latency
(delay) important to iSCSI block
storage?

Hosted by

Is Latency Important to IP Block
Storage?


The Answer is:



Yes, for the most part


It also depends on application

Hosted by

Hosted by

By show of hands, who
believes that TOEs & iSOEs
are an iSCSI block storage
requirement?

Hosted by

Are TOEs & iSOEs an iSCSI block
storage requirement?


The Answer is:

Not necessarily

Hosted by

Spectrum of iSCSI Adapter Solutions

Host

Adapter

Adapter Driver

TCP/IP

iSCSI

SCSI Port to OS

Software iSCSI

“NIC + Driver”

Media Interface

Ethernet

Media Interface

Ethernet


Fast Path TCP/IP

Software iSCSI

With Partial

TCP Off
-
load

TCP/IP

iSCSI

SCSI Port to OS

Media Interface

Ethernet

TCP/IP

iSCSI

Firmware TCP

and iSCSI Off
-
load

SCSI Port to OS

= SW or FW

= Hardware

Media Interface

Ethernet

TCP/IP

iSCSI

Hardware TCP

and Firmware iSCSI

Off
-
load

SCSI Port to OS

Hosted by

iSCSI: No TOE


Definition


Std Ethernet NIC


TCP/IP & iSCSI


Host
-
based in drivers


Who


Microsoft & Cisco


Advantages


Lowest cost


NICs available today


Easy integration with OS


Disadvantages


Lowest performance


High CPU load


High interrupts


Once/packet


Many/ TCP segment

Adapter Driver

TCP/IP

iSCSI

SCSI Port to OS

Software iSCSI

“NIC + Driver”

Media Interface

Ethernet

= SW or FW

= Hardware

Host

Adapter

Hosted by

iSCSI: Little TOE


Definition


NIC w/limited TOE


Packets in order & no frags


Out of order etc. go to OS


Who


Alacritech


Advantages


Relatively low cost


Small layout (low profile card)


Good throughput w/pristine
Ethernet


Disadvantages


Out
-
of
-
order & frags <
performance


Interrupts Once/TCP segment


Many/IO


OS interface challenges

Media Interface

Ethernet


Fast Path TCP/IP

Software iSCSI

W/Partial TCP Off
-
load

TCP/IP

iSCSI

SCSI Port to OS

Host

Adapter

= SW or FW

= Hardware

Hosted by

iSCSI: Firmware TOE + iSOE


Definition


TCP/IP & iSCSI firmware


On
-
board processors


Who


Adaptec, Intel, Emulex,
QLogic


Advantages


Flexibility to change code


Low CPU load


Low interrupt load: < 1/IO


Disadvantages


No 10Gb scaling


Performance


Power, size

Media Interface

Ethernet

TCP/IP

iSCSI

Firmware TCP

and iSCSI Off
-
load

SCSI Port to OS

Host

Adapter

= SW or FW

= Hardware

Hosted by

iSCSI: Hardware TOE + iSOE


Definition


Hardware ASIC


TCP/IP bulk data path


iSCSI digest (CRC)


iSCSI in processors


Who


QLogic, iReady


Advantages


Flexibility to change iSCSI code


Low CPU load


Low interrupt load < 1/IO


Performance, scaling to 10G


Disadvantages


Complex chip


Lack flexibility to change TCP
code

Media Interface

Ethernet

TCP/IP

iSCSI

Hardware TCP & Firmware
iSCSI Off
-
load

SCSI Port to OS

= SW or FW

= Hardware

Host

Adapter

Hosted by

A TOEs Impact on iSCSI

Hosted by

Hosted by

By show of hands, who believes
that iSCSI allows block storage
to go unlimited distance?

Hosted by

Will iSCSI allow block storage to
go unlimited distance?


The Answer is:

Yes & No


Latency is the limiting factor






Application dependent


Transactions cannot exceed 1ms one way (100
miles)


Asynch replication is not distance dependent

“The speed of light, is not

just a limit, it’s a law.”

Hosted by

Key Block IP Storage Issues


Distance


Latency


WAN bandwidth utilization of IP


Security


Encryption


Access


Performance


Must be = to, or > than current expectations

Hosted by

Storage Replication over WANs


Issues


Good Citizen on Shared TCP/IP WANs


Filling the pipe > 50%


End
-
to
-
end throughput


Compression


TCP latency

Hosted by

Changing Paradigm for Asynch
Storage Replication


Native Storage GigE interfaces emerging


EMC Symm5 and DMX are available today


EMC CLARiiON in development


Hitachi developing GigE for Lightning and Thunder


Software Replication Apps over native IP


Leverages IP WAN already in place


Eliminates SAN gateway requirement (FCIP or iFCP)


Significant < cost Mirror/Replication apps

Hosted by

High Speed TCP/IP

Data Transport Challenges


Optimized for


Small payloads & relatively short distances


Employs inefficient


Error recovery & session management techniques


Delivers poor bandwidth utilization


For most high performance applications


Usually < 30% efficiency at extended distances


Even less as distance and bit errors increase

Hosted by

Cost of Inefficiency


Higher Bandwidth Cost


Despite < costs, high speed (DS3, OC3, etc) circuits =
expensive


DSC survey of 200 end
-
users


BW = 50
-
70 % of storage replication costs


Operational Inefficiencies


Can’t complete within time window delaying production
ops


Explosion in data exacerbates the problem


Current = specialized equipment & separate networks


Can’t fully leveraging IP infrastructure = > costs

Hosted by

Native GigE Replication: SRDF


Adaptive Copy & SNAP/Asynch


Performance degrades starting at ~ 300 miles


At 500 miles performance degradation is noticeable
& significant

Ethernet LAN/WAN Switches

TCP/IP

WAN

EMC DMX

SAN

Hosted by

NetEx HyperIP

Native GigE Replication: SRDF +
RFC 3135


Adaptive Copy & SNAP/Asynch


RFC 3135 = TCP/IP Performance Enhancing Proxy


Up to satelite distances (46K miles roundtrip)


90% + bandwidth utilization (T1/E1, DS3, OC3, OC12)


Plus 2 to 4 to 1 compression


Who


NetEx (HyperIP), Expand, NetCera, Digital Fountain

Ethernet LAN/WAN Switches

TCP/IP

WAN

EMC DMX

SAN

Hosted by

EMC SRDF Replication over WANs

Replication Methodologies Illustrated

Ethernet LAN/WAN Switches

Cisco MDS 9216

CNT UESR 3000

Brocade 3xxx

TCP/IP

WAN

McDATA/Nishan 3300

EMC DMX

NetEx HyperIP

= GigE

= FC

Hosted by

Considerations for Designing IP
Storage Networks


Separate LAN fabric


Minimally, separate VLAN


Layer 2 switching


Best latency for Ethernet switching


Nothing less than GigE


Understand LUNs


Mapping and Masking

Hosted by

Questions?