Windows Azure Storage - Net

wittyexecutiveΔίκτυα και Επικοινωνίες

17 Φεβ 2014 (πριν από 3 χρόνια και 6 μήνες)

521 εμφανίσεις

Compute
Bound

IO
Bound

Hadoop

HPC




Internet of things

Audio / Video

Log Files

Text/Image

Social Sentiment

Data Market Feeds

eGov Feeds

Weather

Wikis / Blogs

Click Stream

Sensors / RFID / Devices

Spatial & GPS Coordinates


WEB 2.0

Mobile

Advertising

Collaboration

eCommerce

Digital Marketing


Search Marketing

Web Logs

Recommendations


ERP / CRM

Sales Pipeline

Payables

Payroll

Inventory

Contacts

Deal Tracking

Terabytes

(10E12)

Gigabytes

(10E9)

Exabytes

(10E18)

Petabytes

(10E15)


Velocity
-

Variety
-

variability

Volume

1980

190,000$

2010

0.07$

1990

9,000$

2000

15$

Storage/GB


ERP / CRM


WEB
2.0


Internet of things

Big Data,
BIG OPPORTUNITY

49% CEOs and CIOs
are planning big
data projects

Software Growth

1.8

2.5

3.4

4.6

0
5
2012
2013
2014
2015
Billions $

34%
compound
annual growth
rate

Services Growth

2.7

3.9

5.1

6.5

0
5
10
2012
2013
2014
2015
Billions $

39%
compound
annual growth
rate

1.
McKinsey&Company
, McKinsey Global Survey Results, Minding Your Digital Business, 2012

2.
IDC Market Analysis, Worldwide Big Data Technology and Services 2012

2015 Forecast , 2012




Distributed Storage

(HDFS)

Query

(Hive)

Distributed Processing

(MapReduce)

ODBC
Legend

Red = Core
Hadoop

Blue = Data
processing

Purple =
Microsoft
integration
points and
value adds

Orange = Data
Movement

Green =
Packages

Front
end

Front
end

Stream Layer

Partition Layer

Name Node


de




Data Node



Data Node



Front end

HDFS API

DFS (1 Data Node per Worker Role)

and Compute Cluster

Azure Storage (ASV)



Azure Blob Storage


public

class

FrenchSessionsJob

:
HadoopJob
<
FrenchSessionsMapper
,
SessionsReducer
>


{


public

override

HadoopJobConfiguration

Configure(
ExecutorContext

context)


{


var

config

=
new

HadoopJobConfiguration
()


{


InputPath

=
"
\
"/
AllSessions
/*.
gz
\
""
,


OutputFolder

=
"/
FrenchSessions
/
"


};


return

config
;


}


}
public

class

FrenchSessionsMapper

:
MapperBase


{


public

override

void

Map(
string

inputLine
,
MapperContext

context)


{


if

(
inputLine.Contains
(
"Country=France"
)


{


context.IncrementCounter
(
"
FrenchSession
"
);


context.EmitKeyValue
(
"FR"
,
"1"
);


}



}


}
public

class

SessionsReducer

:
ReducerCombinerBase


{


public

override

void

Reduce(
string

key,
IEnumerable
<string> values,
ReducerContext

context)


{


context.EmitKeyValue
(key,
values.Count
());


}


}