Megatrends and Research Challenges in IT Management

wispxylopolistInternet and Web Development

Aug 7, 2012 (4 years and 11 months ago)

327 views

2008
-
12
-
05
:

ICSOC
08

SOA

Summer

School

1


©

Donald

F
.

Ferguson

2008

Megatrends and Research Challenges

in IT Management

Dr. Donald Ferguson

Distinguished Engineer,

Chief Architect for Enterprise IT Management (Products)

donald.ferguson@ca.com

http://www.donald
-
ferguson.net/blog


Megatrends

Performance of
hardware

Service
-
oriented
architecture

Digitization

of the economy

Software
breakthroughs

Digital
workstyle
,

lifestyle

Wireless, portable,
Tablet PC, phone

2

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

Megatrends

>
Performance of HW


Multi
-
processing instead of clock speed.


Memory replaces disk.


Bandwidth

>
Enables


Virtualization (VMs)


Distributed processing


New programming breakthroughs and apps

>
Enables


The Cloud


New device types

>
Enables


Digital Lifestyle and Work Style


Digitization of the economy

>
If done right, improves


Causes efficiency


Productivity



GDP


GDP/Person


Growth of Real Person/Income



3

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

Implications for Enterprise IT

>
Virtualization: The Best of Times, The Worst of Times


Reduces the number of SW configurations


Increases the number of images


But, need to virtualization the
Composite Applications.

>
The Cloud


Distributed Security Model


IT Management and Governance span organizations

>
New Device Types, Digital Work Style/Lifestyle


Complex Information Governance and Compliance


Reinforces the need for public Cloud applications and CDNs

>
Programming Breakthroughs


Model Driven Development


New conceptual model for
application lifecycle and management.


Web 2.0 drives a completely new datacenter.


Different IT organization and communication with business
stakeholders.



4

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

WWW Evolution: The Structure

Linked

Documents

Dynamic

Documents

Web Services

Web

is a

Database

Editable Web

Programmable

Web

Knowledge

about

Knowledge

Web

is the

Desktop

Picasso

HTML, HTTP, URLs, …

CGI
-
Bin,

Web App Servers

for data driven pages.

Call the “page”

from a program.

Content explosion and

rate of change made

the terminal/menu

like HTML

navigation impractical

Dynamic pages

and basic

skills explode contents

Gadgets,

SaaS,

mashups,

etc.

5

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

The Evolving Web

I.
Distributed Documents


Authors produce, extend and reference content.


End
-
users can read and follow content.


Use cases produced requirements for a common document
model:


HTML, Browsers


HTTP, JPEG, URL, …

II.
Data Driven Documents


End
-
users wanted to see information, often in tables, that
they choose.


Specific simulations and experiments.


Evolved to my frequent flier account or online banking.


Motivation for CGI
-
Bin.


Evolved to


JSPs, ASPs, J2EE, etc for simpler development.


Web application servers for efficiency, application
manageability, portability, …

6

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

The Evolving Web

III.
Further Customization and Personalization


Move beyond simple user selected data via forms.


Personalization


Server/site chooses page content (tiles) based on


User profile and properties


Navigation and information in the current session.


Marketing, promotion, … rules and goals


Examples


Amazon


WebSphere Personalization


End
-
user customization


User chooses or customizes pages


Portlet

catalog


Portal page customizer


Within certain constraints


Delegated administration.


Customized page includes interaction optimization


Examples


My Yahoo


WebSphere Portal Server


iViews


Mashups are the latest incarnation, and a straightforward evolution


AJAX emerged for better browser interaction.


RPC bypassed the Web forms to call the application directly.

7

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

The Evolving Web

IV.
Too Much Content


We used to joke


HTML is just new 3270 maps


Browser is simple a new terminal


But, basic HTML and browser model is similar to menu driven
terminal applications.


The amount of content exploded making the simple page,
menu, link model impractical.


Finding became the most critical and common approach


Search: Google, Yahoo


Aggregators
(Connector)
emerged whose sole model was
domain aware intermediary, e.g.
Froogle
,
eopinion
, CNET.

V.
Turn the Web around


Users became more Web savvy and could author content.
Empowered mavens.


Drove the emergence of blogs, wikis, …


Examples: Wikipedia.org,
Blogspot
,
Wordpress


Exacerbated the content problem,
but provided access to
Mavens.


Examples:
www.whatsrunning.net
,
www.seatguru.com



8

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

The Evolving Web

VI.
The Programmable Web


Mashups, web sites and AJAX transformed the way sites
made applications available.


Well designed applications have a clean separation between
UI, application and data tiers.


The UI running in the browser spawned lightweight RPC
mechanisms (REST) and events (Feeds).


Sites began publishing APIs for developers, including


PHP and Java API libraries for the Amazon Flexible Payment
Service.


An entire book for PHP programming using APIs for eBay,
Google, PayPal, Amazon, FedEx.


Simple tools and examples have emerged to guide
developers, e.g. www.programmableweb.com.

9

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

The Evolving Web

VI.
The Programmable Web (cont)


An inevitable evolution was marrying the wiki concept with
the Programmable Web concept to produce
application
wikis.


Relatively sophisticated end
-
users can


Build data driven wiki/Web sites


To reproduce the value from Web application servers.


The Programmable Web and simple tools are


Empowering end
-
users in a way analogous to


Spreadsheets


Simple query builders and report generators.


This benefits business users because the Mavens share the
situation applications with them, improving their
productivity.

10

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

Everyone Cane Program

>
Some anecdotes


I study karate


I have a black belt


Don’t let the affect your appraisal of my presentation


There are a lot of high school students in the dojo.


One “kid” wrote


A Microsoft Access application


Tracks students class attendance


Uses a bar code scanner


Prints reports


Kids in the locker room know I worked for IBM


Tell me about their cool PHP, Perl, … …


Tell me about their cool
prog

projects, dynamic Web sites

>
Everyone can program the way we learned

long division.


11

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

Everyone Can Program

>
Severe weather


Meet Dave, an insurance agent.


He sees a news report of a severe storm. What is the company’s risk?


Dave writes a simple script to retrieve affected zip codes and rainfall.


Uses an Excel control fronting Web services to retrieve insured
properties; Filter and select only properties insured for more than
$250,000.


Pass rainfall predictions to ESRI to compute floods level at properties.


Email a risk summary to executives.

>
Emailing documents is frustrating


Changes upon changes upon changes upon comments


Multiple versions


I don’t have the right editor


Dave will make his “reports” a wiki page that refreshes the data


Sever side “
mashup



LAMP


Have the application refresh when the forecast changes, using feeds.



12

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

Everyone Can Program

>
Programming skills, but it does not.


Feeds, URIs and simple REST retrieval of form like data are not
dissimilar to including Access data in a spreadsheet.


Passing information between the various “parts” is similar to
spreadshseets
, references and formulas.


Very simple scripting to sequence actions


Use of basic primitives, like print and e
-
mail the form.

>
This is a new composite application


Composes public data


Composes data and behavior in operational, enterprise systems


Sequences steps


Supports user interaction


Derives new results

>
Reality Roll out


Mavens do what they did for spreadsheets and share


With connectors which


Distributes to everyone


13

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

Simple
Assemebly

and Control Flow

Has support for

MW blocks

14

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

Yahoo Pipes

Yahoo Pipes

15

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

What Are the Implications?

>
Current “IT” Model


The “data center”


Runs a relatively small number of applications


For a relatively large number of employees or customers


The applications change very slowly, to support QA


LOB, department or team have “immediate” need

for some new functions


Respond to a complex RFP/RFP collaborating with new partner


Complex trouble ticket that needs a new team


Extend the managers workspace in in a store to integrate with
local carpet installation, electricians, cabinet makers


What to do?


Submit a plea to the IT department or ISV for a change


Have the priests of programming make the change


Have the data center teams run QA

>
Well,


The business need is no longer there


But, I have a 25 year old who can program “mashups” or LAMP


The new “teenagers” have the attention of a ferret on an
espresso.


They will not wait for the priests



16

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

Lines of Business and IT

>
LOBs and casual application developers build and test
applications, perhaps just in time applications.

>
Pressures will drive application redeployment into IT
center


Governance and compliance


Server management is a drag


Publication outside the enterprise

Natural

PHP/LAMP/AJAX

Development

Experience

Snarf and Redeploy

17

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

Evolving Datacenters

Metric


Today


Tomorrow

No. of servers


1,000



1,000

No. of apps


≈ 10





100,000

Apps. change




6
-
9 months




2
-
3 days

Users/app


1,000,000


10

Challenges


Availability


Isolation






Scale
-
out


Governance





Change mgmt

Delegated
admin





… …







Additive
problem
for middleware and IT:

“Build a mission critical
middleware solution for integrating core business systems into
enterprise composite applications,
and

provide a platform for
100,000 server side, autonomous mashups
.”

18

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

Evolving Datacenters

Metric


Today


Tomorrow

No. of servers

1,000



1,000

No. of apps


≈ 10





100,000

Apps. change


≈ 6
-
9 months


≈ 2
-
3 days

Users/app


1,000,000


10

Challenges


Availability


Isolation






Scale
-
out


Governance





Change mgmt

Delegated
admin





… …







Additive
problem
for middleware and IT:

“Build a mission critical
middleware solution for integrating core business systems into
enterprise composite applications,
and

provide a platform for
100,000
server side, autonomous mashups
.”

Oh.

Do not forget that this has
to embrace the Cloud and

govern, manage and secure

the whole thing.

19

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

20

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

Actual IT Architecture

Too many connections.

What do I connect where?

What
happens if I change something?



No single place that

knows what has

done what.


How can I figure out the

status of my request?



2008
-
12
-
05
:

ICSOC
08

SOA

Summer

School

21


©

Donald

F
.

Ferguson

2008

Integration Platform

Unknown connection.

Complex configuration.

NxN formats and protocols

Directory of Services

Adaptors

Normalized

Protocols

Route, Transform,

Log, Split/Join,

… …

M1

M2

M3

22

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

End
-
to
-
End SOA



SOA is about
Composite Applications


Composite applications are not a new concept,


But SOA adds new complexity, e.g.


Cross
-
organization


Dynamic routing and binding


Logical, end
-
to
-
end process

Applications

Firewall

Network

Switch

Load
Balancer

Portal

Credential
Store

End User

Web

Servers

Router

SAP

PSFT

Siebel

Web Services

Other
Applications

Databases

Mainframe

Database

23

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

Emerging Scale and Complexity

24

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

Management of SOA


SOA significantly reduces complexity.


Common approaches to


Formats and Protocols


Defining Interfaces


Policies


etc.


Prebuilt integration between vendors.


Builds on Internet technologies, e.g. HTTP.


But primarily focuses on application development,
business process modeling, …


Significant challenges and unsolved problems for IT
management.


End
-
to
-
end monitoring.


Vast scale of message processors and dynamic states.


Old concepts like topology and connections no longer apply.



25

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

Defining the Cloud


Four Key Groupings

25

Process
-
as
-
a
-
service

Applications
-
as
-
a
-
service

Platforms
-
as
-
a service

Infrastructure
-
as
-
service

A Scenario

Contoso

Acme Travel

Receive
Filter
Transform
Transform
Cancel
New
Hotel
Reservation
Success
Inform
Car
Service
New
Hotel
Reservation
Success
Feed
Contoso


Mid
-
size insurance company


Works with independent agents,


branches, …


Relies on Acme Corporate Travel

Acme Travel


Self
-
service portal


B2B links to airlines, hotels, etc.


Internal itinerary mgmt and other

processes.


User identity and profile info.

26

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

A Scenario

Contoso

Acme

Travel

Receive
Filter
Transform
Transform
Cancel
New
Hotel
Reservation
Success
Inform
Car
Service
New
Hotel
Reservation
Success
Feed
We need to have

a regional partners

meeting.

27

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

A Scenario

Contoso

Acme

Travel

Receive
Filter
Transform
Transform
Cancel
New
Hotel
Reservation
Success
Inform
Car
Service
New
Hotel
Reservation
Success
Feed
We need to have

a regional partners

meeting.

Small IT staff

wants to build

simple, customizable,

connected BPs for

meeting mgmt.

Needs to quickly do a POC
for entering the

convention mgmt

business, and then

expand into production.


Each customer is different.

Need to enable

connectivity with

people and simple

apps

End Users and

SMBs build
mashups

or
quick

Modification to
Web sites

28

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

Internet Service Bus

Identity

Services

Connec
-
tivity

Workflow

Other identity

providers

Web UI

Workflow uses the

Connectivity

Services

Communication with

other enterprises

through the

Connectivity Service

Attached

building block

services

29

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

An Example

Cancel

XML

“LE
-


“OL”

XML


䡔䵌H

E浡楬

H呔倠偏ST

Cancel a town car

reservation

Simple select base

on reservation confirmation

prefix.

Simple adaptors

30

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

Functions (Over Time)

>
Connectivity


Multiple formats and protocols


e.g. WS
-
*, WCF, REST, …


Protocol adaptation (e.g. WS
-
*


REST)


Firewall traversal enables intra
-
”thing” servers


Builds on HTTP protocols


Security equivalent to the way browsers work (e.g. mashups)


Value added functions


Transformation


Routing, Multicast


WS
-
Eventing

(pub/sub)


WS
-
ReliableMessaging


… …

>
Workflow


Simple workflow (that stitches together services, which may be
enterprise workflows).


Palette of useful activity types

>
Identity and Access


Based on WS
-
Federation


Create and manage identities and communities through a UI


Manage access to “my in the bus URLs”

31

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

Applications and URIs

>
sb://contoso.com/


Conventions/


SFO
-
2007
-
Nov
-
12/


Attendees/


Meetings/





NYC
-
2007
-
Dec
-
10








PolicyManagement
/


… …

>
sb://ImperialHotels








Little Scripts

Associate workflows, pub/sub, reliable messaging,

Protocol and message transformation, …

With URIs

32

SSIAE Summer School


June 2009



© Donald
F.
Ferguson

33

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

Internet Service Bus


Most of the focus in cloud computing is on


Base infrastructure and SW


Applications


The world is evolving to a programmable web.
Every API I ever wanted is on the Web.


Convergent evolution will cause the rise of Internet
Service Buses (ISB).


Creates new challenges in security and
management.


Federated and multiple span of control.


Privacy versus transparency.


Reputation and SLAs


Scale





34

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

Pre
-
Management Using Web Web Services

Disparate


Formats and protocols


Instrumentation APIs (CLI,
Config

File, SQL)


Managed object models and
metamodels


RDB Catalog


XSD/XML


Name/value pairs


UIs and portals


Customization and scripting, defining processes


Multiple, conflicting agents on a systems



Web UI

JMX

Protocol XX

SNMP

Management

Applications

35

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

Management Using Web Services


System and application management is similar to
general purpose integration


Enterprise Service Bus


Business Process Modeling/Management


Information Integration/Federation


Portals


MUWS


Applies SOA/Web service integration


To systems and application management


Benefits


Replace arcane tools with Java, Eclipse, etc.


Reuse common skills


Systems/Applications Management is often part of
larger end
-
to
-
end business processes, e.g. employee
on boarding drives security, desktop
config
, payroll

and HR

2008
-
12
-
05
:

ICSOC
08

SOA

Summer

School

36


©

Donald

F
.

Ferguson

2008

Logical Model

M1

M2

M3

Management

Products

Management

Product

Management Solution AAA

Management Solution BBB

37

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

Logical Model


Define a coherent “To Be” schema


End
-
points and managers have “As Is” schema in
subordinate
management data repositories.


Build a
Configuration Management Database (CMDB)


Integrate MDRs


CMDB


Extract the
business services


Composites, like UML Instance Diagrams.


Transaction flows through business services


Define
end
-
to
-
end business processes
,
workflows

and
methods
for


Granting access rights to a service.


Applying a set of patches


etc.


Build a
management portal


A set of
portlets


Workflows for task requiring humans, e.g. trouble ticket.


Launch in context



Taking a work item provides a page with
the information you need to perform the task.

38

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

CMDB

39

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

CIM Example

See Core Model
,
page
2
LogicalElement
ManagedSystemElement
See Core Model
,
page
1
See Core Model
,
page
1
ManagedElement
*
Component
*
Dependency
*
*
System
See Core Model
,
page
2
J
2
eeDomain
Vendor
:
string
Version
:
string
J
2
eeServer
DeploymentDescriptor
:
string
J
2
eeDeployedObject
J
2
eeModule
J
2
eeApplicationModule
*
Title
:
App Server CIM Model
2
.
13
Filename
:
CIM
_
Application
-
J
2
eeAppserver
.
vsd
Author
:
DMTF Application WG
Date
:
07
September
2006
Page
1
:
-

J
2
EE Server
Page
2
:
J
2
EE Modules
&
Beans
Page
3
:
J
2
EE Resource
&
Indication
Page
4
:
J
2
EE Statistics
(
1
)
Page
5
:
J
2
EE Statistics
(
2
)
Page
6
:
Association Hierarchy
Page
7
:
Aggregation Hierarchy
0
..
1
1
..
n
InstanceID
:
{
Key
}
J
2
eeResource
J
2
eeResourceOnServer
1
See Core Model
,
page
2
EnabledLogicalElement
See Application Model
ApplicationSystem
J
2
eeManagedObjectCapabilities
StateManageable
:
boolean
StatisticsProvider
:
boolean
EventProvider
:
boolean
J
2
eeJVM
InstanceID
{
Key
}
JavaVendor
:
string
JavaVersion
:
string
Node
:
string
J
2
eeServerUsesJVM
1
..
n
*
SystemResource
See System Model
,
page
9
J
2
eeServerInDomain
1
Service
See Core Model
,
page
2
Capabilities
See Core Model
,
page
8
See Core Model
,
Page
2
AdminDomain
Association
Aggregation
Association with WEAK reference
Inheritance
Aggregation with WEAK reference
w
w
*
Equivalent to
:
0
..
n
Composition Aggregation
1
..
n
J
2
eeManagedObject
InstanceID
{
Key
}
StartTime
:
datetime

StartRecursive
() :
uint
32

J
2
eeApplication
J
2
eeApplicationHostedOnServer
1
*
40

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

Business Service Management


Business Service


Types


Structure


End
-
to
-
End Trans.


End
-
to
-
End Mgmt

Processes


Implement a
logical operation
on the business
service.


Map to operations
on the actual
objects


Which flow
through to the
subordinate
MDRs.

A Very Hard Problem:

Policy, rule algorithms for
inferring

relationships

between resources, message exchanges, etc.

41

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

CMDB

Information Integration and Master Data
Management Problem


Schema mapping and conversion


Complex, federated query


How do updates work?


Rules and policies for data cleansing,
e.g


Are “Donald Ferguson” and “Ferguson, D.F.” the same user?


Are


{168.1.0.202, “
dfflaptop
”} and


{168.1.0.202, “
DonLaptop
} the same?


Or is the data bad?


Compute and derive relationships from the properties
because no one explicitly created them.

42

SSIAE

Summer

School

June

2009


©

Donald

F
.

Ferguson

2009

MUWS


Summary and Conclusions


MUWS focuses on ESB and process patterns.


There is a fundamental, neglected aspect


“Information Integration.”


But introduces challenges not present in class
EII/MDM


Scale


Dynamic information


Operate through the data


But, once there is a data model, the problem
becomes very “patterned.”


There is an opportunity to create a DSL


Focused on IT


Extensible operators and templates


Decomposition into operations on sub
-
elements