Business Intelligence through Data Mining - CogNova Technologies

separatesnottyΛογισμικό & κατασκευή λογ/κού

25 Νοε 2013 (πριν από 3 χρόνια και 7 μήνες)

87 εμφανίσεις

1

Dalhousie

University

CogNova

Technologies

Business Intelligence


through

Data Mining


with

Daniel L. Silver

Copyright (c), 1999

All Rights Reserved

2

Dalhousie

University

CogNova

Technologies



About myself ...


Ph.D. in Comp. Sci./Machine Learning, UWO


Chair
-
Associate, Business Informatics,



Faculty of Management, Dalhousie University


Founder of
CogNova Technologies

(London, 1993)


London Health Science Center, 3M, London Life,
MT&T, NSPI, QEII Health Science Center


My Objective ...


To discuss data warehousing and data
mining within the context of knowledge
management and business intelligence.

3

Dalhousie

University

CogNova

Technologies

CogNova Technologies Offers


Consultation
-

situation analysis and requirements
definition, selection of third party systems, project management,
and trouble shooting


Services
-

installation and application of third party
software, data analysis and model generation using CogNova
proprietary systems, summary and analysis of results


Education
-

courses and seminars on the theory and
application of data mining technologies, and the knowledge
discovery process


Research
-

investigation and development of advanced
machine learning systems and the application of KDD practices

4

Dalhousie

University

CogNova

Technologies

Outline


Introduction


Knowledge Management




and Business Intelligence


Knowledge Discovery Process


Data Warehousing and Data Mining


Opportunities, Benefits, Costs

5

Dalhousie

University

CogNova

Technologies

Introduction
-

The Buzz Words

Hype vs. Reality


Knowledge Management


Business Intelligence


Data Warehouse, Corp. Repository,
Data Mart


Knowledge Creation or Discovery


Data Mining

6

Dalhousie

University

CogNova

Technologies

Introduction
-

Motivation

Organization

Global

Opportunities

Customer

Demands

Regulatory

Change

Technological

Change

Employee

Turn
-
over

Competition

7

Dalhousie

University

CogNova

Technologies

Introduction
-

Rationale


Management of

Organizational

Knowledge

Gov’t Reg.

Competitors

Customers

Channels

Partners

Suppliers

Employees

Products

Services

8

Dalhousie

University

CogNova

Technologies

The Knowledge Management Cycle

INFORMATION

S
torage

P
rocessing

C
ommunication

Knowledge

Consolidation

Observation

and Analysis

Testing and

Application

Theory

Generation

Environmental data

Problems

Opportunities

Approach

Methods

Results

Information

“Business Intelligence”

9

Dalhousie

University

CogNova

Technologies

KM and Business Intelligence

Why should it matter to you?


Knowledge becoming substantial asset


Maximum sharing of information


Employees leave, business value remains


Betterment of internal and external
structures, personal competencies


Competitive advantage
-

leading
organizations now adopting


10

Dalhousie

University

CogNova

Technologies

KM and Business Intelligence

Key Solution Components:


Internet / Intranet & Groupware


Document management systems


EDI
-

Electronic Data Interchange


E
-
Commerce methods


Data Warehousing


Data Mining

11

Dalhousie

University

CogNova

Technologies

Knowledge Management


information
=> <=
people

Technology Centred


Info. Technologists


info. and comp.
sciences, database,
telecomm., analysis


KM = objects


explicit knowledge
-

easily encoded


People Centred


Org. Theorists


org. behavior, group
dynamics, HCI,
psychology


KM = process


tacit knowledge
-

difficult to encode

12

Dalhousie

University

CogNova

Technologies

Knowledge Management

Intellectual Capital

Human Capital
= Knowledge + Capabilities + Skill

Structural Capital
= Everything that remains after
the employees go home

Intellectual Capital
= Human Capital + Structural
Capital

Intellectual Capital
= Market Value
-

Book Value
(e.g. Microsoft’s MV = 15 * BV)

13

Dalhousie

University

CogNova

Technologies

Knowledge Management

The Invisible Balance Sheet

Assets


Liability & S.H. Equity

Cash

Accounts

Receivable

Equipment

Property

Short
-
term Loans

Long
-
term Debt

S.H. Equity

Tangible

External Structure

Internal Structure

Competence

Invisible

Share Holder

Equity

Obligation

Intangible

Book Value

Market Value

14

Dalhousie

University

CogNova

Technologies

KM and Business Intelligence

Gardner says ....


Leaders
-

will move on intangible





benefits


Followers
-

will move only on tangible




savings/profits


Others
-

will wait and try to catch up

15

Dalhousie

University

CogNova

Technologies

KM and Business Intelligence

HYPE


KM is primarily
technology centred:


Data Warehousing


Data Mining


Intranets


Groupware

REALITY


KM is primarily a
people centred
philosophy which
necessarily involves
and will promote
the use of such
technologies

16

Dalhousie

University

CogNova

Technologies

Knowledge Management

Access to Recent Information


Books:


Working Knowledge : How
Organizations Manage What They Know



T. Davenport & L. Prusak
(http://www.amazon.com/exec/obidos/ASI)


The Web:


http://www.brint.com/km/


www.sveiby.com.au


knowledge management mail
-
list:




km@MCCMEDIA.COM

17

Dalhousie

University

CogNova

Technologies


“We are drowning in information, but
starving for knowledge.”
John Naisbett






author of Megatrends





Knowledge Discovery through

Data Warehousing


and

Data Mining

18

Dalhousie

University

CogNova

Technologies

Knowledge Discovery and Data Mining


What is KDD?

A Process


The selection and processing of data for:


the identification of novel, accurate, and
useful patterns, and


the modeling of real
-
world phenomenon.


Data Warehousing
and

Data mining
are
major components of the KDD process

19

Dalhousie

University

CogNova

Technologies

The Knowledge


Discovery Process

Selection and

Preprocessing

Data Mining

Interpretation

and Evaluation

Data


Warehousing

Knowledge

p(x)=0.02

Warehouse

Internal and External

Data Sources

Patterns &

Models

Prepared Data

Consolidated

Data

20

Dalhousie

University

CogNova

Technologies

Knowledge Discovery in Context

C og No va
T ec hnologi es
9
The KDD Process
The KDD Process
Selection and

Preprocessing
Data Mi ning
Interpretation
and Evaluation
Data
Consolidation
K no w le d g e
p (x) =0.02
W are house
D ata So u r c es
P att er n s &
M o d els
P r ep ar e d Da ta
C o n s o lid a ted
D ata
Identify

Problem or

Opportunity

Measure Effect

of Action

Act on

Knowledge

“The Virtuous

Cycle”

Knowledge

Results

New Insight

Problem

21

Dalhousie

University

CogNova

Technologies

Why? …

Relationship


Marketing

a.k.a


Customer
Relationship
Management

Marketing Embraces KM, DW, DM

Marketing

Traditional

Marketing

MIS

Data

Warehousing


Data Mining



22

Dalhousie

University

CogNova

Technologies

What is Relationship Marketing
all about?


Knowing your customers
on an individual basis


Maximizing life
-
time
value not individual
sales


Developing and
maintaining a mutually
beneficial relationship


Acquire, retain, win
-
back
desirable customers

Arbuckle’s

Market

“ The Corner Store ”

23

Dalhousie

University

CogNova

Technologies

Knowledge Discovery

What can KDD do for an organization?

Impact on Marketing


Target marketing at a credit card company


Consumer usage analysis at a telecomm
provider


Loyalty assessment at a service bureau


Quality of service analysis at an appliance
chain

24

Dalhousie

University

CogNova

Technologies

The Knowledge


Discovery Process

Selection and

Preprocessing

Data Mining

Interpretation

and Evaluation

Data


Warehousing

Knowledge

p(x)=0.02

Warehouse

Internal and External

Data Sources

Patterns &

Models

Prepared Data

Consolidated

Data

25

Dalhousie

University

CogNova

Technologies

Data Warehousing

From data sources to consolidated data
repository

RDBMS

Legacy

DBMS

Flat Files

Data

Consolidation

and Cleansing

Warehouse

or Datamart

Object/Relation DBMS

Multidimensional DBMS


External

Analysis and

Info Sharing

26

Dalhousie

University

CogNova

Technologies

Data Warehousing

Operational DB


Application oriented


Current


Details


Changes continually

Data Warehouse



Subject Oriented


Current + historical


Details + Summaries


Stable

Major DW Framework suppliers / consultants:

DMR, IBM, SHL, NCR; SAS, Oracle, Sybase

27

Dalhousie

University

CogNova

Technologies


Relationship between DW and DM?

Source of

consolidated

data

Rationale

for data

consolidation

Data


Warehousing

Analysis

Query/Reporting

OLAP

Data Mining

Strategic

Tactical

28

Dalhousie

University

CogNova

Technologies

Data Warehousing


Must be business benefits driven


It’s not a project .. It’s a way of life


Keys to success are top
-
down strategy with
bottom
-
up tactical deployment:


communicate vision of Data Warehouse


construct departmental Data Marts


evolve to enterprise Data Warehouse


Rapid change in technology and business
requirements
-
>








demands short deployment cycles

29

Dalhousie

University

CogNova

Technologies

Data Warehousing

HYPE


Corporate data
stored within a DW
will solve all your
business problems

REALITY


The identification of
business problems is
the first step
-

DW,
DM are solutions


Analysis and DW
will necessarily
mature in parallel

30

Dalhousie

University

CogNova

Technologies

Data Warehousing

Access to Recent Information


Text Books:



W.H. Inmon, Claudia Imhoff



Web Pages:


DWI
-

The Data Warehouse Institute



www.dw
-
institute.com


DW Information Centre






pwp.starnetic.com/larryg


31

Dalhousie

University

CogNova

Technologies

The Knowledge


Discovery Process

Selection and

Preprocessing

Data Mining

Interpretation

and Evaluation

Data


Warehousing

Knowledge

p(x)=0.02

Warehouse

Internal and External

Data Sources

Patterns &

Models

Prepared Data

Consolidated

Data

32

Dalhousie

University

CogNova

Technologies

Knowledge Discovery Process

Core Problems & Approaches


Problems:


identification

of relevant data


representation

of data


search

for valid pattern or model


Approaches:


top
-
down
verification
by expert


interactive
visualization
of data/models


* bottom
-
up

induction

from data *

Probability

of sale

Income

Age

Data

Mining

On
-
Line

Analytical

Processing

33

Dalhousie

University

CogNova

Technologies

OLAP:
On
-
Line Analytical Processing

OLAP Functionality


Dimension selection


slice & dice


Rotation


allows change in perspective


Filtration



value range selection


Hierarchies


drill
-
downs to lower levels


roll
-
ups to higher levels


OLAP

cube

Year

by Month

Product Class

by Product Name

Sales

Region

Profit Values

34

Dalhousie

University

CogNova

Technologies

Top
-
down Verification

Technology


DEMO

Cognos
-

PowerPlay

An On
-
line Analytical Processing

(OLAP) System

35

Dalhousie

University

CogNova

Technologies

Overview of Data Mining Methods


Discovery of patterns



clustering systems

e.g. customer segmentation




Predictive modeling


regression, neural networks

e.g. target marketing, risk assessment


Descriptive modeling



inductive decision trees

e.g. client characterization

Prob.

of Sale

Age

if age > 45


and income < $32k


then ...

Age

Marital

Status

36

Dalhousie

University

CogNova

Technologies

Data Mining Technology


DEMO

Angoss
-

KnowledgeSEEKER

An inductive decision tree/rule

system

37

Dalhousie

University

CogNova

Technologies

Data Mining Example

Health Care

Situation:
A life style data on 360 persons

Problem:

Characterize those most likely
to have high/low blood pressure.

Solution:

Inductive Decision Tree

38

Dalhousie

University

CogNova

Technologies

Application Areas and Opportunities


Finance:
investment support, portfolio management


Banking & Insurance:
credit approval, risk assessment


Marketing:
segmentation, customer targeting, ...


Science and medicine:
hypothesis discovery,




prediction, classification, diagnosis


Security:
bomb, iceberg, and fraud detection


Manufacturing:
process modeling, quality control,





resource allocation


Engineering:
simulation and analysis, pattern




recognition, signal processing


Internet:
smart search engines, web marketing

39

Dalhousie

University

CogNova

Technologies

The Current Status and Trends


Standards and methodology lag technology


Many products:


micro DM packages (Cognos, Angoss)


macro
-

integrated suites (SAS, IBM)


Software costs have risen 1000% over 2 years


Beware
-

major players yet to be determined


KDD experts fear the hype being generated


Legal and ethical issues on the horizon


Internet
-

“the” sink and source of data

40

Dalhousie

University

CogNova

Technologies

Integrated Knowledge Discovery Suites

Graphical User Interface

Data

Consolidation

Selection

and

Preprocessing

Data

Mining

Interpretation

and Evaluation

Warehouse

Knowledge

Data Sources


41

Dalhousie

University

CogNova

Technologies

Benefits of KDD


Maximum utility from corporate data


discovery of new knowledge


generation of models


Important feedback to data warehousing effort


identification and justification of essential data


Reduction of application dev ’t backlog


model development
vs.
software development


Effect on bottom line of organization


cost reduction, increased productivity, risk
avoidance … competitive advantage

42

Dalhousie

University

CogNova

Technologies

Requirements and Costs of KDD


Hardware

-

computationally intensive


Software

-

micro < $20k, integrated suites < $300k


Data

-

internal collection, surveys, external sources


Human resources



DB/DP/DC expertise to consolidate and
preprocess data


Machine learning and stats competence


Application knowledge & project mgmt


70%
of the effort is expended on the data
consolidation and preprocessing activities

43

Dalhousie

University

CogNova

Technologies

KDD and Data Mining

HYPE


Expensive hardware
and software is
always required


DM is now turn
-
key
“just give it the data”

REALITY


Micro $2k
-
$10k
DM packages can
produce results


DM is data analysis
-

requires business
sense plus statistics
and AI skills

44

Dalhousie

University

CogNova

Technologies

Access to Recent Information


Book:
Data Mining Techniques for
Marketing, Sales and Customer Support,
by M. Berry & G. Linoff, Wiley & Sons


Journal:
Data Mining and Knowledge
Discovery
, Kluwer Publishing


Conference:
KDD’99


Web
-
pages:
Bus. Informatics KDD page
http://www.mgmt.dal.ca/ChrBusInf/knowdis



Knowledge Discovery Mine





http://www.kdnuggets.com

45

Dalhousie

University

CogNova

Technologies

THE END


daniel.silver@dal.ca

www3.ns.sympatico.ca/~dsilver