Machine Intelligence - Ohio University

journeycartAI and Robotics

Oct 15, 2013 (3 years and 9 months ago)

197 views

10th Kovacs Colloquium UNESCO


Water Resource Planning and
Management using Motivated
Machine Learning

Janusz Starzyk

School of Electrical Engineering and
Computer Science, Ohio University, USA


www.ent.ohiou.edu/~starzyk

10th Kovacs Colloquium UNESCO,
Hydrocomplexity:

New Tools for Solving Wicked Water

10th Kovacs Colloquium UNESCO




Challenges in Water Management


Embodied Intelligence (EI)


Embodiment of Mind


EI Interaction with Environment


How to Motivate a Machine


Motivated Learning


ML Experiment


Abstract Motivations and Goal
Hierarchy


Promises of EI

Outline

Flood in Poland

10th Kovacs Colloquium UNESCO


Water management is challenging since:



Strategies are developed mostly on national
level


There is a competition between countries for
water


Water policy plans effects environment and
society, health and development, and
economy


Growing demands for water


Need to integrate water management and
policy making


There is an acute need for legitimate
scientific data


Decision making in water
-
related health, food
and energy systems are critical to economical
development and national security

Challenges in Water
Management

South

North Water Transfer Project China


10th Kovacs Colloquium UNESCO


Decision makers must consider important
questions:



How do we make water use sustainable?


How to protect water resources from overuse
and contamination?


Water problems are interconnected and too
complex to be handled by a single institution or
a single group of people



It is a challenge to evolve strategies and
institutional frameworks for quick policy
changes towards an acceptable water use


It is necessary to create assessment and
modeling tools to improve policy making resolve
conflicting issues and facilitate interaction.

Challenges in Water
Management

10th Kovacs Colloquium UNESCO


Why accurate integrated models to support
decision making are important ?



Computerized models were used for many
years to support water related decisions.


Models often simplify dynamics of economic,
social and environmental interactions and lead
to inappropriate policy making and
management decisions.


This work proposes models that emerge from
interaction with real dynamically changing
environments with all of their complexities and
societal dependencies.


The main objective is to suggest an integrated
modeling framework that may assist with the
tasks of water related decision making.


Challenges in Water
Management

10th Kovacs Colloquium UNESCO


What are deficiencies of machine created models?



Artificial neural networks, fuzzy logic, and genetic
algorithms have been used to model resource
planning and water management


It is difficult to apply these tools in real
-
life decisions
as the related parameters are not explicitly known


This work presents a
machine learning approach
that
motivates

machine to develop into a useful toll.


Motivated machine learning

can characterize data
and make predictions about their dynamic changes


It could support intelligent decision making in
dynamically changing environment


It could observe impacts of alternative water
management policies


It may consider social, cultural, political, economic and
institutional elements of decision making

Challenges in Water
Management

10th Kovacs Colloquium UNESCO


Embodied intelligence may
support decision making:



EI mimics biological intelligent systems,
extracting
general principles of intelligent behaviour


It uses emerging, self
-
organizing,
goal creation

(GC)
system that motivates EI to learn how to interact with
the environment



Knowledge is not entered into such systems, but is a
result of useful actions in a given environment.


This knowledge is validated through active interaction
with the environment.



Motivated intelligent systems adapt to unpredictable
and dynamic situations in the environment by learning,
which gives them a high degree of autonomy


Learning in such systems is incremental, with
continuous prediction of the input associations based
on the emerging models
-

only new information is
registered in the memory

Challenges in Water
Management

10th Kovacs Colloquium UNESCO


How to use the motivated learning scheme to integrate
modelling and decision making?



It is suggested to apply ML approach to water
management in changing environments where the
existing methods fail or work with difficulty.


For

instance,

using

classical

machine

learning

to

represent

physical

processes

works

only

under

the

assumption

that

the

same

processes

will

repeat
.


However,

if

a

process

changes

beyond

certain

limits,

the

prediction

will

not

be

correct
.



ML

systems

may

overcome

this

difficulty

and

such

systems

can

be

trained

to

advice

humans
.


Design concepts, computational mechanisms, and
architectural organization of embodied intelligence
are presented in this talk


The talk will explain internal motivation mechanism
that leads to effective goal oriented learning, abstract
goal creation and goal management

Challenges in Water
Management

10th Kovacs Colloquium UNESCO


Intelligence


http://www.home
-
business
-
smarts.net/

Mainstream Science on Intelligence
December 13, 1994:
An Editorial With
52 Signatories, by Linda S. Gottfredson,
University of Delaware


Intelligence is a very general
mental capability that, among
other things, involves the ability
to
reason, plan, solve problems,
think abstractly, comprehend
complex ideas, learn quickly and
learn from experience
.


10th Kovacs Colloquium UNESCO


Animals’ Intelligence


Defining intelligence
through humans is not
appropriate to design
intelligent machines:


Animals are intelligent too




Dog IQ test:


Dogs can learn 165 words (similar to 2 year olds)


Average dog has the mental abilities of a 2
-
year
-
old child (or better)


They would beat a 3
-

or 4
-
year
-
old in basic arithmetic,


Dogs show some basic emotions, such as happiness, anger and disgust


“The social life of dogs is very complex
-

more like human teenagers
-

interested in who is moving up in the pack, who is sleeping with who etc,“
says professor Stanleay Coren from University of British Columbia

10th Kovacs Colloquium UNESCO


Computational Models of Intelligence


Five paradigms of
Computational
intelligence

http://rtpis.mst.edu/images/paradigms_of_CI.jpg


How to define and
compute
intelligence?

10th Kovacs Colloquium UNESCO


Traditional AI



Embodied Intelligence


Abstract intelligence


attempt to simulate
“highest” human faculties:


language, discursive
reason, mathematics,
abstract problem solving


Environment model


Condition for problem
solving in abstract way



“brain in a vat”



Embodiment


knowledge is implicit in the
fact that we have a body


embodiment supports brain
development


Intelligence develops
through interaction with
environment


Situated in environment


Environment is its best model


10th Kovacs Colloquium UNESCO


Design principles of intelligent systems

from Rolf Pfeifer “Understanding of Intelligence”, 1999



Interaction with
complex environment


cheap design


ecological balance


redundancy principle


parallel, loosely
coupled processes


asynchronous


sensory
-
motor


coordination


value principle


Agent

Drawing by Ciarán O’Leary
-

Dublin Institute of Technology

10th Kovacs Colloquium UNESCO


Embodied Intelligence




Mechanism:

biological, mechanical or virtual agent


with embodied sensors and actuators


EI acts on environment and perceives its actions


Environment hostility is persistent and stimulates EI to act


Hostility:

direct aggression, pain, scarce resources, etc


EI learns so it must have associative self
-
organizing memory


Knowledge is acquired by EI


Definition


Embodied Intelligence (EI) is a
mechanism that learns how to survive
in a hostile environment

10th Kovacs Colloquium UNESCO


Embodied Intelligence



For w
ater
r
esource
p
lanning and
m
anagement

hostility of the environment means



Insufficient water resources


Poor water quality


Growing demand of industry for water


Conflicts between stakeholders, etc



These hostile signals represent the primitive
pains that grow unless they are addressed by
proper actions



Surviving in this environment (politically) is to
keep these signals below specified level,
otherwise economical crises, social unrest,
drought or famine will follow


10th Kovacs Colloquium UNESCO


Embodiment of a Mind



Embodiment is a
part of the
environment that EI controls
to
interact with the rest of the
environment


It contains intelligence core
and sensory motor interfaces
under its control


Necessary for development of
intelligence


Not necessarily constant or in
the form of a physical body


Boundary transforms
modifying brain’s self
-
determination

10th Kovacs Colloquium UNESCO



Brain learns own body’s dynamic


Self
-
awareness is a result of
identification with own embodiment


Embodiment can be extended by
using tools and machines


Successful operation is a function
of correct perception of
environment and own embodiment


Embodiment of a Mind

10th Kovacs Colloquium UNESCO


INPUT

OUTPUT

Simulation or

Real
-
World System

Task

Environment

Agent Architecture

Long
-
term Memory

Short
-
term Memory

Reason

Act

Perceive

RETRIEVAL

LEARNING

EI Interaction with Environment

From Randolph M. Jones, P : www.soartech.com

10th Kovacs Colloquium UNESCO


How to Motivate a Machine

?


The fundamental question is how to
motivate a machine to do anything, in
particular to increase its “brain”
complexity?


How to motivate it to explore the
environment and learn how to
effectively work in this environment?


Can a machine that only implements
externally given goals be intelligent?

If not how these goals can be
created
?



10th Kovacs Colloquium UNESCO






I suggest that hostility of environment motivates us
.


It is the pain that moves us.


Our intelligence that tries to minimize this pain motivates our actions,
learning and development


We need both the environment hostility and the mechanism
that learns how to reduce inflicted by the environment pain


How to Motivate a Machine

?




In this work I propose, based on the
pain, mechanism that motivates the
machine to act, learn and develop.


Without the pain there will be no motivation to
develop
.

10th Kovacs Colloquium UNESCO


Motivated Learning






I suggest a goal
-
driven mechanism to motivate
a machine to act, learn, and develop.


A simple pain based goal creation system.


It uses externally defined pain signals that are
associated with primitive pains.


Machine is rewarded for minimizing the primitive
pain signals.


Definition: Motivated learning (ML)
is learning based on the
self
-
organizing system of goal creation in embodied agent.


Machine creates abstract goals based on the primitive pain signals.


It receives internal rewards for satisfying its goals (both primitive and
abstract).


ML applies to EI working in a hostile environment.

10th Kovacs Colloquium UNESCO


Pain
-
center and Goal Creation


Simple Mechanism


Creates hierarchy of
values


Motivation is to reduce
the primitive pain level


Leads to formulation of
complex goals


Reinforcement :


Pain increase


Pain decrease


Forces exploration


+

-

Environment

Sensor

Motor

Pain level

Dual pain level


Pain increase

Pain decrease

(
-
)

(+)

Motivation

(
-
)

(
-
)

(+)

(+)

Wall
-
E’s
goal is to keep

his plants from dying

(+)

(
-
)

Goal

10th Kovacs Colloquium UNESCO


Primitive Goal Creation

-

+

Pain

Dry soil

Primitive

level

open

tank

sit on

garbage

refill

faucet

w. can

water

Dual pain


Reinforcing a proper action


10th Kovacs Colloquium UNESCO


Abstract Goal Hierarchy




Abstract goals

are
created to reduce
abstract pains

in order to
satisfy the primitive goals


A hierarchy of abstract
goals

is created
-

they
satisfy the lower level
goals

Activation

Stimulation

Inhibition

Reinforcement

Echo

Need

Expectation

-

+

+

Dry soil

Primitive
Level

Level I

Level II

faucet

-

w. can

open

water

+

Sensory pathway

(perception, sense)

Motor pathway

(action, reaction)

Level III

tank

-

refill

10th Kovacs Colloquium UNESCO


Motivated Learning Experiment

Sensory
-
motor pairs and their effect on the environment

Sensory

Motor

Increases

Decreases

Dry soil

Water from Can

Moisture

Water in Can

No Water in Can

Water from Tank

Water in Can

Water in Tank

No Water in Tank

Water from Reservoir

Water in Tank

Water in Reservoir

No Water in Reservoir

Water from Lake

Water in Reservoir

Water in Lake

No Water in Lake

Regulate Usage

Water in Lake

-

Case study: “How can
Wall
-
E

water his plants if
the water resources are limited and hard to find?”

10th Kovacs Colloquium UNESCO


Action scatters in 5 ML simulations

Motivated Learning Experiment

10th Kovacs Colloquium UNESCO


The average pain signals in 100 ML simulations

0

100

200

300

400

500

600

0

0.5

Primitive pain


dry soil

Pain

0

100

200

300

400

500

600

0

0.1

0.2

Lack of water in can

Pain

0

100

200

300

400

500

600

0

0.1

0.2

Lack of water in tank

Pain

0

100

200

300

400

500

600

0

0.1

0.2

Lack of water in reservoir

Pain

0

100

200

300

400

500

600

0

0.05

0.1

Lack of water in lake

Pain

Discrete time

Motivated Learning Experiment

10th Kovacs Colloquium UNESCO


Averaged performance over 10 trials:

ML

RL

Machine using ML learns to control all abstract pains and
maintains the primitive pain signal on a low level.


ML vs. Reinforcement Learning

10th Kovacs Colloquium UNESCO


Multiple
dependencies:

-

two resources that
can provide money

Motivated Learning Experiment II

10th Kovacs Colloquium UNESCO



When the
environment is
abundant in both
resources


Competition
between
Work
and
Sell valuables


The loser and
its associated
further goals will
be ignored by the
system

Motivated Learning Experiment II

10th Kovacs Colloquium UNESCO


ML Abstract Goal Hierarchy

10th Kovacs Colloquium UNESCO


Compare ML and RL

Mean primitive pain P
p

value as a function of
the number of
iterations.

>10 levels of hierarchy
>complex environment


-

green line for RL

-

blue line for ML.

10th Kovacs Colloquium UNESCO


Reinforcement Learning


Motivated Learning


Single value function


Measurable rewards


Can be optimized


Predictable


Objectives set by
designer


Maximizes the reward


Potentially unstable


Learning effort increases
with complexity


Always active



Multiple value functions


One for each goal


Internal rewards


Cannot be optimized


Unpredictable


Sets its own objectives


Solves minimax problem


Always stable


Learns better in complex
environment than RL


Acts when needed


http://www.bradfordvts.co.uk/images/goal.jpg

10th Kovacs Colloquium UNESCO


Machine Working for Humanity?


If you’re trying to look far ahead, and
what you see seems like science
fiction, it might be
wrong
.


But if it

doesn’t seem
like science
fiction, it’s

definitely wrong.

From presentation by Foresight Institute

10th Kovacs Colloquium UNESCO


Questions?