Try to save computation time in large- scale neural network modeling with population density methods, or just fuhgeddaboudit

chickenchairwomanΤεχνίτη Νοημοσύνη και Ρομποτική

19 Οκτ 2013 (πριν από 4 χρόνια και 8 μήνες)

95 εμφανίσεις

Try to save computation time in large
scale neural network

modeling with population density
methods, or just fuhgeddaboudit

Daniel Tranchina, Felix Apfaltrer & Cheng Ly

New York University

Courant Institute of Mathematical Sciences

Department of Biology

Center for Neural Science

Supported by NSF grant BNS0090159


Can the population density function (PDF) method be made
into a practical time
saving computational tool in large
neural network modeling?

Motivation for thinking about PDF methods

General theoretical and practical issues

The dimension problem in realistic single
neuron models

Two dimension reduction methods

Moving eigenfunction basis (Knigh, 2000)

only good news

Moment closure method (Cai et al., 2004)

good news; bad news; worse news; good news

Example of a model neuron with a 2
D state space

Future directions

Why Consider PDF Methods in Network Modeling?

Synaptic noise makes neurons behave stochastically: synaptic
failure; random sizes of unitary events; random synaptic delay.

Important physiological role: mechanism for modulation of
gain/kinetics of population responses to synaptic input;
prevents synchrony in spiking neuron models as in the brain.

Important to capture somehow the properties of noise in
realistic models.

Large number of neurons required for modeling physiological
phenomena of interest, e.g. working memory; orientation
tuning in primary visual cortex.

Number of neurons is determined by the
; e.g. an orientation hypercolumn in V1:

TYPICAL MODELS: ~ (1000 neurons)/(orientation
hypercolumn) for input layer V1 ( 0.5 X 0.5 mm

or roughly
0.25 X 0.25 deg

34,000 neurons
75 million synapses

Many hypercolumns are required to study some problems,
e.g. dependence of spatial integration area on stimulus

Why Consider PDF Methods

Why PDF?


Tracking by computation the activity of ~10

neurons and

synapses taxes computational resources: time and

e.g. 8 X 8 hypercolumn
model for V1 with 64,000 neurons

(Jim Wielaard and collaborators, Columbia):

1 day to simulate 4 seconds real time

But stunning recent progress by Adi Rangan & David Cai

What to do?

Quest for the
Holy Grail
: a low
dimensional system of
equations that approximates the behavior of a truly high
dimensional system.

Firing rate

(Dyan & Abbott, 2001): system of ODEs or



(system of PDEs or integro

The PDF Approach

Large number of interacting stochastic units suggests a
statistical mechanical approach.

Lump similar neurons into discrete groups.

Example: V1 hypercolumn. Course graining over: position,
orientation preference; receptive
field structure (spatial
phase); simple
complex; E vs. I may give ~ 50
neurons/population (~ tens OK for PDF methods).

Each neuron has a set of dynamical variables that
determines its state; e.g.

for a leaky I&F neuron.

Track the distribution of neurons over state space and
firing rate for each population.

Rich History of PDF Methods in Computational

Wilbur & Rinzel 1983

Kuramoto 1991

Abbott & van Vreeswijk 1993

Gerstner 1995

Knight et al., 1996

Omurtage et al., 2000

Sirovich et al., 2000

Casti et al. 2002

Cai et al., 2004

Huertas & Smith, 2006

PDF Methods Recently Espoused and Tested as a Faster
Alternative to Monte Carlo Simulations

PDF Theory

Most applications of PDF methods as a computational tool
have involved single
neuron models with a 1
D state space:
instantaneous synaptic kinetics; V jumps abruptly up/down
with each unitary excitatory/inhibitory synaptic input event.

Synaptic kinetics play an enormously important role in
determining neural network dynamics.

Bite the bullet and include realistic synaptic kinetics.

Problem with PDF methods: as underlying neuron model is
made more realistic, dimension of the state space increases,
so does the computation time to solve the PDF equations.

Time saving advantage of PDF over (direct) MC vanishes.

Minimal I&F model with synaptic kinetics has 3 state variables:

voltage, excitatory and inhibitory conductances.

Minimal I&F Model: How Many State Variables?

Take Baby Steps: Introduce Dimensions One at a
Time and See What We Can Do



PDF vs. MC and Mean
Field for 2
D Problem.

PDF cpu time is ~ 400 single uncoupled neurons

cpu time: 0.8 s for PDF; 2 s per 1000 neurons for MC.

1000 neurons

100,000 neurons




Computation Time Comparison: PDF vs. Monte Carlo (MC):

PDF grows linearly; MC grows quadratically

50 neurons per population;1 run; 25% connectivity

PDF Method is plenty fast for model neurons
with a 2
D state space.

More realistic models (e.g. with E and I
input) require additional state variables

Explore dimension reduction methods.

Use the 2
D problem as a test problem

Dimension Reduction by Moving Eigenvector Basis:

Bowdlerization of Bruce Knight’s (2000) Idea

Dimension Reduction by Moving Eigenvector Basis

Example with

state space, instantaneous synaptic kinetics

Only 3 eigenvectors
for low, and 7 for
high synaptic input

Large time steps

method is
times faster

full 1
D solution

Suggested by Knight, 2000.

Dimension Reduction by Moving Eigenvector Basis

Example with

state space: state variables V & G

Out of 625
eigenvectors: 10
for high, 30 for
medium, and 60
for low synaptic
input rates.

Large time steps

method is
60 times faster

than full 1

Dimension Reduction by Moment Closure

Dimension Reduction by Moment Closure:



Stimulus Firing Rate Response

Near perfect agreement between results from dimension
reduction by moment closure, and full 2
D PDF method.

Dimension Reduction by Moment Closure:



Response to Square
Wave Modulation of Synaptic Input Rate

moment closure performs better than 2

at high input rates.


Dynamical solutions “breakdown” when synaptic input
rates drop below ~ 1240 Hz, where actual firing rate
(determined by MC and full 2
D solution) ~ 60 spikes/s.

Numerical problem or theoretical problem?

Is moment closure problem ill
posed for some
physiological parameters?

Examine the more tractable steady
state problem

Trouble with Moment Closure and Troubleshooting

State Moment Closure Problem: Existence Study

Phase Plane Analysis of Steady
State Moment Closure
Problem to Study Existence/Nonexistence of Solutions

Phase Plane and Solution at High Synaptic Input Rate

solution trajectory

must intersect

must not intersect

must intersect

must not intersect

trajectory 1

trajectory 2

State Solution Doesn’t Exist for Low Synaptic Input Rate

Promise of a New Reduced Kinetic Theory with
Wider Applicability, Using Moment Closure

A numerical
method on a
fixed voltage
grid that
introduces a
boundary layer
with numerical
diffusion finds
solutions in
agreement with

(Cai, Tao, Shelley, McLaughlin, 2004)


PDF methods show promise

Small population size OK, but connectivity cannot be dense

Realistic synaptic kinetics introduce state
space variables

Time saving benefit lost when state space dimension is high

Dimension reduction methods could maintain efficiency:

Moving eigenvector basis speeds up 2
D PDF method 60 X

Moment closure method (unmodified) has existence problems

Numerical implementations suggest moment closure can
work well

Challenge is to find methods that work for >= 3 dimensions


Bruce Knight

Charles Peskin

David McLaughlin

David Cai

Adi Rangan

Louis Tao

E. Shea

B. Doiron

Larry Sirovich

Edges of parameter space:


r from 2D PDM

5 ms

1804.11 Hz

51.35 Hz

2 ms

2377.84 Hz

74.22 Hz

1 ms

2584.88 Hz

85.82 Hz

0.5 ms

2760.47 Hz

109.16 Hz

0.2 ms

539.05 Hz

<0.01 Hz

0.1 ms

535.19 Hz

<0.01 Hz


input rate:


r from 2D PDM

5 ms

7.47 mV

61.l5 Hz

2 ms

4.89 mV

65.70 Hz

1 ms

3.40 mV

69.24 Hz

fix at mean
field threshold,

increase EPSP ( ) until solution exists

Minimal EPSP

with fixed mean G:

Parameter Values