Try to save computation time in large- scale neural network modeling with population density methods, or just fuhgeddaboudit

chickenchairwomanΤεχνίτη Νοημοσύνη και Ρομποτική

19 Οκτ 2013 (πριν από 3 χρόνια και 9 μήνες)

79 εμφανίσεις

Try to save computation time in large
-
scale neural network

modeling with population density
methods, or just fuhgeddaboudit
?

Daniel Tranchina, Felix Apfaltrer & Cheng Ly

New York University

Courant Institute of Mathematical Sciences

Department of Biology

Center for Neural Science


Supported by NSF grant BNS0090159

SUMMARY


Can the population density function (PDF) method be made
into a practical time
-
saving computational tool in large
-
scale
neural network modeling?



Motivation for thinking about PDF methods


General theoretical and practical issues



The dimension problem in realistic single
-
neuron models


Two dimension reduction methods

1)
Moving eigenfunction basis (Knigh, 2000)


only good news

2)
Moment closure method (Cai et al., 2004)


good news; bad news; worse news; good news


Example of a model neuron with a 2
-
D state space



Future directions

Why Consider PDF Methods in Network Modeling?



Synaptic noise makes neurons behave stochastically: synaptic
failure; random sizes of unitary events; random synaptic delay.


Important physiological role: mechanism for modulation of
gain/kinetics of population responses to synaptic input;
prevents synchrony in spiking neuron models as in the brain.


Important to capture somehow the properties of noise in
realistic models.


Large number of neurons required for modeling physiological
phenomena of interest, e.g. working memory; orientation
tuning in primary visual cortex.




Number of neurons is determined by the
functional
subunit
; e.g. an orientation hypercolumn in V1:


TYPICAL MODELS: ~ (1000 neurons)/(orientation
hypercolumn) for input layer V1 ( 0.5 X 0.5 mm
2

or roughly
0.25 X 0.25 deg
2
).


REALITY: ~
34,000 neurons
,
75 million synapses


Many hypercolumns are required to study some problems,
e.g. dependence of spatial integration area on stimulus
contrast.

Why Consider PDF Methods
(continued)

Why PDF?

(continued)



Tracking by computation the activity of ~10
3
--
10
4

neurons and
~10
4
--
10
6

synapses taxes computational resources: time and
memory.



e.g. 8 X 8 hypercolumn
-
model for V1 with 64,000 neurons


(Jim Wielaard and collaborators, Columbia):

1 day to simulate 4 seconds real time


But stunning recent progress by Adi Rangan & David Cai

What to do?

Quest for the
Holy Grail
: a low
-
dimensional system of
equations that approximates the behavior of a truly high
-
dimensional system.

Firing rate

model
(Dyan & Abbott, 2001): system of ODEs or

PDF

model

(system of PDEs or integro
-
PDEs)?

The PDF Approach


Large number of interacting stochastic units suggests a
statistical mechanical approach.


Lump similar neurons into discrete groups.


Example: V1 hypercolumn. Course graining over: position,
orientation preference; receptive
-
field structure (spatial
-
phase); simple
--
complex; E vs. I may give ~ 50
neurons/population (~ tens OK for PDF methods).


Each neuron has a set of dynamical variables that
determines its state; e.g.

for a leaky I&F neuron.


Track the distribution of neurons over state space and
firing rate for each population.

Rich History of PDF Methods in Computational
Neuroscience


Wilbur & Rinzel 1983



Kuramoto 1991



Abbott & van Vreeswijk 1993



Gerstner 1995



Knight et al., 1996



Omurtage et al., 2000



Sirovich et al., 2000



Casti et al. 2002



Cai et al., 2004



Huertas & Smith, 2006

PDF Methods Recently Espoused and Tested as a Faster
Alternative to Monte Carlo Simulations

PDF Theory


Most applications of PDF methods as a computational tool
have involved single
-
neuron models with a 1
-
D state space:
instantaneous synaptic kinetics; V jumps abruptly up/down
with each unitary excitatory/inhibitory synaptic input event.


Synaptic kinetics play an enormously important role in
determining neural network dynamics.


Bite the bullet and include realistic synaptic kinetics.


Problem with PDF methods: as underlying neuron model is
made more realistic, dimension of the state space increases,
so does the computation time to solve the PDF equations.


Time saving advantage of PDF over (direct) MC vanishes.


Minimal I&F model with synaptic kinetics has 3 state variables:

voltage, excitatory and inhibitory conductances.

Minimal I&F Model: How Many State Variables?

Take Baby Steps: Introduce Dimensions One at a
Time and See What We Can Do

v

nullcline

PDF vs. MC and Mean
-
Field for 2
-
D Problem.

PDF cpu time is ~ 400 single uncoupled neurons

cpu time: 0.8 s for PDF; 2 s per 1000 neurons for MC.

1000 neurons

100,000 neurons

PDF

mean
-
field

MC

Computation Time Comparison: PDF vs. Monte Carlo (MC):

PDF grows linearly; MC grows quadratically

50 neurons per population;1 run; 25% connectivity


PDF Method is plenty fast for model neurons
with a 2
-
D state space.


More realistic models (e.g. with E and I
input) require additional state variables


Explore dimension reduction methods.


Use the 2
-
D problem as a test problem

Dimension Reduction by Moving Eigenvector Basis:

Bowdlerization of Bruce Knight’s (2000) Idea

Dimension Reduction by Moving Eigenvector Basis

Example with
1
-
D

state space, instantaneous synaptic kinetics


Only 3 eigenvectors
for low, and 7 for
high synaptic input
rates.


Large time steps


Eigen
-
method is
60
times faster

than
full 1
-
D solution

Suggested by Knight, 2000.

Dimension Reduction by Moving Eigenvector Basis

Example with
2
-
D

state space: state variables V & G
e


Out of 625
eigenvectors: 10
for high, 30 for
medium, and 60
for low synaptic
input rates.


Large time steps


Eigen
-
method is
60 times faster

than full 1
-
D
solution

Dimension Reduction by Moment Closure

Dimension Reduction by Moment Closure:

2
nd

Moment


Stimulus Firing Rate Response

Near perfect agreement between results from dimension
reduction by moment closure, and full 2
-
D PDF method.

Dimension Reduction by Moment Closure:

3
rd

Moment

Response to Square
-
Wave Modulation of Synaptic Input Rate

3
rd
-
moment closure performs better than 2
nd

at high input rates.

ZOOM


Dynamical solutions “breakdown” when synaptic input
rates drop below ~ 1240 Hz, where actual firing rate
(determined by MC and full 2
-
D solution) ~ 60 spikes/s.


Numerical problem or theoretical problem?


Is moment closure problem ill
-
posed for some
physiological parameters?


Examine the more tractable steady
-
state problem

Trouble with Moment Closure and Troubleshooting

Steady
-
State Moment Closure Problem: Existence Study

Phase Plane Analysis of Steady
-
State Moment Closure
Problem to Study Existence/Nonexistence of Solutions

Phase Plane and Solution at High Synaptic Input Rate

solution trajectory

must intersect

must not intersect

must intersect

must not intersect

trajectory 1

trajectory 2

Steady
-
State Solution Doesn’t Exist for Low Synaptic Input Rate

Promise of a New Reduced Kinetic Theory with
Wider Applicability, Using Moment Closure

A numerical
method on a
fixed voltage
grid that
introduces a
boundary layer
with numerical
diffusion finds
solutions in
good
agreement with
direct
simulations.

(Cai, Tao, Shelley, McLaughlin, 2004)

SUMMARY


PDF methods show promise


Small population size OK, but connectivity cannot be dense


Realistic synaptic kinetics introduce state
-
space variables


Time saving benefit lost when state space dimension is high


Dimension reduction methods could maintain efficiency:



Moving eigenvector basis speeds up 2
-
D PDF method 60 X


Moment closure method (unmodified) has existence problems


Numerical implementations suggest moment closure can
work well


Challenge is to find methods that work for >= 3 dimensions

THANKS



Bruce Knight



Charles Peskin



David McLaughlin



David Cai



Adi Rangan



Louis Tao



E. Shea
-
Brown



B. Doiron



Larry Sirovich

Edges of parameter space:


Minimal

r from 2D PDM

5 ms

1804.11 Hz

51.35 Hz

2 ms

2377.84 Hz

74.22 Hz

1 ms

2584.88 Hz

85.82 Hz

0.5 ms

2760.47 Hz

109.16 Hz

0.2 ms

539.05 Hz

<0.01 Hz

0.1 ms

535.19 Hz

<0.01 Hz

Minimal

input rate:

Min EPSP

r from 2D PDM

5 ms

7.47 mV

61.l5 Hz

2 ms

4.89 mV

65.70 Hz

1 ms

3.40 mV

69.24 Hz

fix at mean
-
field threshold,

increase EPSP ( ) until solution exists

Minimal EPSP

with fixed mean G:

Parameter Values