Try to save computation time in large

scale neural network
modeling with population density
methods, or just fuhgeddaboudit
?
Daniel Tranchina, Felix Apfaltrer & Cheng Ly
New York University
Courant Institute of Mathematical Sciences
Department of Biology
Center for Neural Science
Supported by NSF grant BNS0090159
SUMMARY
•
Can the population density function (PDF) method be made
into a practical time

saving computational tool in large

scale
neural network modeling?
•
Motivation for thinking about PDF methods
•
General theoretical and practical issues
•
The dimension problem in realistic single

neuron models
•
Two dimension reduction methods
1)
Moving eigenfunction basis (Knigh, 2000)
only good news
2)
Moment closure method (Cai et al., 2004)
good news; bad news; worse news; good news
•
Example of a model neuron with a 2

D state space
•
Future directions
Why Consider PDF Methods in Network Modeling?
•
Synaptic noise makes neurons behave stochastically: synaptic
failure; random sizes of unitary events; random synaptic delay.
•
Important physiological role: mechanism for modulation of
gain/kinetics of population responses to synaptic input;
prevents synchrony in spiking neuron models as in the brain.
•
Important to capture somehow the properties of noise in
realistic models.
•
Large number of neurons required for modeling physiological
phenomena of interest, e.g. working memory; orientation
tuning in primary visual cortex.
•
Number of neurons is determined by the
functional
subunit
; e.g. an orientation hypercolumn in V1:
•
TYPICAL MODELS: ~ (1000 neurons)/(orientation
hypercolumn) for input layer V1 ( 0.5 X 0.5 mm
2
or roughly
0.25 X 0.25 deg
2
).
•
REALITY: ~
34,000 neurons
,
75 million synapses
•
Many hypercolumns are required to study some problems,
e.g. dependence of spatial integration area on stimulus
contrast.
Why Consider PDF Methods
(continued)
Why PDF?
(continued)
•
Tracking by computation the activity of ~10
3

10
4
neurons and
~10
4

10
6
synapses taxes computational resources: time and
memory.
e.g. 8 X 8 hypercolumn

model for V1 with 64,000 neurons
(Jim Wielaard and collaborators, Columbia):
1 day to simulate 4 seconds real time
•
But stunning recent progress by Adi Rangan & David Cai
What to do?
Quest for the
Holy Grail
: a low

dimensional system of
equations that approximates the behavior of a truly high

dimensional system.
Firing rate
model
(Dyan & Abbott, 2001): system of ODEs or
PDF
model
(system of PDEs or integro

PDEs)?
The PDF Approach
•
Large number of interacting stochastic units suggests a
statistical mechanical approach.
•
Lump similar neurons into discrete groups.
•
Example: V1 hypercolumn. Course graining over: position,
orientation preference; receptive

field structure (spatial

phase); simple

complex; E vs. I may give ~ 50
neurons/population (~ tens OK for PDF methods).
•
Each neuron has a set of dynamical variables that
determines its state; e.g.
for a leaky I&F neuron.
•
Track the distribution of neurons over state space and
firing rate for each population.
Rich History of PDF Methods in Computational
Neuroscience
•
Wilbur & Rinzel 1983
•
Kuramoto 1991
•
Abbott & van Vreeswijk 1993
•
Gerstner 1995
•
Knight et al., 1996
•
Omurtage et al., 2000
•
Sirovich et al., 2000
•
Casti et al. 2002
•
Cai et al., 2004
•
Huertas & Smith, 2006
PDF Methods Recently Espoused and Tested as a Faster
Alternative to Monte Carlo Simulations
PDF Theory
•
Most applications of PDF methods as a computational tool
have involved single

neuron models with a 1

D state space:
instantaneous synaptic kinetics; V jumps abruptly up/down
with each unitary excitatory/inhibitory synaptic input event.
•
Synaptic kinetics play an enormously important role in
determining neural network dynamics.
•
Bite the bullet and include realistic synaptic kinetics.
•
Problem with PDF methods: as underlying neuron model is
made more realistic, dimension of the state space increases,
so does the computation time to solve the PDF equations.
•
Time saving advantage of PDF over (direct) MC vanishes.
•
Minimal I&F model with synaptic kinetics has 3 state variables:
voltage, excitatory and inhibitory conductances.
Minimal I&F Model: How Many State Variables?
Take Baby Steps: Introduce Dimensions One at a
Time and See What We Can Do
v
nullcline
PDF vs. MC and Mean

Field for 2

D Problem.
PDF cpu time is ~ 400 single uncoupled neurons
cpu time: 0.8 s for PDF; 2 s per 1000 neurons for MC.
1000 neurons
100,000 neurons
PDF
mean

field
MC
Computation Time Comparison: PDF vs. Monte Carlo (MC):
PDF grows linearly; MC grows quadratically
50 neurons per population;1 run; 25% connectivity
•
PDF Method is plenty fast for model neurons
with a 2

D state space.
•
More realistic models (e.g. with E and I
input) require additional state variables
•
Explore dimension reduction methods.
•
Use the 2

D problem as a test problem
Dimension Reduction by Moving Eigenvector Basis:
Bowdlerization of Bruce Knight’s (2000) Idea
Dimension Reduction by Moving Eigenvector Basis
Example with
1

D
state space, instantaneous synaptic kinetics
•
Only 3 eigenvectors
for low, and 7 for
high synaptic input
rates.
•
Large time steps
•
Eigen

method is
60
times faster
than
full 1

D solution
Suggested by Knight, 2000.
Dimension Reduction by Moving Eigenvector Basis
Example with
2

D
state space: state variables V & G
e
•
Out of 625
eigenvectors: 10
for high, 30 for
medium, and 60
for low synaptic
input rates.
•
Large time steps
•
Eigen

method is
60 times faster
than full 1

D
solution
Dimension Reduction by Moment Closure
Dimension Reduction by Moment Closure:
2
nd
Moment
Stimulus Firing Rate Response
Near perfect agreement between results from dimension
reduction by moment closure, and full 2

D PDF method.
Dimension Reduction by Moment Closure:
3
rd
Moment
Response to Square

Wave Modulation of Synaptic Input Rate
3
rd

moment closure performs better than 2
nd
at high input rates.
ZOOM
•
Dynamical solutions “breakdown” when synaptic input
rates drop below ~ 1240 Hz, where actual firing rate
(determined by MC and full 2

D solution) ~ 60 spikes/s.
•
Numerical problem or theoretical problem?
•
Is moment closure problem ill

posed for some
physiological parameters?
•
Examine the more tractable steady

state problem
Trouble with Moment Closure and Troubleshooting
Steady

State Moment Closure Problem: Existence Study
Phase Plane Analysis of Steady

State Moment Closure
Problem to Study Existence/Nonexistence of Solutions
Phase Plane and Solution at High Synaptic Input Rate
solution trajectory
must intersect
must not intersect
must intersect
must not intersect
trajectory 1
trajectory 2
Steady

State Solution Doesn’t Exist for Low Synaptic Input Rate
Promise of a New Reduced Kinetic Theory with
Wider Applicability, Using Moment Closure
A numerical
method on a
fixed voltage
grid that
introduces a
boundary layer
with numerical
diffusion finds
solutions in
good
agreement with
direct
simulations.
(Cai, Tao, Shelley, McLaughlin, 2004)
SUMMARY
•
PDF methods show promise
•
Small population size OK, but connectivity cannot be dense
•
Realistic synaptic kinetics introduce state

space variables
•
Time saving benefit lost when state space dimension is high
•
Dimension reduction methods could maintain efficiency:
•
Moving eigenvector basis speeds up 2

D PDF method 60 X
•
Moment closure method (unmodified) has existence problems
•
Numerical implementations suggest moment closure can
work well
•
Challenge is to find methods that work for >= 3 dimensions
THANKS
•
Bruce Knight
•
Charles Peskin
•
David McLaughlin
•
David Cai
•
Adi Rangan
•
Louis Tao
•
E. Shea

Brown
•
B. Doiron
•
Larry Sirovich
Edges of parameter space:
Minimal
r from 2D PDM
5 ms
1804.11 Hz
51.35 Hz
2 ms
2377.84 Hz
74.22 Hz
1 ms
2584.88 Hz
85.82 Hz
0.5 ms
2760.47 Hz
109.16 Hz
0.2 ms
539.05 Hz
<0.01 Hz
0.1 ms
535.19 Hz
<0.01 Hz
Minimal
input rate:
Min EPSP
r from 2D PDM
5 ms
7.47 mV
61.l5 Hz
2 ms
4.89 mV
65.70 Hz
1 ms
3.40 mV
69.24 Hz
fix at mean

field threshold,
increase EPSP ( ) until solution exists
Minimal EPSP
with fixed mean G:
Parameter Values
Comments 0
Log in to post a comment