Real-time segmentation of 3D echocardiograms, using a state estimation approach with deformable models

unclesamnorweiganΤεχνίτη Νοημοσύνη και Ρομποτική

18 Οκτ 2013 (πριν από 4 χρόνια και 20 μέρες)

92 εμφανίσεις

1

Real
-
time segmentation

of 3D echocardiograms,

using a state estimation approach

with deformable models

Fredrik Orderud

Norwegian University of Science & Technology

2

Outline


Background and motivation


Framework description


Publications

1.
Ellipsoid model (computers in cardiology, 2006)

2.
Spline model (MICCAI, 2007)

3.
Active
-
shape model (CAIP, 2007)

4.
Subdivision models (CVPR, 2008)

5.
Speckle
-
tracking (f
-
MICCAI, 2008)

6.
Edge + speckle
-
tracking (IUS, 2008)

7.
Coupled segmentation (IUS, 2008)

8.
Automatic alignment (SPIE, 2009)


Conclusions

3

Ultrasound background

Cardiac ultrasound is moving
from 2D to 3D


Latest generation of scanners
are capable of acquiring
dense image volumes in real
-
time


Important competitive
advantage to MR & CT


Image analysis is lagging
behind


Only post
-
processing tools
are currently available


We want analysis tools
robust and fast enough to
operate during imaging

4

Left ventricle


One of four heart
chambers.


Pumps oxygenated
blood from lungs to
the rest of the body.


The most clinically
interesting heart
chamber.


(c) 2006, Wikipedia

5

Ultrasound segmentation

Image segmentation:


Problem of labeling image data
with information about
boundaries, structure locations
etc.


Difficult problem, especially in
ultrasound.


Clutter, reverberation, angle
dependency and drop
-
out
disrupts images, and makes
segmentation difficult.


Example:


How to handle that part of the
cardiac wall is missing?


Approach
:


Use a
geometric model

to
assist the segmentation process

6

Common types of 3D models


Level sets


Implicit surface: Zero
-
crossing of finite diff
quantification (Malladi, PAMI 1995)


PDE
-
based update scheme to evolve surface



Discrete model


Flat polygonal surface, e.g. simplex mesh (Delingette,
IJCV 199)


Iterative force
-
based update scheme



Statistical shape model


Discrete surface, with predefined deformation modes
(Cootes, IPMI 1993)


Iterative displacement
-
based update scheme


Can be extended to also capture texture and motion
pattern
-

AAMM (Bosch, TMI 2002)



Parametric surface


Continuous surface description (FEM, spline,
subdivision etc.)


Typ. iterative gradient descent update scheme

Iterative
update
schemes

7

[1] Blake A, Curwen R, Zisserman A.
A framework for spatiotemporal

control in the tracking of visual contours
. International

Journal of Computer Vision 1993;11(2):127

145.

[2] Blake A, Isard M, Reynard D.
Learning to track the visual

motion of contours
. Artificial Intelligence 1995;78(1
-

2):179

212.

[3] Blake A, Isard M.
Active Contours
. Secaucus, NJ, USA:

Springer
-
Verlag New York, Inc., 1998. ISBN 3540762175.

Kalman segmentation background


Prof. Andrew Blake & al.,
univ. Oxford, 1990’ies


Kalman filter to update a
spline contour


Noniterative least
-
squares
fitting

8

[4] Jacob G, Noble JA, Mulet
-
Parada M, Blake A.
Evaluating

a robust contour tracker on echocardiographic sequences
.

Medical Image Analysis 1999.

[5] Jacob G, Noble JA, Kelion AD, Banning AP.
Quantitative

regional analysis of myocardial wall motion
. Ultrasound in

Medicine Biology 2001.

[6] Jacob G, Noble JA, Behrenbruch CP, Kelion AD, Banning

AP
. A shape
-
space based approach to tracking myocardial

borders and quantifying regional left ventricular function

applied in echocardiography
. IEEE Medical Imaging 2002

Prior real
-
time cardiac work

2D: G. Jacob, A. Noble (
univ
. Oxford):


ASM model, updated by a Kalman filter


Investigated relationship between shape
variation and wall thickening to pathology.


2D: Siemens corporate research:


Similar Kalman
-
filter algorithms


Subspace
-
constrained deformations


Anisotropic measurement error


3D: Q
Duan
, E.
Angelini

(Columbia
univ
.):


Cubic spline surface, updated by “level
-
set”
Mumford
-
Shah gradient descent.


33 ms processing time per frame


[7] D. Comaniciu, X.S. Zhou & al:
Robust real
-
time myocardial border
tracking for echocardiography: An information fusion approach
, IEEE
Medical imaging, 2004

[8] X.S. Zhou, A. Gupta, D. Comaniciu:
An information fusion
framework for robust shape tracking
, IEEE Pattern analysis and
machine intelligence, 2004

[9] Qi Duan, E. Angelini & al:
Real
-
time segmentation of 4D ultrasound
by Active Geometric Functions
, IEEE Intl. Symposium on Biomedical
Imaging (ISBI) 2008.

9

Kalman filter:


Invented in 1960 (R.
Kalman).


Widely used for
navigation and RADAR
tracking.


Kalman filtering has some
limitations related to
linearity and Gaussian
assumptions.


Is it out of date?


Apollo guidance computers

10

Research goals


Extend Kalman tracking
framework to 3D, and
support:


Different types of
deformable models
.


Different
image

measurement
.


Multiple simultaneous
models
, arranged in a
hierarchy.



Use the framework to
measure aspects of left
ventricular function:


Chamber volumes.


Myocardial muscle mass.


Regional myocardial
strain.


11

Real
-
time 3D Kalman tracking

framework

12

Approach


Use deformable surface
models


Described by a limited set of
parameters


Combine with global transform for
position, size and orientation


Treat tracking as a sequential
state estimation problem


Multivariate normal distribution


Use a Kalman filter to predict and
update state, based on image
measurements and a kinematic
model

Tracking driven by propagation
of uncertainty through time

13

Tracking framework

Deformable model(s)

Kinematic prediction

Volume curves

Strain curves

Kinematic prediction

Segmentations

Speckle tracking

Kalman filter

Edge detection

Next frame

Attractors

Confidence maps

Anatomic landmarks

14

Processing chain

1.
Predict


Predict state, based on a kinematic model
and previous estimates

2.
Model


Evaluate surface points, based on state
vector.


Compute Jacobian matrices

3.
Measure


Search for edges in surface normal
direction, and/or


Track speckle pattern.

4.
Assimilate


Perform outlier rejection.


Assimilate measurement in information
space.

5.
Update


Compute updated state estimate, based
on prediction and measurements.


Update surface model on screen.


State
-
vector parameters:


Global pose (trans, rot, scale).


Local shape.

Bayesian least squares
solution instead of iterative
refinement

15

Edge detection


Extract intensity “profiles” in
surface normal direction


Search for edges in profiles:


“Transition criterion”, where the
edge forms a transition from
one intensity level to another


Determine edge position that
minimizes the sum of squared
errors.

Green

-

edge discovered outside the surface

Red

-

edge detected inside the surface

Yellow

-

discarded outlier edge

Black

-

discarded too weak edge

16

Speckle tracking

Block matching:


Extract “kernel volumes” in
myocardium


Match to bigger “search
volumes” in next the frame


Search for best integer
displacement (SAD).


Sub
-
sample refinement (optical
flow).

Green

-

displacement vectors

Red

-

discarded displacement vectors


(weak or outside sector)

Yellow

-

discarded outlier displacements

17

Advantages


Flexible


Supports a range of different
parametric models.


Models can be arranged in a
hierarchy to support multi
-
model tracking


Robust


Wide range of convergence.


Enables fully automatic
initialization.


Intuitive


Parameterized by physical
quantities


Spatial uncertainty for image
measurements


No unobservable “forces” to
tune.


Efficient



4 ms/frame for segmentation
using edge
-
detection.


40 ms/frame when using
speckle
-
tracking.

10
-
100x faster than
existing methods

18

Papers

19

A Framework for Real
-
Time Left Ventricular
Tracking in 3D+T Echocardiography,

Using Nonlinear Deformable Contours and
Kalman Filter Based Tracking

F. Orderud

Proceedings of Computers in Cardiology
2006

20

Ellipsoid Model


Truncated ellipsoid



Parameters:


Translation (t
x
, t
y
, t
z
)


Rotation/orientation (r
x
, r
y
)


Scaling (s
x
, s
y
, s
z
)


“Bending” (c
x
, c
y
)

10 degrees of freedom.

21

Results

Quality

Count

Description

Good

16

Tracking performed well

Adequate

3

Tracking with reduced
accuracy

Fair

1

Tracking with low accuracy

Poor

1

Unable to track

Data:


Collection of 21 unselected 3D
echocardiography recordings


Initial contour placed at 8 cm depth in first
frame

Objective:


Not accurate segmentation


Crude approximation to LV size, position
and orientation

Subjectively scored by author

azimuth view

elevation view

initialization

After one cycle

22

Real
-
time Tracking of the Left Ventricle in 3D
Echocardiography Using a State Estimation
Approach

F. Orderud, J. Hansegård, S.I. Rabben

Proceedings of the 10th International
Conference on Medical Image Computing
and Computer Assisted Intervention (MICCAI)
2007

23

Spline Model


Using a quadratic spline
-
surface to segment the
endocardial wall


4 x 6 control points that
are allowed to move in/out
to adjust shape


Global transform for
positioning, scaling and
orientation

24

Example

25

Results

Protocol:


Tested in 21 unselected 3D
echocardiography recordings (GE
Vivid 7).


Compared to GE AutoLVQ.

Results:


Bland
-
Altman: 4.1
±
24.6 ml
agreement (95%), and a strong
correlation (r = 0.92).


2.7 mm average point to surface
distance against reference.

26

Real
-
Time Active Shape Models for
Segmentation of 3D Cardiac Ultrasound

J. Hansegård, F. Orderud, S.I. Rabben

Proceedings of the 12th International
Conference on Computer Analysis of Images
and Patterns
-

CAIP 2007

27

Deformable model

3D active shape model (ASM)

Linear model consisting of:


Average shape


Deformation modes
A
i


Built by PCA on training set (N=31)
segmented with AutoLVQ (J.
Hansegård).



20 deformation modes explains
98% of variation in training set


Shape controlled by state
x
l


Project deformations to normal
direction
n
i

to reduce
computational cost.

+

+

+

28

Initialization example

29

Results

Good agreement in volumes
and EF.

2.2
±
1.1 mm point
-
to
-
surface
error.


Discussion:


Few parameters to estimate


Regularizes segmentation to
physiological realistic shapes


30

GE GRC model (unpublished)

31

Real
-
time 3D Segmentation of the Left Ventricle
Using Deformable Subdivision Surfaces

F. Orderud, S.I. Rabben

Proceedings in the IEEE Computer Society
Conference on Computer Vision and Pattern
Recognition (CVPR) 2008

32

Subdivision surfaces


Model the left ventricle with a
subdivision surface


Extension of spline surfaces to
arbitrary topology


Parameterized by a mesh of control
points


Surface defined as the limit of recursive
refinement process (implicit)


Smooth surface with compact
description


Intuitive, easy to model



Utilized method of J. Stam to
evaluate arbitrary surface points
without recursive subdivision.


J. Stam:
Exact Evaluation of Catmull
-
Clark Subdivision
Surfaces at Arbitrary Parameter Values
, SIGGRAPH'98

33

Example


5ms processing time/frame

34

Results


Better agreement
compared to spline/ASM.


Discussion:


Very popular in computer
graphics

35

Real
-
time Left Ventricular Speckle
-
Tracking in 3D Echocardiography With
Deformable Subdivision Surfaces

F. Orderud, G. Kiss, S. Langeland, E.
Remme, H. Torp, S.I. Rabben

Proceedings of the MICCAI workshop on
Analysis of Medical Images 2008

36

Speckle tracking


Idea


Edge
-
detection to initialize


3D speckle
-
tracking as
measurement



Block matching:


Integer displacement estimation
with sum of absolute differences
(SAD)


Sub
-
sample correction with Lucas
-
Kanade optical flow


37

Simulations


Finite
-
element (FEM)
simulation of a left
ventricle deformation (E.
Remme)


“K
-
space” ultrasound
simulator (S. Langeland)

Ellipsoid simulation

Dog
-
heart simulation

infarction

E. Remme, O. Smiseth:
Characteristic Strain Pattern of Moderately Ischemic
Myocardium Investigated in a Finite Element Simulation Model
, Functional
Imaging and Modeling of the Heart, 2007, 330
-
339

T. Hergum, J. Crosby, M. Langhammer, H. Torp:
The Effect of Including Fiber
Orientation in Simulated 3D Ultrasound Images of the Heart
, IEEE Ultrasonics
Symposium, 2006, 1991
-
1994

infarction

38

Simulation results


Able to identify infarcted
region in both simulations


Absolute strain values are
underestimated

Sim. B:

Sim. A:

Estimated

Ground truth

End systolic area strain:

39

In
-
vivo results


Tested in 21 unselected in
-
vivo recordings


50% with cardiac disease


No ground truth available



In
-
vivo challenges:


Lower image quality, especially
in the near
-
field


Lower spatial & temporal
resolution than 2D recordings

Absolute drift

Relative drift

Simulation drift

0.58 +/
-

0.70mm

8.58 +/
-

10.59%

In
-
vivo drift

2.7 +/
-

1.0mm

12.08 +/
-

2.09%

95% conf. intervals (mean +/
-

1.96std)

Drift after tracking in one cycle:

37ms processing
time per frame
(decimated data)

40

Combining Edge Detection With Speckle
-
Tracking for Cardiac Strain Assessment in
3D echocardiography

F. Orderud, G. Kiss, S. Langeland, E.
Remme, H. Torp, S.I. Rabben

Proceedings of the IEEE Ultrasonics
symposium (IUS) 2008

41

3D strain


Approach:


Combine edge
-
detection with
speckle
-
tracking.


Edge
-
detection to align to
myocardium


Speckle
-
tracking for material
deformations


Advantages
:


Only track residual deformation,
after shape changes are
corrected for


Smaller search
-
windows


Less drift

42

Results

Combined edge + tracking
yields:


Improved tracking
accuracy


Improved processing time

ES strain

ES displacement vectors

ES displacement errors

Speckle
-
tracking

Speckle
+ edge

Processing time:

Edge:


130ms

Edge+speckle:

68ms

(non
-
decimated data)

43

Automatic coupled segmentation of endo
-

and
epicardial borders in 3D echocardiography

F. Orderud, G. Kiss, H. Torp

Proceedings of the IEEE Ultrasonics
symposium (IUS) 2008

44

LV mass


Simultaneous
segmentation:


Endocard LV model


Epicard LV models


Models share position, size
and orientation.



Applications:


Myocardial mass/muscle
volume


Wall thickening analysis

10ms processing time/frame

45

LV mass


Dataset: 5 recordings of
high image quality.


Reference: Endo+epi
segmentation with
AutoLVQ at ED and ES


Results:


Good correspondence
(<20ml difference) in 4 out
of 5 recordings.

46

Automatic Alignment of Standard Views in 3D
Echocardiograms Using Real
-
time Tracking

F. Orderud, H. Torp, S.I. Rabben

Proceedings of the SPIE Medical Imaging
conference 2009

47

Apical View Alignment


Combine LV model, with
“RV sail” and “LVOT
cylinder”


Extract:


Long
-
axis and rotation
from landmarks on
models


Generate 4CH, 2CH &
LAX views (assume fixed
angle between slices)


4CH

2CH

LAX

SAX

48

Short
-
axis Alignment


Dynamic long
-
axis vector
from LV


Equidistant short
-
axis slices
that compensate for out
-
of
-
plane motion


in atrium

Basal SAX slice example:

MV

49

Results

Experiment:


Dataset of 35 recordings (>70%
of myocardium visible).


Compared to manually
annotated landmarks from GE
AutoLVQ.

Mean +/
-

std. absolute position error for
apex and mitral valve points

Mean +/
-

std. absolute angle error around
long
-
axis for A4C, A2C and LAX views

Discussion:


Use image analysis to improve
visualization.


Does not depend on 100%
accurate segmentation.

50

Conclusions

51

Conclusions


Extend Kalman
-
tracking
framework to 3D


Extend the framework
to support:


Different types of
deformable models

(ellipsoid, spline, active
-
shape, subdivision)


Different
image

measurement

(edge,
speckle
-
tracking)


Multiple simultaneous
models

arranged in a
hierarchy (thick
-
wall,
alignment)


Demonstrated
feasibility

of:


LV volume (in
-
vivo).


LV mass (in
-
vivo)


LV strain (simulations)


Automatic view alignment
(in
-
vivo)


Conducted in/close to real
-
time in 3D echocardiograms.


52

Other applications

Bladder segmentation

Cardiac CT LV+RV segmentation


Technology is general,
and not limited to
either ultrasound or
cardiac imaging.

Simultaneous LV + LA segmentation

53

Future extensions


Developments for 2D echo


Auto segmentation for hand
-
held scanner


Scanning feedback


Image quality scoring.


Probe positioning feedback.


Auto
-
positioning of color
-
flow
areas


Algorithmic developments


Simultaneous edge
-
detection
in all profiles


Multi
-
resolution segmentation
(coarse to fine)


Bidirectional tracking


Distributed filtering

54

Thank you

55

Acknowledgements


Dep. computer science
(NTNU) for financing my
PhD program


PhD Stein Inge Rabben
(GE), for supervising my
work related to model
-
based
segmentation.


Prof. Hans Torp (NTNU), for
supervising my work related
to cardiac ultrasound and
speckle
-
tracking


+ many more...