Advanced beam simulations in AFRD
J.

L.
Vay
Lawrence Berkeley National Laboratory
May 2
, 2012
1
Office of Science
SciDAC

II
Compass
2
Outline
AFRD beam simulation tools

Overview

Strength in innovative methods

Effort toward integration
Selected sample of recent
applications

BELLA, NGLS, NDCX

II, E

cloud SPS, multilevel
parallelism,
ping

pong
modes
Summary
AFRD Review
–
May 2

3, 2012

Vay
3
AFRD develops and supports several important physics codes
Code
Developers
Type
Description
BeamBeam3D
(F95)
J.
Qiang
,
R.
Ryne
3D parallel ES PIC
modeling strong

strong or strong

weak beam

beam
interactions in high energy colliders
CSR3D
(F95)
R.
Ryne
3D
Lienard

Wiechert
non

self

consistent parallel code for CSR studies
Ginger
(F95)
W.
Fawley
et al
2D

RZ and 3D PIC
polychromatic FEL simulation code
Impact
(F95)
J.
Qiang
,
R.
Ryne
3D parallel ES PIC w/
maps
framework for modeling high intensity, high brightness
beams in accelerators
INF&RNO
(C++)
C. Benedetti
2D

RZ parallel EM
PIC/fluid
lab/boosted frame w/
ponderomotive
approximation +
envelope model for the laser
Marylie

Impact
(F95)
R.
Ryne
, J.
Qiang
et
al.
3D parallel
ES PIC
w/ maps
Combination of
MaryLie
and IMPACT including both high
order optics based on Lie algebraic maps along with parallel
3D space

charge effects
POSINST
(F95)
M. Furman
et al.
2D
ES PIC
electron cloud buildup studies
Warp
(Python+F95)
D.
Grote, J

L Vay,
A.Friedman et al.
2D/3D parallel
ES & EM
PIC
framework with PIC & accelerator lattice for modeling of
particle beam generation, transport
& neutralization, or
laser

plasma interaction
Warp

POSINST
(Python+F95)
J

L Vay, M. Furman,
D. Grote
2D/3D parallel ES PIC w/
maps
Combination of Warp and POSINST for self

consistent
electron cloud studies
PIC=Particle

In

Cell: Lagrangian macro
particles + fields on grids (finite difference solvers);
interpolation between particles and
fields (ES=electrostatic; EM=electromagnetic).
AFRD Review
–
May 2

3, 2012

Vay
Many
users
at
institutions
worldwide
4
CERN (P,W)
Diamond
(I)
ESS (I)
Fermi/
Elettra
(I)
Frankfurt
(I
)
GSI
(I
,W)
Hiroshima U. (W)
Hong Kong U. (W)
IBS (I)
ANL (B,I,P)
BNL (B,I,P,W)
Cornell
(I
,P)
FNAL (B,I,P,W)
ISU (I)
Jlab
(B,
I,P)
LANL
(I
,P)
LBNL (B,I,P,W)
LLNL (W)
MSU
(I
,W
)
NIU
(I)
ORNL (I
,P)
SLAC
(I
,P,W)
Stanford
(I)
Tech

X (P)
UM (W)
UMD (W)
UW
(I
)
UCLA
(I
)
WSU (W
)
Yale U (B,I)
United States
Europe/Asia
B
I
P
W
–
BeamBeam3D
–
Impact
–
Posinst
–
Warp
IHEP (I
,P)
IMPCAS
(I
)
KEK
(I
)
PAL
(I
)
PSI
(I)
RRCAT (I)
RAL (I)
SINAP
(I
)
Technion (W)
AFRD Review
–
May 2

3, 2012

Vay
AFRD tools are applied to the design, optimization, risk
minimization and support of particle accelerators
•
Application to
–
Linacs
, transfer lines, rings, colliders, injectors, particle traps (e.g. anti

H) and l
aser plasma
accelerators (LPA)
–
Hadron machines, lepton machines, multi

charge state beams
•
w
ith
i
mpacts spanning
the breadth of DOE/SC
–
HEP
:
Tevatron
, Main Injector, NML
photoinjector
, LHC, SPS, LHC injector upgrades,
ILC,
Bella, CESR

TA, Project

X
–
NP
: FRIB, e


ion colliders
–
BES
: SNS, LCLS, NGLS
–
FES
: Ion beam dynamics for Heavy Ion Fusion and
HEDP (NDCX

II)
•
partially funded by SciDAC/ComPASS, collaborations with ASCR
–
also partially funded by LDRD and SBIR partnership in the past
5
AFRD Review
–
May 2

3, 2012

Vay
AFRD code developers and
users
J. Barnard
3
*
C.
Benedetti
4
S.
Bulanov
4
M.
Chen
4
R.
Cohen
3
*
A
.
Friedman
3
*
M.
Furman
2
C.
Geddes
4
D.
Grote
3
*
E.
Henestroza
3
Q.
Ji
3
S.
Lund
3
*
H. Nishimura
1
A. Persaud
3
6
C. Papadopoulos
1,
2
S.
Paret
2
G.
Penn
2
J.
Qiang
2
M.
Reinsch
2
S.
Rykovanov
4
R.
Ryne
2
W.
Sharp
3
*
C. Sun
1
M.
Terry
3
*
J.

L.
Vay
4,
2
,
3
M.
Venturini
2
W. Wan
1
L.
Yu
4
28 total (21+7
guests)
1
ALS (4)
2
CBP (7+)
3
HIF/
IBT (10+)
4
LOASIS (6+)
*
guest (LLNL
)
AFRD Review
–
May 2

3, 2012

Vay
Fruitful collaborations between
AFRD & CRD/NERSC
Recent & present AFRD

CRD/NERSC collaborations
–
E. Wes Bethel et al (VACET): beam path analysis for LPA

O.
Rubel
, C. G. R. Geddes, E. Cormier

Michel, K. Wu,
Prabhat
, G. H. Weber, D. M.
Ushizima
, P.
Messmer
, H.
Hagen, B.
Hamann
, E. W. Bethel,
Automatic beam path analysis of laser
wakeﬁeld
particle acceleration data
,
Computational Science & Discovery, vol. 2, 015005 (2009)
–
ExaHDF5 team: parallel I/O, analysis,
visualization

Chou, Wu,
Rubel
,
Howison
,
Qiang
,
Prabhat
, Austin, Bethel,
Ryne
,
Shoshani
,
Parallel Index and Query for Large
Scale Data Analysis
, to appear in
SuperComputing
2011.
–
H. Shan and X.
Li: parallel performance optimization
–
P
.
Colella
: LOASIS AMR modeling of capillary
–
A. Koniges: B. Austin (parallel optimization), B. Liu (ALE

AMR)
Proximity of CRD & NERSC is a great asset.
AFRD Review
–
May 2

3, 2012

Vay
8
Outline
AFRD beam simulation tools

Overview

Strength in innovative methods

Effort toward integration
Selected sample of recent
applications

BELLA, NGLS, NDCX

II, E

cloud SPS, multilevel
parallelism,
ping

pong
modes
Summary
AFRD Review
–
May 2

3, 2012

Vay
Many algorithms
invented, improved or
pioneered in AFRD codes
9
Algorithm/method
Reference
Originated
Adopted
by
Damped
EM & particle pushers
Friedman, JCP 1990
Warp
LSP,Elixirs
Warped coordinates
PIC in bends
Friedman
et al, Phys. Fluid 1992
Warp
Integrated Maps for
rf
cavity dynamics
Ryne
, LANL Report 1995
ML/I
D.
Abell
(nonlinear
model)
Stochastic Leap

Frog
for Brownian motion
Qiang
&
Habib
, PRE 2000
Impact
Spectral

finite difference
multigrid
solver
Qiang
&
Ryne
, CPC 2001
Impact
Improved Perfectly
Matched Layers
Vay, JCP 2000
/JCP 2002
Warp
Osiris
AMR

PIC electrostatic
Vay et al,
LPB2002/PoP2004
Warp
Secondary emission of
electrons algorithm
Furman &
Pivi
, PRST

AB 2003
Posinst
TxPhysics
AMR

PIC electromagnetic
Vay et al,
CPC 2004
Emi2D
Warp
3D Poisson solver with large aspect ratio
Qiang
&
Gluckstern
,
CPC 2004
Impact
Shift

Green
f
unction method
Qiang
et al, CPC 2004
BBeam3D
Integrated
Green function
Ryne
&
Qiang
ML/I
BB3D,Impact
Hybrid Lorentz
particle pusher
Cohen et al, NIMA 2007
Warp
AFRD Review
–
May 2

3, 2012

Vay
and adopted by other codes
10
Algorithm/method (cont.)
Reference
Originated
Adopted
by
Lorentz boosted frame*
Vay, PRL 2007
Warp
Inf&rno,JPIC
,
Osiris,Vorpal
,VPIC
Explicit Lorentz invariant particle pusher
Vay,
PoP
2008
Warp
Tristan (
astro
), QED
New convolution integral w/ smooth kernel
Qiang
, CPC 2010
N/A
Mixed
Particle

Field decomposition method
Qiang & Li, CPC 2010
BBeam3D
Improved laser envelope
model for LPA
Cowan,
Esarey et al, JCP 2011
Vorpal
PIC with tunable
electromagnetic solver
Vay et al, JCP 2011
Warp
Vorpal,Osiris
Efficient digital filter for PIC
Vay et al, JCP 2011
Warp
Vorpal,Osiris
Laser launcher
from moving antenna
Vay et al,
PoP
2011
Warp
Vorpal,Osiris
High precision laser envelope
model
Benedetti et al, 2011
Inf&rno
*
Phys. Rev. Lett.
98
(2007)
Example
Lorentz boosted frame method and associated numerical techniques have been
adopted by others:
•
have helped Tech

X implementing moving antenna algorithm in
Vorpal
,
•
UCLA has requested help for implementation in Osiris.
AFRD Review
–
May 2

3, 2012

Vay
11
Outline
AFRD beam simulation tools

Overview

Strength in innovative methods

Effort toward integration
Selected sample of recent
applications

BELLA, NGLS, NDCX

II, E

cloud SPS, multilevel
parallelism,
ping

pong
modes
Summary
AFRD Review
–
May 2

3, 2012

Vay
RB1
RB2
…
RA1
RA2
…
In the past, codes developed mostly separately
12
RE1
RE2
…
Developers
Codes
J.
Qiang
, R.
Ryne
BeamBeam3D
I
mpact
C. Benedetti
INF&RNO
M. Furman
POSINST
RC1
RC2
…
RD1
RD2
…
D. Grote, J

L Vay
Warp
CBP
Fusion
CBP
LOASIS
AFRD Review
–
May 2

3, 2012

Vay
RB1
RB2
…
RA1
RA2
…
Recently, Warp & Posinst were bridged and Warp expanded
its reach within the division
13
RE1
RE2
…
Developers
Codes
J.
Qiang
, R.
Ryne
BeamBeam3D
I
mpact
C. Benedetti
INF&RNO
M. Furman
POSINST
RC1
RC2
…
RD1
RD2
…
D. Grote, J

L Vay
Warp
Warp

POSINST
CBP
Fusion/IBT,CBP,LOASIS
CBP
LOASIS
AFRD Review
–
May 2

3, 2012

Vay
RB1
RB2
…
RA1
RA2
…
Toward further integration

some AFRD codes ported in common repository
14
Field
solver A
Particle
pusher A
A
B
C
CRD
Field
solver B
AFRD Common Repository
1
Particle pusher C
Other
1
3
4
Developers
Codes
2
Together with increased modularity, this will provide opportunities for:

co

development within AFRD,

collaboration with CRD (easier & more productive on isolated modules than on full codes).
CRD Repository
Field solver C
Other
2
CRD=Computational Research Division
AFRD Review
–
May 2

3, 2012

Vay
15
•
No black box: 100% knowledge of numerical methods and implementation

g
reatly reduces uncertainties in understanding simulation results,

thus greatly enhance chances of agreement, or understanding of reason for
disagreement, between simulations and experiments,

allows fast development of specialized or improved
algorithms.
•
C
ontrol of priorities and pace of needed capabilities.
•
S
imulation codes are increasingly becoming critical strategic assets

bigger, faster computers and improved numerical methods increase fidelity,

the embodied physical models are constantly improved.
A
ccess to the best simulation codes allows for faster design, commissioning
and understanding of experiments.
Why we are developing codes within AFRD
AFRD Review
–
May 2

3, 2012

Vay
16
Outline
AFRD beam simulation tools

Overview

Strength in innovative methods

Effort toward integration
Selected sample of recent
applications

BELLA, NGLS, NDCX

II, E

cloud SPS
, multilevel
parallelism
,
ping

pong
modes
Summary
AFRD Review
–
May 2

3, 2012

Vay
Wide array
of applications
17
0.04
0.02
0.00
0.02
0.04
y
[
m
]
0.04
0.02
0.00
0.02
0.04
x [m]
CESRTA, By=0.22 T
Nb=7e10, Eb=5.3 GeV
tb=14 ns, dmax=1.3
2.0x10
13
1.5
1.0
0.5
0.0
m
*
*

3
HEDP/HIF driver
Warp
Traps
Warp
Warp

Posinst
Multi

charge state beams
Warp
Beam

Beam effects
Beam

Beam3D
m
bunching in FEL linac injectors
Impact
Electron cloud effects
Posinst
Laser plasma acceleration
Warp
Inf&rno
beam dynamics in rings & linacs
Impact
Warp
CSR3D
Coherent Synchrotron Radiation
6 h on 2k CPUs
5 Billions part.
5
h on 1k CPUs
6 h on 12k CPUs
6 h on 80k CPUs
Injection
Transport
Plasma
neutralization
LEBT
–
Project X
LHC, RHIC, Tevatron, KEK

B
Alpha anti

H trap
PS
SNS
Montague resonance
SPS
Impact
FRIB
Paul trap
Courtesy H. Sugimoto
AFRD Review
–
May 2

3, 2012

Vay
18
BELLA Project

state

of

the

art
PW facility
for laser accelerator science
e

beam
~10
GeV
Laser
Plasma
~
1m
l
~1
m
m
Modeling from first principles challenging because of
scale separation
z
1D PIC simulation of 1 Bella stage demanded
~
5,000 CPU

hours in 2007
AFRD Review
–
May 2

3, 2012

Vay
19
L=0.8 m
l
=0.8
m
m
0.8 m/0.8
m
m
=
1,000,000.
Lab frame
compaction
X20,000.
l’=0.16
mm
8 mm/0.16 mm=
50.
Boosted frame
=
100
Hendrik Lorentz
L’=8 mm
Simulation in a Lorentz boosted frame reduces range of scales by
orders of magnitude*
*J.

L. Vay, P
hys. Rev.
Lett
.
98
, 130405 (2007)
Initial applications of the method (Berkeley Lab, Tech

X and UCLA) promising but
limited by:
Warp 2D simulation 10
GeV
LPA
(
n
e
=10
17
cc,
=130)
Longitudinal electric field
laser
plasma
numerical instability
Larger size of laser due to shorter
Rayleigh length in boosted frame
Lab frame
Boosted frame
AFRD Review
–
May 2

3, 2012

Vay
Cole

Karkkainen
EM solver with tunable numerical dispersion
J
.

L.
Vay,
et al.,
J.
Comput
. Phys.
230
(2011
)
“Strided” digital
filtering
Special time step
Instability level
Time step
Yee
Exact
20
Speedup limitations for boosting frame simulations have been overcome
Moving antenna enables compact and
efficient laser launching
J
.

L.
Vay,
et al
.,
PoP
18
, 123103 (2011
)
Lab frame
Wake
frame
converts laser
sp
at
ial
os
cil
la
ti
on
s
into
time
beating
Hyperbolic rotation from Lorentz T.
J
.

L.
Vay,
et al
.,
PoP Lett.
18
, 030701 (2011
)
Novel numerical techniques and key
observations allowed for efficient
mitigation of numerical instability
AFRD Review
–
May 2

3, 2012

Vay
21
Over
1 million
×
speedup
demonstrated on a single 1
TeV stage
J
.

L.
Vay,
et al
.,
PoP
18
(
2011
)
Warp
e

beam
>10
4
speedup for BELLA stage
BF method has enabled direct simulation of 10
GeV
stages with strong depletion.
AFRD Review
–
May 2

3, 2012

Vay
22
NGLS will deliver coherent
X

rays with high repetition rate,
unprecedented average brightness, and ultrafast pulses
CW superconducting
linac
,
l
aser heater, bunch compressor
High

brightness, high
rep

rate gun and
injector
Array of independent FELs
X

ray
beamlines
and
endstations
Beam dynamics simulations have to capture with sufficient
fidelity the:
•
interaction of
the electrons
with
external fields
(
to accelerate, transport
, and
compress
),
•
self

interaction that
tends to
spoil the beam
quality (
space

charge, radiation effects,
wakefields
)
.
Spatial scales:
•
l
inac
: radiation ~1
m
m, bunch length 0.1

1mm, machine length ~500m,
•
FEL:
radiation ~
1nm,
beamline
length ~
50m
.
AFRD Review
–
May 2

3, 2012

Vay
Start

to

end simulation
of
NGLS with
real number of electrons (
~
2 Billions)
First start

to

end simulation, required
~
8 hours on
2k
cores NERSC Hopper
computer.
beam kinetic energy and RMS sizes evolution
final current profile before undulator (A)
bunch length (
m
m)
z (m)
averaged FEL radiation power (MW) evolution
radiation power temporal distribution
at the end of the undulator
bunch length (
m
m)
z (m)
AFRD Review
–
May 2

3, 2012

Vay
IMPACT

T
IMPACT

Z
Genesis
24
Proof of principle 3D
boosted frame full EM
simulation
with Warp.
Efficient
modeling including
full 3D dynamics, arbitrary
beam shape and
topology.
(
future
work to include
conductors)
Boosted frame method accelerates first principle modeling of CSR effects
W.Fawley and J.

L. Vay,
Proc IPAC10
(2010)
AFRD Review
–
May 2

3, 2012

Vay
The
Heavy Ion Inertial Fusion (HIF)
program is studying the science of
ion

heated matter, as well as drivers & targets for inertial fusion
energy
Artist view of a
Heavy Ion Fusion
power plant
Deuterium+Tritium
Space/time scales span 8 orders of magnitude:
from <mm to km>/<ps to 100 ms>
from source…
…to target
25
Simulation goal
–
integrated self

consistent
predictive capability
including:
•
beam(s) generation, acceleration, focusing and
compression along accelerator,
•
loss of particles at walls, interaction with
desorbed gas and electrons,
•
neutralization from plasma in chamber,
•
target physics and diagnostics.
=> Need large

scale multiphysics computing
NDCX

II is our new
platform for studies of

space

charge

dominated beams

Warm Dense Matter
physics

beam

target energy
coupling
AFRD Review
–
May 2

3, 2012

Vay
Compression
Acceleration
Generation
3D & RZ Warp simulations used to design NDCX

II
A. Friedman,
et al
,
Phys. Plasmas
17
, 056704
(2010)
Injection of
neutralizing plasma
Aligned solenoids
Misaligned solenoids
(random offsets)
Plasma neutralization
Versatility of Warp code allows for integrated beam and plasma simulations,
combining all the necessary physics self

consistently.
AFRD Review
–
May 2

3, 2012

Vay
27
Simulation of e

cloud driven instability and its attenuation
using a feedback system in the CERN SPS
Transverse instability observed in SPS beams due to electron clouds
We use the Particle

In

Cell framework Warp

Posinst
to investigate dynamics of
instability as well as feasibility and requirements of feedback system
Pipe
e

gas
e

bunch 1
bunch 2
Beam ions
Electrons
Spurious image charges
from irregular meshing
controlled via guard cells
true sec.
back

scattered
elastic
re

diffused
Posinst provides advanced secondary electrons model
Monte

Carlo
generation of electrons
with energy and
angular dependence.
Warp’s mesh refinement &
parallelism provide efficiency
AFRD Review
–
May 2

3, 2012

Vay
28
Warp and
Posinst
have been further integrated, enabling fully self

consistent simulation of e

cloud effects: build

up
&
beam dynamics
CERN SPS
at injection (26
GeV
)
Turn 1
Turn 500
AFRD Review
–
May 2

3, 2012

Vay
29
Warp

Posinst
enabled the
first direct simulation
of a
train of 72 bunches

using 2880 CPUs on Franklin (NERSC)
Unexpected
substantial
density rise
for bunch ≥25
between turn
400 and 600.
Bunch 25
Average electron cloud density history at fixed station
E

cloud density rise associated with emittance and beam radius
growth
=> positive coupling between bunches evolution and electron generation.
J.

L. Vay, et al,
Ecloud10 Proc.
, (2010)
AFRD Review
–
May 2

3, 2012

Vay
30
Fractional tune
Comparison with experimental measurements

collaboration with SLAC/CERN
Good qualitative agreement: separation between
core and
tail
with similar tune
shift.
Warp is also applied to study
of
feedback control system (R.
Secondo in collaboration with SLAC)
Warp

Posinst
2
Bunch 29, Turn 100

200
head
tail
head
tail
Fractional tune
Bunch slice
Experiment
1
Bunch 119, Turn 100

200
Nominal
fractional
tune=0.185
Bunch slice
1
J. Fox, et al,
IPAC10 Proc.
, p. 2806 (2011)
2
J.

L. Vay, et al,
Ecloud10 Proc.
, (2010)
AFRD Review
–
May 2

3, 2012

Vay
Sensitivity to solenoid offset & voltage jitter in NDCX

II (Warp)
ensemble of 256 cases
~
4.5 hours on 6,144
CPU
s
(simulations by D. Grote)
Optimization LHC luminosity (BeamBeam3D)
ensemble of 100 populations
~
3 hours on 12,800 CPUs
Multilevel
parallelism
based on MPI
groups
is
used
for
parameter scans and optimization (
Ryne
, 2009)
31
solenoid alignment
voltage jitter
A. Friedman,
et al
,
Phys. Plasmas
17
, 056704
(2010)
J.
Qiang
,
et al
,
Proc. PAC 11
, p. 1770, (2011)
Multilevel parallelism enables very efficient parameter scans and optimization.
AFRD Review
–
May 2

3, 2012

Vay
32
AFRD codes used to discover new physics

Warp simulations of multipactor predicted new “ping

pong” modes*
0.000
0.001
0.002
0.003
1.0
0.5
0.0
0.5
1.0
10^+7
Vy vs Y
Y
Vy
z window0 = 2.2400e02, 2.2400e02
Vy vs Y
Y
Vy
z window0 = 2.2400e02, 2.2400e02
Step 240, T = 1.1628e9 s, Zbeam = 0.0000e+0 m
Rectangular Waveguide: BDC=0; E=34.22kV/m
dt= 4.8ps;nx,ny,nz=64x8x128;egrdnx,ny,nz=22x16x44
R.A. Kishek warp r2 rect!_MPC!_noB!_01
22
*R.A. Kishek,
Phys. Rev. Lett.
108
, 035003 (2012)
.
WARP 3D simulation of
rectangular waveguide
Red: primaries
Blue: secondaries
v
o
E
o
v
o
E
o
Warp
•
Modes lead to broadening of area of parameter
space where multipactor can occur.
•
Excellent agreement with WARP
“The nice thing is WARP predicted it first, and then
resulted in good agreement once I worked out the
details of the theory.”
–
R.
Kishek
, U. Maryland
Schematic of particle orbits in a period

2
ping

pong multipactor.
AFRD Review
–
May 2

3, 2012

Vay
Summary
33
AFRD develops and maintains cutting

edge accelerator codes

main codes have a worldwide user base
Major impact on DOE/SC (HEP,NP,BES,FES) programs

design, optimize and support accelerators

d
iscover new physics
AFRD algorithms pushing limits of state

of

the

art

s
everal have spread to other majors codes outside the lab
Development of in

house codes provides an edge to AFRD

c
onsolidation of efforts underway within division
Applications of the codes are at the forefront in several important
areas of accelerator physics
AFRD Review
–
May 2

3, 2012

Vay
Comments 0
Log in to post a comment