Advanced beam simulations in AFRD

hardtofindcurtainΠολεοδομικά Έργα

16 Νοε 2013 (πριν από 3 χρόνια και 11 μήνες)

93 εμφανίσεις

Advanced beam simulations in AFRD

J.
-
L.
Vay



Lawrence Berkeley National Laboratory

May 2
, 2012





1

Office of Science

SciDAC
-
II

Compass

2

Outline


AFRD beam simulation tools

-
Overview

-
Strength in innovative methods

-
Effort toward integration


Selected sample of recent
applications

-
BELLA, NGLS, NDCX
-
II, E
-
cloud SPS, multilevel
parallelism,
ping
-
pong

modes


Summary

AFRD Review


May 2
-
3, 2012
-

Vay


3

AFRD develops and supports several important physics codes

Code

Developers

Type

Description

BeamBeam3D
(F95)

J.
Qiang
,

R.
Ryne


3D parallel ES PIC


modeling strong
-
strong or strong
-
weak beam
-
beam
interactions in high energy colliders

CSR3D

(F95)

R.
Ryne

3D
Lienard
-
Wiechert

non
-
self
-
consistent parallel code for CSR studies

Ginger
(F95)

W.
Fawley

et al

2D
-
RZ and 3D PIC

polychromatic FEL simulation code

Impact
(F95)

J.
Qiang
,

R.
Ryne

3D parallel ES PIC w/
maps

framework for modeling high intensity, high brightness
beams in accelerators

INF&RNO
(C++)

C. Benedetti

2D
-
RZ parallel EM
PIC/fluid

lab/boosted frame w/
ponderomotive

approximation +
envelope model for the laser

Marylie
-
Impact
(F95)

R.
Ryne
, J.
Qiang

et
al.

3D parallel
ES PIC

w/ maps

Combination of
MaryLie

and IMPACT including both high
order optics based on Lie algebraic maps along with parallel
3D space
-
charge effects

POSINST
(F95)

M. Furman

et al.

2D
ES PIC

electron cloud buildup studies

Warp

(Python+F95)

D.

Grote, J
-
L Vay,
A.Friedman et al.

2D/3D parallel

ES & EM
PIC

framework with PIC & accelerator lattice for modeling of
particle beam generation, transport

& neutralization, or
laser
-
plasma interaction

Warp
-
POSINST

(Python+F95)

J
-
L Vay, M. Furman,
D. Grote

2D/3D parallel ES PIC w/
maps

Combination of Warp and POSINST for self
-
consistent
electron cloud studies

PIC=Particle
-
In
-
Cell: Lagrangian macro
particles + fields on grids (finite difference solvers);
interpolation between particles and
fields (ES=electrostatic; EM=electromagnetic).

AFRD Review


May 2
-
3, 2012
-

Vay


Many
users
at
institutions
worldwide

4

CERN (P,W)

Diamond
(I)

ESS (I)

Fermi/
Elettra

(I)

Frankfurt
(I
)

GSI
(I
,W)

Hiroshima U. (W)

Hong Kong U. (W)

IBS (I)

ANL (B,I,P)

BNL (B,I,P,W)

Cornell
(I
,P)

FNAL (B,I,P,W)

ISU (I)

Jlab

(B,
I,P)

LANL
(I
,P)

LBNL (B,I,P,W)

LLNL (W)

MSU
(I
,W
)

NIU
(I)

ORNL (I
,P)

SLAC
(I
,P,W)

Stanford
(I)

Tech
-
X (P)

UM (W)

UMD (W)

UW
(I
)

UCLA
(I
)

WSU (W
)

Yale U (B,I)

United States

Europe/Asia

B

I

P

W



BeamBeam3D



Impact



Posinst



Warp

IHEP (I
,P)

IMPCAS
(I
)

KEK
(I
)

PAL
(I
)

PSI
(I)

RRCAT (I)

RAL (I)

SINAP
(I
)

Technion (W)

AFRD Review


May 2
-
3, 2012
-

Vay


AFRD tools are applied to the design, optimization, risk
minimization and support of particle accelerators


Application to


Linacs
, transfer lines, rings, colliders, injectors, particle traps (e.g. anti
-
H) and l
aser plasma
accelerators (LPA)


Hadron machines, lepton machines, multi
-
charge state beams



w
ith
i
mpacts spanning
the breadth of DOE/SC


HEP
:
Tevatron
, Main Injector, NML
photoinjector
, LHC, SPS, LHC injector upgrades,
ILC,
Bella, CESR
-
TA, Project
-
X


NP
: FRIB, e
-
-
ion colliders


BES
: SNS, LCLS, NGLS


FES
: Ion beam dynamics for Heavy Ion Fusion and
HEDP (NDCX
-
II)



partially funded by SciDAC/ComPASS, collaborations with ASCR


also partially funded by LDRD and SBIR partnership in the past

5

AFRD Review


May 2
-
3, 2012
-

Vay


AFRD code developers and
users

J. Barnard
3
*

C.
Benedetti
4

S.
Bulanov
4

M.
Chen
4

R.
Cohen
3
*

A
.
Friedman
3
*

M.
Furman
2

C.
Geddes
4

D.
Grote
3
*

E.
Henestroza
3

Q.
Ji
3

S.
Lund
3
*

H. Nishimura
1

A. Persaud
3


6

C. Papadopoulos
1,
2

S.
Paret
2

G.
Penn
2

J.
Qiang
2

M.
Reinsch
2

S.
Rykovanov
4

R.
Ryne
2

W.
Sharp
3
*

C. Sun
1

M.
Terry
3
*

J.
-
L.
Vay
4,
2
,
3

M.
Venturini
2

W. Wan
1

L.
Yu
4


28 total (21+7
guests)


1
ALS (4)

2
CBP (7+)

3
HIF/
IBT (10+)

4
LOASIS (6+)

*
guest (LLNL
)


AFRD Review


May 2
-
3, 2012
-

Vay


Fruitful collaborations between
AFRD & CRD/NERSC

Recent & present AFRD
-
CRD/NERSC collaborations



E. Wes Bethel et al (VACET): beam path analysis for LPA

-
O.
Rubel
, C. G. R. Geddes, E. Cormier
-
Michel, K. Wu,
Prabhat

, G. H. Weber, D. M.
Ushizima
, P.
Messmer
, H.
Hagen, B.
Hamann
, E. W. Bethel,
Automatic beam path analysis of laser
wakefield

particle acceleration data
,
Computational Science & Discovery, vol. 2, 015005 (2009)



ExaHDF5 team: parallel I/O, analysis,
visualization

-
Chou, Wu,
Rubel
,
Howison
,
Qiang
,
Prabhat
, Austin, Bethel,
Ryne
,
Shoshani
,
Parallel Index and Query for Large
Scale Data Analysis
, to appear in
SuperComputing

2011.



H. Shan and X.
Li: parallel performance optimization



P
.
Colella
: LOASIS AMR modeling of capillary



A. Koniges: B. Austin (parallel optimization), B. Liu (ALE
-
AMR)

Proximity of CRD & NERSC is a great asset.

AFRD Review


May 2
-
3, 2012
-

Vay


8

Outline


AFRD beam simulation tools

-
Overview

-
Strength in innovative methods

-
Effort toward integration


Selected sample of recent
applications

-
BELLA, NGLS, NDCX
-
II, E
-
cloud SPS, multilevel
parallelism,
ping
-
pong

modes


Summary

AFRD Review


May 2
-
3, 2012
-

Vay


Many algorithms

invented, improved or
pioneered in AFRD codes

9

Algorithm/method

Reference

Originated

Adopted

by

Damped

EM & particle pushers

Friedman, JCP 1990

Warp

LSP,Elixirs

Warped coordinates
PIC in bends

Friedman

et al, Phys. Fluid 1992

Warp

Integrated Maps for
rf

cavity dynamics

Ryne
, LANL Report 1995

ML/I

D.
Abell

(nonlinear
model)

Stochastic Leap
-
Frog

for Brownian motion

Qiang

&
Habib
, PRE 2000

Impact

Spectral
-
finite difference
multigrid

solver

Qiang

&
Ryne
, CPC 2001

Impact

Improved Perfectly

Matched Layers

Vay, JCP 2000
/JCP 2002

Warp

Osiris

AMR
-
PIC electrostatic

Vay et al,

LPB2002/PoP2004

Warp

Secondary emission of
electrons algorithm

Furman &

Pivi
, PRST
-
AB 2003

Posinst

TxPhysics

AMR
-
PIC electromagnetic

Vay et al,

CPC 2004

Emi2D

Warp

3D Poisson solver with large aspect ratio

Qiang

&
Gluckstern
,

CPC 2004

Impact

Shift
-
Green

f
unction method

Qiang

et al, CPC 2004

BBeam3D

Integrated

Green function

Ryne

&
Qiang

ML/I

BB3D,Impact

Hybrid Lorentz

particle pusher

Cohen et al, NIMA 2007

Warp

AFRD Review


May 2
-
3, 2012
-

Vay


and adopted by other codes

10

Algorithm/method (cont.)

Reference

Originated

Adopted

by

Lorentz boosted frame*

Vay, PRL 2007

Warp

Inf&rno,JPIC
,

Osiris,Vorpal
,VPIC

Explicit Lorentz invariant particle pusher

Vay,
PoP

2008

Warp

Tristan (
astro
), QED

New convolution integral w/ smooth kernel

Qiang
, CPC 2010

N/A

Mixed

Particle
-
Field decomposition method

Qiang & Li, CPC 2010

BBeam3D

Improved laser envelope

model for LPA

Cowan,

Esarey et al, JCP 2011

Vorpal

PIC with tunable

electromagnetic solver

Vay et al, JCP 2011

Warp

Vorpal,Osiris

Efficient digital filter for PIC

Vay et al, JCP 2011

Warp

Vorpal,Osiris

Laser launcher

from moving antenna

Vay et al,
PoP

2011

Warp

Vorpal,Osiris

High precision laser envelope

model

Benedetti et al, 2011

Inf&rno

*
Phys. Rev. Lett.

98

(2007)

Example


Lorentz boosted frame method and associated numerical techniques have been
adopted by others:



have helped Tech
-
X implementing moving antenna algorithm in
Vorpal
,



UCLA has requested help for implementation in Osiris.

AFRD Review


May 2
-
3, 2012
-

Vay


11

Outline


AFRD beam simulation tools

-
Overview

-
Strength in innovative methods

-
Effort toward integration


Selected sample of recent
applications

-
BELLA, NGLS, NDCX
-
II, E
-
cloud SPS, multilevel
parallelism,
ping
-
pong

modes


Summary

AFRD Review


May 2
-
3, 2012
-

Vay


RB1

RB2



RA1

RA2



In the past, codes developed mostly separately

12

RE1

RE2



Developers

Codes

J.
Qiang
, R.
Ryne

BeamBeam3D

I
mpact

C. Benedetti

INF&RNO

M. Furman

POSINST

RC1

RC2



RD1

RD2



D. Grote, J
-
L Vay

Warp

CBP

Fusion

CBP

LOASIS

AFRD Review


May 2
-
3, 2012
-

Vay


RB1

RB2



RA1

RA2



Recently, Warp & Posinst were bridged and Warp expanded
its reach within the division

13

RE1

RE2



Developers

Codes

J.
Qiang
, R.
Ryne

BeamBeam3D

I
mpact

C. Benedetti

INF&RNO

M. Furman

POSINST

RC1

RC2



RD1

RD2



D. Grote, J
-
L Vay

Warp

Warp
-
POSINST

CBP

Fusion/IBT,CBP,LOASIS

CBP

LOASIS

AFRD Review


May 2
-
3, 2012
-

Vay


RB1

RB2



RA1

RA2



Toward further integration



--

some AFRD codes ported in common repository

14

Field

solver A

Particle

pusher A

A

B

C

CRD

Field

solver B

AFRD Common Repository


1

Particle pusher C

Other
1


3


4

Developers

Codes


2

Together with increased modularity, this will provide opportunities for:

-
co
-
development within AFRD,

-
collaboration with CRD (easier & more productive on isolated modules than on full codes).

CRD Repository

Field solver C

Other
2

CRD=Computational Research Division

AFRD Review


May 2
-
3, 2012
-

Vay


15


No black box: 100% knowledge of numerical methods and implementation

-
g
reatly reduces uncertainties in understanding simulation results,

-
thus greatly enhance chances of agreement, or understanding of reason for
disagreement, between simulations and experiments,

-
allows fast development of specialized or improved
algorithms.



C
ontrol of priorities and pace of needed capabilities.



S
imulation codes are increasingly becoming critical strategic assets

-
bigger, faster computers and improved numerical methods increase fidelity,

-
the embodied physical models are constantly improved.


A
ccess to the best simulation codes allows for faster design, commissioning
and understanding of experiments.

Why we are developing codes within AFRD

AFRD Review


May 2
-
3, 2012
-

Vay


16

Outline


AFRD beam simulation tools

-
Overview

-
Strength in innovative methods

-
Effort toward integration


Selected sample of recent
applications

-
BELLA, NGLS, NDCX
-
II, E
-
cloud SPS
, multilevel
parallelism
,
ping
-
pong

modes


Summary

AFRD Review


May 2
-
3, 2012
-

Vay


Wide array
of applications

17

-0.04
-0.02
0.00
0.02
0.04
y

[
m
]
-0.04
-0.02
0.00
0.02
0.04
x [m]
CESRTA, By=0.22 T
Nb=7e10, Eb=5.3 GeV
tb=14 ns, dmax=1.3
2.0x10
13
1.5
1.0
0.5
0.0
m
*
*
-
3
HEDP/HIF driver

Warp

Traps

Warp

Warp
-
Posinst

Multi
-
charge state beams

Warp

Beam
-
Beam effects

Beam
-
Beam3D

m
bunching in FEL linac injectors

Impact

Electron cloud effects

Posinst

Laser plasma acceleration

Warp

Inf&rno


beam dynamics in rings & linacs

Impact

Warp

CSR3D

Coherent Synchrotron Radiation

6 h on 2k CPUs

5 Billions part.

5

h on 1k CPUs

6 h on 12k CPUs

6 h on 80k CPUs

Injection

Transport

Plasma

neutralization

LEBT


Project X

LHC, RHIC, Tevatron, KEK
-
B

Alpha anti
-
H trap

PS

SNS

Montague resonance

SPS

Impact

FRIB

Paul trap

Courtesy H. Sugimoto

AFRD Review


May 2
-
3, 2012
-

Vay


18

BELLA Project


--

state
-
of
-
the
-
art
PW facility

for laser accelerator science

e
-

beam

~10
GeV

Laser

Plasma
~
1m

l
~1
m
m

Modeling from first principles challenging because of
scale separation

z

1D PIC simulation of 1 Bella stage demanded
~
5,000 CPU
-
hours in 2007

AFRD Review


May 2
-
3, 2012
-

Vay


19

L=0.8 m


l
=0.8
m
m

0.8 m/0.8
m
m
=
1,000,000.

Lab frame

compaction


X20,000.

l’=0.16
mm

8 mm/0.16 mm=
50.

Boosted frame


=
100

Hendrik Lorentz

L’=8 mm

Simulation in a Lorentz boosted frame reduces range of scales by
orders of magnitude*

*J.
-
L. Vay, P
hys. Rev.
Lett
.

98
, 130405 (2007)

Initial applications of the method (Berkeley Lab, Tech
-
X and UCLA) promising but
limited by:

Warp 2D simulation 10
GeV

LPA
(
n
e
=10
17
cc,

=130)

Longitudinal electric field

laser

plasma

numerical instability

Larger size of laser due to shorter
Rayleigh length in boosted frame

Lab frame

Boosted frame

AFRD Review


May 2
-
3, 2012
-

Vay


Cole
-
Karkkainen

EM solver with tunable numerical dispersion

J
.
-
L.
Vay,
et al.,
J.
Comput
. Phys.

230

(2011
)

“Strided” digital
filtering

Special time step

Instability level

Time step

Yee

Exact

20

Speedup limitations for boosting frame simulations have been overcome

Moving antenna enables compact and
efficient laser launching

J
.
-
L.
Vay,
et al
.,
PoP

18
, 123103 (2011
)


Lab frame

Wake
frame

converts laser
sp
at
ial
os
cil
la
ti
on
s

into
time

beating

Hyperbolic rotation from Lorentz T.

J
.
-
L.
Vay,
et al
.,
PoP Lett.

18
, 030701 (2011
)

Novel numerical techniques and key
observations allowed for efficient
mitigation of numerical instability

AFRD Review


May 2
-
3, 2012
-

Vay


21

Over
1 million
×

speedup
demonstrated on a single 1
TeV stage

J
.
-
L.
Vay,
et al
.,
PoP

18

(
2011
)


Warp


e
-
beam

>10
4

speedup for BELLA stage

BF method has enabled direct simulation of 10
GeV

stages with strong depletion.

AFRD Review


May 2
-
3, 2012
-

Vay


22

NGLS will deliver coherent
X
-
rays with high repetition rate,
unprecedented average brightness, and ultrafast pulses

CW superconducting
linac
,

l
aser heater, bunch compressor

High
-
brightness, high
rep
-
rate gun and
injector

Array of independent FELs

X
-
ray
beamlines

and
endstations

Beam dynamics simulations have to capture with sufficient
fidelity the:


interaction of
the electrons
with
external fields
(
to accelerate, transport
, and
compress
),


self
-
interaction that
tends to
spoil the beam
quality (
space
-
charge, radiation effects,
wakefields
)
.


Spatial scales:


l
inac
: radiation ~1
m
m, bunch length 0.1
-
1mm, machine length ~500m,


FEL:
radiation ~
1nm,
beamline

length ~
50m
.


AFRD Review


May 2
-
3, 2012
-

Vay


Start
-
to
-
end simulation
of
NGLS with
real number of electrons (
~
2 Billions)

First start
-
to
-
end simulation, required
~
8 hours on
2k
cores NERSC Hopper
computer.

beam kinetic energy and RMS sizes evolution

final current profile before undulator (A)

bunch length (
m
m)

z (m)

averaged FEL radiation power (MW) evolution

radiation power temporal distribution

at the end of the undulator

bunch length (
m
m)

z (m)

AFRD Review


May 2
-
3, 2012
-

Vay


IMPACT
-
T

IMPACT
-
Z

Genesis

24

Proof of principle 3D
boosted frame full EM
simulation
with Warp.


Efficient
modeling including
full 3D dynamics, arbitrary
beam shape and
topology.


(
future
work to include
conductors)

Boosted frame method accelerates first principle modeling of CSR effects

W.Fawley and J.
-
L. Vay,
Proc IPAC10
(2010)

AFRD Review


May 2
-
3, 2012
-

Vay


The
Heavy Ion Inertial Fusion (HIF)
program is studying the science of
ion
-
heated matter, as well as drivers & targets for inertial fusion
energy

Artist view of a

Heavy Ion Fusion
power plant

Deuterium+Tritium




Space/time scales span 8 orders of magnitude:


from <mm to km>/<ps to 100 ms>

from source…

…to target

25

Simulation goal


integrated self
-
consistent
predictive capability

including:


beam(s) generation, acceleration, focusing and
compression along accelerator,


loss of particles at walls, interaction with
desorbed gas and electrons,


neutralization from plasma in chamber,


target physics and diagnostics.




=> Need large
-
scale multiphysics computing


NDCX
-
II is our new
platform for studies of

-
space
-
charge
-
dominated beams

-
Warm Dense Matter
physics

-
beam
-
target energy
coupling

AFRD Review


May 2
-
3, 2012
-

Vay


Compression

Acceleration

Generation


3D & RZ Warp simulations used to design NDCX
-
II

A. Friedman,
et al
,
Phys. Plasmas

17
, 056704

(2010)

Injection of

neutralizing plasma

Aligned solenoids

Misaligned solenoids

(random offsets)

Plasma neutralization

Versatility of Warp code allows for integrated beam and plasma simulations,
combining all the necessary physics self
-
consistently.

AFRD Review


May 2
-
3, 2012
-

Vay


27

Simulation of e
-
cloud driven instability and its attenuation
using a feedback system in the CERN SPS

Transverse instability observed in SPS beams due to electron clouds





We use the Particle
-
In
-
Cell framework Warp
-
Posinst

to investigate dynamics of
instability as well as feasibility and requirements of feedback system


Pipe

e
-

gas

e
-

bunch 1

bunch 2


Beam ions


Electrons


Spurious image charges
from irregular meshing
controlled via guard cells

true sec.

back
-
scattered

elastic

re
-
diffused

Posinst provides advanced secondary electrons model

Monte
-
Carlo
generation of electrons
with energy and
angular dependence.

Warp’s mesh refinement &
parallelism provide efficiency

AFRD Review


May 2
-
3, 2012
-

Vay


28

Warp and
Posinst

have been further integrated, enabling fully self
-
consistent simulation of e
-
cloud effects: build
-
up
&

beam dynamics

CERN SPS

at injection (26
GeV
)

Turn 1

Turn 500

AFRD Review


May 2
-
3, 2012
-

Vay


29

Warp
-
Posinst

enabled the
first direct simulation

of a
train of 72 bunches



--

using 2880 CPUs on Franklin (NERSC)

Unexpected
substantial
density rise

for bunch ≥25
between turn
400 and 600.

Bunch 25

Average electron cloud density history at fixed station

E
-
cloud density rise associated with emittance and beam radius
growth

=> positive coupling between bunches evolution and electron generation.

J.
-
L. Vay, et al,
Ecloud10 Proc.
, (2010)

AFRD Review


May 2
-
3, 2012
-

Vay


30

Fractional tune

Comparison with experimental measurements



--

collaboration with SLAC/CERN

Good qualitative agreement: separation between
core and
tail
with similar tune
shift.


Warp is also applied to study
of
feedback control system (R.
Secondo in collaboration with SLAC)

Warp
-
Posinst
2

Bunch 29, Turn 100
-
200

head

tail

head

tail

Fractional tune

Bunch slice

Experiment
1

Bunch 119, Turn 100
-
200

Nominal
fractional
tune=0.185

Bunch slice

1
J. Fox, et al,
IPAC10 Proc.
, p. 2806 (2011)

2
J.
-
L. Vay, et al,
Ecloud10 Proc.
, (2010)

AFRD Review


May 2
-
3, 2012
-

Vay



Sensitivity to solenoid offset & voltage jitter in NDCX
-
II (Warp)


ensemble of 256 cases


~
4.5 hours on 6,144
CPU
s





(simulations by D. Grote)



Optimization LHC luminosity (BeamBeam3D)


ensemble of 100 populations


~
3 hours on 12,800 CPUs





Multilevel
parallelism
based on MPI
groups

is
used
for
parameter scans and optimization (
Ryne
, 2009)

31

solenoid alignment

voltage jitter

A. Friedman,
et al
,
Phys. Plasmas

17
, 056704

(2010)

J.
Qiang
,
et al
,
Proc. PAC 11
, p. 1770, (2011)

Multilevel parallelism enables very efficient parameter scans and optimization.

AFRD Review


May 2
-
3, 2012
-

Vay


32

AFRD codes used to discover new physics



--

Warp simulations of multipactor predicted new “ping
-
pong” modes*

0.000
0.001
0.002
0.003
-1.0
-0.5
0.0
0.5
1.0
10^+7
Vy vs Y
Y
Vy
z window0 = -2.2400e-02, 2.2400e-02
Vy vs Y
Y
Vy
z window0 = -2.2400e-02, 2.2400e-02
Step 240, T = 1.1628e-9 s, Zbeam = 0.0000e+0 m
Rectangular Waveguide: BDC=0; E=34.22kV/m
dt= 4.8ps;nx,ny,nz=64x8x128;egrdnx,ny,nz=22x16x44
R.A. Kishek warp r2 rect!_MPC!_noB!_01
22
*R.A. Kishek,

Phys. Rev. Lett.
108
, 035003 (2012)
.

WARP 3D simulation of
rectangular waveguide

Red: primaries

Blue: secondaries


v
o

E
o

v
o

E
o
Warp


Modes lead to broadening of area of parameter
space where multipactor can occur.


Excellent agreement with WARP


“The nice thing is WARP predicted it first, and then
resulted in good agreement once I worked out the
details of the theory.”


R.
Kishek
, U. Maryland

Schematic of particle orbits in a period
-
2

ping
-
pong multipactor.

AFRD Review


May 2
-
3, 2012
-

Vay


Summary

33


AFRD develops and maintains cutting
-
edge accelerator codes

-
main codes have a worldwide user base


Major impact on DOE/SC (HEP,NP,BES,FES) programs

-
design, optimize and support accelerators

-
d
iscover new physics


AFRD algorithms pushing limits of state
-
of
-
the
-
art

-
s
everal have spread to other majors codes outside the lab


Development of in
-
house codes provides an edge to AFRD

-
c
onsolidation of efforts underway within division


Applications of the codes are at the forefront in several important
areas of accelerator physics


AFRD Review


May 2
-
3, 2012
-

Vay