High-Fidelity Terascale Simulations of Turbulent

busyicicleMécanique

22 févr. 2014 (il y a 3 années et 3 mois)

71 vue(s)

High
-
Fidelity Terascale Simulations of Turbulent
Combustion

Jacqueline H. Chen, Evatt R. Hawkes, Ramanan Sankaran, James
C. Sutherland, and Joseph C. Oefelein

Combustion Research Facility

Sandia National Laboratories

Livermore CA


Supported by

Division of Chemical Sciences, Geosciences, and Biosciences,

Office of Basic Energy Sciences
, and DOE SciDAC


Computing: LBNL NERSC, ORNL CCS/NLCF, PNNL MSCF,

SNL HPCNP Infiniband test
-
bed,

SNL CRF BES Opteron cluster


Visualization: Kwan
-
Liu Ma and Hiroshi Akiba UC Davis

Outline

1.
DNS of turbulent combustion


challenges and opportunities


2.
Sandia S3D DNS capability


terascale simulations on
Office of Science platforms


3.
New combustion science to
advance predictive models:



3D simulations of turbulent jet
flames with detailed chemistry.



Layering Large Eddy Simulation
and DNS approaches.

Vorticity fields in DNS of a turbulent jet flame

(volume rendering by

Kwan
-
Liu Ma and Hiroshi Akiba)

Turbulent combustion is a grand challenge!

Diesel Engine Autoignition, Soot Incandescence

Chuck Mueller, Sandia National Laboratories


Stiffness : wide range of
length and time scales


turbulence


flame reaction zone


Chemical complexity


large number of species and
reactions (100’s of species,
thousands of reactions)


Multi
-
Physics complexity


multiphase (liquid spray, gas
phase, soot, surface)


thermal radiation


acoustics ...


All these are tightly coupled

Several decades of relevant scales


Typical range of spatial scales


Scale of combustor:
10


100
cm


Energy containing eddies:
1


10
cm


Small
-
scale mixing of eddies:
0.1


10
mm


Diffusive
-
scales, flame thickness:
10


100

m



Molecular interactions, chemical reactions:
1


10
nm


Spatial and
temporal dynamics inherently
coupled


All scales are relevant

and must be resolved or
modeled


O(4)

Range

O(4)

Continuum


Terascale computing:

~3 decades in scales

(cold flow)

Combustion CFD Approaches to tackle

different length scale ranges


Reynolds

Averaged Navier

Stokes (RANS)


Coarse meshes, full range of dynamic scales
modeled


Empirical

closure, bulk approximation, current engineering CFD


Large Eddy Simulation (LES)


Energetic scales
resolved
, sub
-
grid scale dynamics
modeled


combustion

closure required, captures large
-
scale unsteadiness


Direct Numerical Simulation (DNS)


No sub
-
grid modeling

required but limited on range of scales


Building
-
block configurations, research tool

Increasing cost

and resolution for

fixed physical problem


uL

Re
Role of Direct Numerical Simulation (DNS)


A

tool

for

fundamental

studies

of

the

micro
-
physics

of

turbulent

reacting

flows








A

tool

for

the

development

and

validation

of

reduced

model

descriptions

used

in

macro
-
scale

simulations

of

engineering
-
level

systems


DNS

Physical

Models

Engineering
-
level

CFD codes

(RANS and

future LES)

DNS


Physical

insight

into

chemistry

turbulence

interactions


Full

access

to

time

resolved

fields


CH4

Oxidizer

Fuel

CH3O

HO2

O

Piston Engines

S3D MPP DNS capability at Sandia


S3D code characteristics:


Solves compressible reacting Navier
-
Stokes


F90/F77, MPI, domain decomposition.


Highly scalable and portable


8
th

order finite
-
difference spatial


4
th

order explicit RK integrator


hierarchy of molecular transport models


detailed chemistry


multi
-
physics (sprays, radiation and soot) from
SciDAC “Terascale Simulation of Turbulent
Combustion”


70% parallel efficiency on 4096 processors on
NERSC; 90% parallel efficiency on 5120
CrayXT3 processors; 75% parallel efficiency
on 1000 CrayX1E processors



Terascale computations => need scalar
optimization customized to architecture


CCS, NERSC consultants

S3D is a state
-
of
-
the
-
art DNS code developed with

13 years of BES sponsorship.

S3D scales up to 1000s of
processors… and beyond?


S3D performance improvements on Seaborg

Application Software Case Studies in FY05 for MICS (K. Roche)



8% improvement in performance by minimizing
conversion between dimensional and nondimensional
units for interfacing with Chemkin



interface to Mathematical Acceleration Subsystem
(MASS) library for exponential and logarithmic
functions resulted in 10% improvement. Another 5%
due to using vectorized exp function in VMASS


Tabulating Gibbs energy as a function of temperature
resulted in 8% savings


Remove excessive dynamic allocation calls resulted
in 12% savings.


Eliminate MPI derived data types for communicating
noncontiguous boundary data with neighbors


Using Xprofiler, IPM and working with David Skinner:

45% efficiency improvement

Outline

1.
DNS of turbulent combustion


challenges and opportunities


2.
Sandia S3D DNS capability


terascale simulations on
Office of Science platforms


3.
New combustion science to
advance predictive models:



3D simulations of turbulent jet
flames with detailed chemistry



Layering Large Eddy Simulation
and DNS approaches

Vorticity fields in DNS of a turbulent jet flame

(volume rendering by

Kwan
-
Liu Ma and Hiroshi Akiba)

FY05 INCITE award: Direct simulation of a 3D turbulent nonpremixed
CO/H2/air jet flame with detailed chemistry (350 million grids, 12 species)


Objectives:


Study fine
-
grained coupling between
turbulent mixing and finite
-
rate
chemical effects



Extinction
-
reignition dynamics



Mixing time scales



Reynolds number effect



Preferential diffusion



Flow unsteadiness


Test assumptions in subgrid mixing
and reaction models

Isoscalar surface of scalar dissipation rate (local mixing rate)

INCITE run test case (1/2 resolved)
350 million grid points, Re=5000

x

x

y

z

The effect of chemistry
-
turbulence interactions
in turbulent nonpremixed jet flames:

mixing of passive and reacting scalars

Evatt Hawkes, Ramanan Sankaran,

James Sutherland, and Jacqueline Chen

Computational Platforms: NERSC Seaborg, ORNL CCS Phoenix Cray
X1E, XT3 and PNNL MPP2

Description of runs

-

Temporally evolving non
-
premixed plane jet flames



A canonical flow with shear
-
driven turbulence


Heat release iso
-
contours

Mixing,

Reaction

Mixing,

Reaction

x

y


Large computing allocations are enabling new science runs


INCITE at NERSC, capability computing at NLCF ORNL, ERCAP at PNNL, NERSC


Detailed H
2
/CO chemistry (17 d.o.f., Li et al. 2005)


Parameters selected to maximize Re



Case A:



Re 6000


40 million grid points


480 processors on PNNL MPP2


~ 1 TB data



Case B:


Re 8000


100 million grid points


1728 processors on Seaborg


192/240 processors on ORNL CCS Cray X1/E


~ 2.5 TB data

DNS data
-
sets of turbulent nonpremixed
H
2
/CO flames (mixing, extinction/reignition)


INCITE calculation:


Currently running on Seaborg


Re 5000


350 million grid points


~4096 Seaborg processors


3 million hours total


~ 9TB raw data



Two additional calculations,
parametric study in Re


Re 2500, 150 million grid points


Re 10,000, 500 million grid points

Community data sets


New opportunities for numerical benchmarks


highly resolved LES and
DNS



TNF workshop (1996
-
2004): International Collaboration of
Experimental and Computational Researchers


Framework for
detailed

comparison

of measured and modeled results
(http://www.ca.sandia.gov/TNF/abstract.html
)

Non
-
premixed combustion concepts


Mixture fraction Z: the amount of fluid from the fuel stream in the
mixture



Z is a conserved (passive) scalar


(no reactive source term)



Scalar dissipation, a measure of local molecular mixing rate:




Z
Z
D




2

Example development of the jet:

40 million grid PNNL run


Left: Vorticity. Right: Simultaneous volume rendering of mixture
fraction, scalar dissipation and OH radical.

(Rendering by Hiroshi Akiba and Kwan
-
Liu Ma, UC Davis)

100 million grid run on

NERSC Seaborg and ORNL Cray X1: Re = 8000

Scalar Dissipation

Vorticity

u




Z
Z
D




2

100 million grid run on

NERSC Seaborg and ORNL Cray X1: Re = 8000

HO
2

dissipation

OH dissipation

2
2
2
2
2
HO
HO
HO
HO
Y
Y
D





OH
OH
OH
OH
Y
Y
D




2

Mixing timescales


Models for molecular mixing are required in the PDF
approach to turbulent combustion (Pope 1985), a sub
-
grid
model used in RANS and LES approaches.



TNF workshop


CFD predictions are dependent on mixing
timescale choice.



Models assume that scalar mixing timescales are identical
for all scalars and determined by the turbulence timescale.


scalars with different diffusivities?


reactive scalars?

Definitions


Mechanical time
-
scale:




Scalar time
-
scale:




Time
-
scale ratio:



k
u










D
2
'
2




u
r

is assumed to be order unity in most models


r
is assumed to be the same for all scalars


r
Mixture fraction to mechanical timescale
ratio


Confirmation that mixture fraction to mechanical time scale ratio is
order unity.


Average value about 1.5, similar to values reported by
experiments, simple chemistry DNS, and used successfully in
models.


Timescale Ratio r
Z

Effect of diffusivity


Smaller, more highly
diffusive species do
have faster mixing
timescales


Increasing

diffusivity

r
φ

Finite
-
rate chemistry effects on mixing


HO
2

and H
2
O
2

have faster
mixing times in the middle
of the simulation, while
OH and O are lower




Diffusivity trend does not
appear to hold for HO
2

and H
2
O
2

versus O and
OH.




What is going on?

r
φ

Radical production and destruction in high
dissipation regions


OH is consumed while HO
2

is produced in high dissipation
regions

HO
2

OH

Color scale: mass fraction (blue is low; red is high)

White iso
-
contours:
scalar dissipation rate

Dissipation of passive and reactive scalars


Blue:

Z
, Green:

OH
, Red:

HO2



Dissipation fields of Z and HO
2

are co
-
incident and aligned with
principal strain directions



OH dissipation occurs elsewhere,
more in the centre of the jet



OH is reduced in the high
dissipation regions, leading to
longer mixing times, opposite for
HO
2



At later times, mixing rates relax,
OH returns and HO
2

decreases


100 million grid run on Seaborg / Cray X1


We need to establish
any Re dependence



Need parametrics at
larger Re



Run in progress…
but seems to be
showing same
effects

Conclusions
-

mixing timescales


New finding: detailed transport and chemistry effects can
alter the observed mixing timescales



Models may need to incorporate these effects


a poor mixing model could lead to incorrectly predicting a stable
flame when actually extinction occurs



This type of information cannot be determined any other
way at present


ambiguities in a
-
posteriori model tests


too difficult to measure


need 3D and detailed chemistry to see this


Knowledge Discovery From Terascale Datasets


Challenge
:

Need interactive
knowledge discovery software for
terascale high
-
fidelity scientific data
sets where computing pipeline is
distributed geographically.

Solution:

Intelligent visualization
pipeline



Simultaneous cognitive
visualization


Intelligent feature
extraction/tracking


Scalable transparent data sharing
and parallel I/O across platforms


INCITE Award
:
DNS of a 3D
turbulent flame (
9 TB

raw data
-

350
million grids, 17 variables, 100,000
steps)


Advanced post
-
processing for extinction/re
-
ignition


Extinction viewed as radical or
reaction
-
rate “hole” in a mixture
-
fraction isosurface



Stems from fast chemistry flame
-
sheet “flamelet” models of
turbulent combustion



Holes can be expanding or
contracting



Speed of the edge can be
monitored


need algorithm for
edge detection

Advanced post
-
processing for extinction/re
-
ignition


Holes could be tracked to determine any history effects


need parallel feature tracking

Outline

1.
DNS of turbulent combustion


challenges and opportunities


2.
Sandia S3D DNS capability


terascale simulations on
Office of Science platforms


3.
New combustion science to
advance predictive models:



2D DNS of HCCI combustion



3D simulations of turbulent jet
flames with detailed chemistry



Layering Large Eddy Simulation
and DNS approaches

Vorticity fields in DNS of a turbulent jet flame

(volume rendering by

Kwan
-
Liu Ma and Hiroshi Akiba)

Layering of LES and DNS approaches

(the future...)

Joseph C. Oefelein,

Evatt R. Hawkes, Jacqueline H. Chen

RANS
-
LES
-
DNS at High Re


LES gets lots more
than RANS



High Re is too
expensive for DNS
(at present)


can afford 1000s now

useful but…


would like 100000s
and more

Swirl
-
Stabilized Premixed Combustion

Azimuthal Velocity Component

RANS

LES

DNS

Why LES?


LES treats the large
scales directly



Large scales are
geometry
dependent



LES can couple
directly with high
Re experiments



But


LES needs
sub
-
grid models

Layering LES and DNS

High Re

Experiments

LES


Moderate Re:


Canonical problems


Cleverly designed
experiments

DNS


Sub
-
grid models


Better BCs, ICs


Identify modelling
questions


Identify relevant
parameter space

Applications

Multiphysics capabilities and

algorithmic framework


Oefelein


Theoretical framework (LES


DN匩


Fully
-
coupled conservation equations


Compressible, chemically reacting system


Real
-
fluid EOS, thermodynamics, transport


Detailed chemistry, multiphase flow, sprays


Dynamic subgrid
-
scale modeling


Numerical framework


Dual
-
time, all
-
Mach
-
number formulation


Generalized preconditioning methodology


Complex geometry, generalized coordinates


Massively
-
parallel (MPI), highly
-
scalable


Ported to all major platforms


Over 10 years development


Fine
-
grain scalability (NERSC IBM SP


Seaborg)


Grid size fixed, number of processors increased

Magnitude of vorticity (normalized)

DNS

(all scales)

40 million grid, 120,000 hours

LES

(large scales)

0.6 million grid, 600 hours

Summary


S3D is a state
-
of
-
the
-
art DNS capability for turbulent combustion
simulations that scales to thousands of processors and is ported to
all Office of Science platforms.



DOE supercomputing facilities are enabling new combustion
science.



3D DNS of detailed finite
-
rate chemistry effects in turbulent jets
provides new insights and data for combustion modeling:


mixing of reactive scalars can be very different from conserved
scalars



Layering LES and DNS approaches will open new avenues of
exploration with great potential for the future

100 million grid run on

NERSC Seaborg and ORNL Cray X1: Re = 8000

HO
2

dissipation

OH dissipation

S3D performance on different architectures