the Parallel Future

csinparallel.org

Surviving in the Wild:

Teaching and Training for
the Parallel Future

Dick Brown

St. Olaf College

SPLASH Educators’ and Trainers Symposium

October 24, 2011

csinparallel.org

Overview

•
Review of the need for parallelism


•
Strategies we need

–
What to teach

–
How to teach it

–
How to get it taught


•
Surviving in the wild of Parallelism


With interludes, to be announced…


csinparallel.org

The need for parallelism

•
Question:
Why do we need parallelism at
the programming level?

–
Hint:
It’s
not

“because it’s there”

(… although desktop applications do tend to find
ways to use ever greater power per dollar)


Note:
I will use
parallel

as a generic for
concurrent, parallel, distributed, cloud,
accelerator
(e.g., GPGPU), etc.

csinparallel.org

The need for parallelism

•
Answer:
Scale

csinparallel.org

The need for parallelism

•
Answer:
Scale


•
Cloud applications

+

= 30,000,000,000,000,000

csinparallel.org

The need for parallelism

•
Answer:
Scale


•
Cloud applications

•
Scientific applications: Particle
-
level
simulation of
turbulance

is
exascale


•
Can’t achieve
exascale

performance without

many cores (Berkeley “walls”), accelerators


csinparallel.org

Challenges for industry

•
Technology:

–
Heterogeneous computing (CPU + accelerators)

–
Sophisticated “on the fly” runtime systems

–
“Wall” of memory hierarchy vs. on
-
chip access


csinparallel.org

Challenges for industry

•
Technology:

–
Heterogeneous computing (CPU + accelerators)

–
Sophisticated “on the fly” runtime systems

–
“Wall” of memory hierarchy vs. on
-
chip access


•
Examples

–
AMD Fusion System Architecture: CPU+GPU

–
Intel MIC (Many Integrated Cores):

50+ CPUs on a chip, as a cluster
-
like accelerator


csinparallel.org

Challenges for industry

•
Technology:

–
Heterogeneous computing (CPU + accelerators)

–
Sophisticated “on the fly” runtime systems

–
“Wall” of memory hierarchy vs. on
-
chip access


•
Programming models:

–
Higher level; more “human
-
centric”

–
Scalable

–
Versatile


csinparallel.org

Challenges for Education/Training


We want to prepare our students for what

they’ll need, before the demand explodes,
but


•
What are the enduring principles?

•
Technologies, (hence) tools change rapidly!

•
(Educators:)
Change the curriculum???


csinparallel.org

A wild ecosystem

•
Industry/Academia


csinparallel.org

A wild ecosystem

•
Industry/Academia

•
Learning curve/Rapid change


csinparallel.org

A wild ecosystem

•
Industry/Academia

•
Learning curve/Rapid change

•
Principles/Practices


csinparallel.org

A wild ecosystem

•
Industry/Academia

•
Learning curve/Rapid change

•
Principles/Practices

•
Teaching/Research

–
New research discoveries in technology and
programming models need to get into the
curriculum
yesterday


csinparallel.org

A wild ecosystem

•
Industry/Academia

•
Learning curve/Rapid change

•
Principles/Practices

•
Teaching/Research




We need strategy!

And, it’s coming fast!


–
Took OOPSLA 20 years to become SPLASH…

We can’t wait that long




csinparallel.org

Strategies to be found

•
What to teach

•
How to teach it

•
How to get it taught

ITiCSE

2010

working group,
Strategies for Preparing Computer Science Students

for the
Multicore

World

csinparallel.org

What to teach

•
Parallel computing has a head start:

ACM/IEEE Curriculum ’91

–
3
required
hours on parallel algorithms

–
3
required

hours on distributed and parallel
programming language constructs, with hands
-
on practice

•
Ada
,


Concurrent

Pascal,

Occam,

or

Parlog


(Was not universally embraced…)

csinparallel.org

What to teach

•
Parallel computing has a head start:

ACM/IEEE Curriculum ’91

–
3
required
hours on parallel algorithms

–
3
required

hours on distributed and parallel
programming language constructs, with hands
-
on
practice


But,
ten years later…

•
ACM/IEEE Curriculum ’01

–
0 required hours

of parallel algorithms

–
No mention of

programming language constructs

–
Replaced by: “net
-
centric computing,” etc.

NSF/TCPP Curriculum Standards Initiative
in Parallel and Distributed Computing
–

Core Topics for Undergraduates


Sushil

K. Prasad,
IEEE TCPP Chair, Georgia State University

Richard LeBlanc
, Seattle University, ACM Education Council

Charles Weems
, University of Massachusetts, Amherst

Alan
Sussman
, University of Maryland

Arnold Rosenberg,
Northeastern and Colorado State University

Andrew
Lumsdaine
,
Indiana University





Curriculum Initiative Website: linked through
tcpp.computer.org

http://www.cs.gsu.edu/~tcpp/curriculum/index.php



Who are we?

•
Chtchelkanova, Almadena
-

NSF

•
Dehne, Frank
-

University of Carleton,
Canada

•
Gouda, Mohamed
-

University of Texas,
Austin, NSF

•
Gupta, Anshul
-

lBM T.J. Watson
Research Center

•
JaJa, Joseph
-

University of Maryland

•
Kant, Krishna
-

NSF, Intel

•
La Salle, Anita
-

NSF

•
LeBlanc, Richard, University of Seattle

•
Lumsdaine, Andrew
-

Indiana
University

•
Padua, David
-

University of Illinois at
Urbana
-
Champaign

•
Parashar, Manish
-

Rutgers, NSF

•
Prasad, Sushil
-

Georgia State University

•
Prasanna, Viktor
-

University of
Southern California

•
Robert, Yves
-

INRIA, France

•
Rosenberg, Arnold
-

Colorado State
University

•
Sahni, Sartaj
-

University of Florida

•
Shirazi, Behrooz
-

Washington State
University

•
Sussman, Alan
-

University of Maryland

•
Weems, Chip, University of
Massachussets

•
Wu, Jie
-

Temple University

Specifying Curriculum
Recommendations
–

NSF/TCPP Approach

•
Identify topics in

four existing
areas:
architecture, algorithms, programming, and
cross
-
cutting topics

•
For each topic, recommend

–
Bloom level

–
“Hours” of coverage

–
Suggested learning outcome

–
Possible core course for
coverage

•
Focus: First two years


Bloom Levels

Use first three levels for recommended core topics

•
K= Know the term/recall definition (basic literacy)

•
C = Comprehend so as to paraphrase/illustrate

•
A = Apply it in some way (requires operational



command
)


•
N = Not in core (but may be useful in elective or
advanced courses)

Example

•
Parallel and Distributed Models and
Complexity

–

Costs of computation

Algorithms Topics

Bloom
#

Course

Learning Outcome

Algorithmic problems

The important thing here is to emphasize the
parallel/distributed aspects of the topic

Communication

broadcast

C/A

Data
Struc/Algo

represents method of exchanging information
-

one
-
to
-
all broadcast (by recursive doubling)

multicast

K/C

Data
Struc/Algo

Illustrate macro
-
communications on rings, 2D
-
grids and trees

scatter/gather

C/A

Data Structures/Algorithms

gossip

N

Not in core

Asynchrony

K

CS2

asynchrony as exhibited on a distributed
platform, existence of race conditions

Synchronization

K

CS2, Data
Struc/Algo

aware of methods of controlling race
condition,

Sorting

C

CS2, Data
Struc/Algo

parallel merge sort,

Selection

K

CS2, Data
Struc/Algo

min/max, know that selection can be
accomplished by sorting

K: know term

C: paraphrase/illustrate

A: apply

Programming

•
Assume some conventional (sequential)
programming experience

•
Key is to introduce parallel programming
early
to students

•
Four overall areas

–
Paradigms
–

By target machine model and by
control statements

–
Notations
–

language/library constructs

–
Correctness
–

concurrency control

–
Performance
–

for different machine classes

Parallel Programming
Paradigms
(Selections)

•
By target machine model

–
Shared memory (Bloom classification
A
)

–
Distributed memory (
C
)

–
Client/server (
C
)

–
Hybrid
(
K
)
–

e.g., CUDA for CPU/
GPU

•
By control statements

–
Task/thread spawning (
A
)

–
Parallel Loop (
C
)


How to Read the Proposal

•
Oh no! Not another class to squeeze into our
curriculum!

•
Oh yes!
Not
another class to squeeze into your
curriculum
!






How to Read the Proposal

•
Oh yes!
Not
another class to squeeze into your
curriculum
!



•
Draft curriculum released Dec 2010
(
tcpp.computer.org
)







How to Read the Proposal

csinparallel.org

Enduring skills?

•
Since the tool set is subject to change at any
time, how much investment in those skills?

–
Many parallel languages and features have come
and gone


•
Need hands
-
on experience for effective
learning.

•
Anything may suddenly emerge as important

–
Python as a prototyping language for HPC

csinparallel.org

A candidate for addition



Patterns of parallel programming

csinparallel.org

Patterns, a candidate for addition

•
Background

–
“Gang of Four” book


csinparallel.org

Patterns, a candidate for addition

•
Background

–
“Gang of Four” book, 1994

–
Doug Lea,
Concurrent programming in Java:
Design principles and patterns
, 1999

–
Tim Mattson, et al,
Patterns for Parallel
Programming
, 2005

–
Kurt Kreutzer and Berkeley
Parlab
, the
Dwarves
Motifs

–
Kreutzer and Mattson, OPL
(
parlab.eecs.berkeley.edu
/wiki/patterns)



csinparallel.org

Patterns, a candidate for addition

Why patterns?

•
They capture
reusable units of expert
problem
-
solving strategy

•
Thus, they provide novices with a way to
acquire expertise

•
Many are
supported by tools

–
Loop parallel
,
Message passing
,
Map
-
reduce
, …



csinparallel.org

How to teach it

•
Agree with NSF/TCPP Initiative, that
parallelism should be taught
early and often

–
Scratch team kept concurrent scripts, because
users “not surprised that a sprite can do several
things at once”

–
Lessons of
Vishkin’s

“Peanut Butter Sandwich”
exercise

csinparallel.org

CSinParallel project

•
Add parallelism early and often at all levels

•
Incremental, flexible approach via modules

•
Sharing within our community


csinparallel.org

CSinParallel project

•
Modular Approach

–
Short units (1
-
3 days)

–
Identified learning objectives

–
Self
-
contained

–
Flexible for use in various courses and curricula

•
Make software/libraries more accessible

–
Parallel Platform Packages, Resources

•
Share, discuss and help as a community

–
http://
csinparallel.org


csinparallel.org

CSinParallel project

Some selected module topics

•
Introductory:

–
Map
-
Reduce computing for CS1 using
WebMapReduce

–
Concurrent access to data structures in Java or C++

–
Multicore

programming with Intel’s
Manycore

Testing Lab

•
Intermediate:

–
Introduction to parallel computing concepts

–
Concurrency strategies in programming languages

–
Parallel sorting algorithms



csinparallel.org

Module:
WebMapReduce



csinparallel.org

Module: MTL with
OpenMP

•
Intel’s Manycore Testing Lab


•
Module

–
#pragma omp parallel for num_threads(threadct)
\

shared (a, n, h, integral) private(i)


–
reduction(+: integral)




csinparallel.org

CSinParallel



•
We seek collaborators and contributors

csinparallel.org

Patterns Methodology

•
Kreutzer and Mattson OPL not only provides
a catalog of patterns, but also a

software problem
-
solving methodology


csinparallel.org

Patterns Methodology

•
Kreutzer and Mattson OPL not only provides
a catalog of patterns, but also a

software problem
-
solving methodology

•
Purposes:

–
Education

–
Communication

–
Design


csinparallel.org

How to get it taught

•
Pressures on the professor

–
“Oh no! Not another course to squeeze…”

•
So, take an
incremental spiral approach

(agreeing with NSF/TCPP)

–
Small changes in curriculum in many places

–
Revisit challenging issues

–
Students come to think of parallelism as natural
part of computation

–
Spiral approach is pedagogically effective




csinparallel.org

How to get it taught

Incentives

•
Microgrants
: small (e.g., $1500) amounts for
contributing first steps in teaching
parallelism

–
Intel Academic Community

(
intel.com/AcademicCommunity
)

–
Educational Alliance for a Parallel Future
(
eapf.org
)





NSF/TCPP Initiative

Early
Adopter Program

How to obtain Early Adopter Status?

•
16
Early adopters chosen
for Spring
term 2011

•
17 Early adopters chosen
for Fall term 2011



•
Next round of competition:


Fall 2012;
Deadline

November 5,
2011

–
NSF/Intel funded Stipend/Honorarium

–
Which
course(s
) , topics, evaluation plan?



How to obtain Early Adopter Status?

•
Instructors
for

–
core CS/CS courses
such as CS1/2, Systems, Data
Structures and Algorithms
–

department
-
wide
adoption preferred

–
elective courses
such as Algorithms, Architecture,
Programming Languages, Software
Engg
, etc.

–
introductory/advanced
PDC course

–
dept chairs, dept curriculum committee members
responsible


csinparallel.org

How to get it taught

Other supports needed

•
Platform availability

•
Support community

•
Educational elements

–
Learning objectives, assessment tools, etc.





csinparallel.org

Surviving in the wild ecosystem

–
Industry/Academia

–
Learning curve/Rapid change

–
Principles/Practices

–
Teaching/Research

csinparallel.org

Surviving in the wild ecosystem

–
Industry/Academia

–
Learning curve/Rapid change

–
Principles/Practices

–
Teaching/Research

•
Mine from the heritage of the past

•
Incremental approach

•
Spiral exposition

•
Pattern
-
based methods

csinparallel.org

An example


(go
-
lang.org
)


•
Mine from the heritage of the past

–
Hoare’s CSP; CCS


Pi Calculus [teach/research]

•
Incremental approach

–
Not far from C [academic/industry]

•
Spiral exposition

–
Midway steps towards explicit threads, message passing
[Learning curve/rapid change]

•
Pattern
-
based methods

–
Message passing
,
Fork
-
join
, channel as
Parallel Queue
¨

[Principles/Practice]

csinparallel.org

Questions?