Parallel computing - raghavendra's

compliantprotectiveΛογισμικό & κατασκευή λογ/κού

1 Δεκ 2013 (πριν από 4 χρόνια και 7 μήνες)

93 εμφανίσεις

Importance of parallel computing in today’s



Motivation for parallel computing

Why parallelism is essential

Is parallel computing really dead?

Factors affecting commercial sales of

computer systems

Steps for the development of parallel computing

Applications of parallel computing


Parallel computing:

Parallel computing is the concurrent use of multiple processors (CPUs) to do
computational work. In tradition
al (serial) programming, a single processor executes the
program instructions in a step
step manner. Some operations, however, have multiple
steps that do not have time dependencies and that can therefore be broken up into
multiple tasks to be executed
simultaneously. For example, adding a number to all the
elements of a matrix does not require that the result obtained from summing one element
be acquired before summing the next element. Elements in the matrix can be made
available to several processors
and the sums performed simultaneously, with the results
available much more quickly than if the operations had all been performed serially.
Parallel computations can be performed on shared
memory systems with multiple CPUs,
or on distributed
memory cluster
s made up of smaller shared
memory systems or single
CPU systems. Coordinating the concurrent work of the multiple processors and
synchronizing the results are handled by program calls to parallel libraries, and usually
require parallel programming experti

Overview of parallel architectures and programming models

block diagram

of a "generic parallel computer" with p processors is shown below.
Let’s assume that each processor Proc_i executing some program, sometimes referring to
data stored in its own
memory Mem_i, and sometimes communicating with other
processors over the Interconnection Network. This picture is sufficiently abstract to
include the case where the entire parallel computer resides in one "processor" (as is the
case with the dual floating

point units in the RS6000/590) where the parallel computer
sits in one room (the IBM SP

which consists of multiple RS6000s

or most other
commercial parallel computers), or where the parallel computer consists of computers
spread across continents,

and the interconnection network is the Internet, telephone
system, or some other network.

There are three standard functions any parallel archi
tecture must provide.



or getting processor to work simultaneously.


Interprocessor communication

or having processors exchange information.


, such as getting processors to ``agree'' on the value of a
variable, or a time at whi
ch to start or stop.


The two main styles of


, or single
data and multiple
data. This is old terminology, dating to

of parallel machines in the 1960s. SIMD

means performing the same operation
(e.g. single instruction) on multiple pieces of data in parallel, and MIMD means
performing arbitrary operations (e.g. multiple instructions) on different data at the same
time. In the SIMD case, one can think of the ge
neric parallel processor in the figure
above as being augmented by another
control processor
, which at every cycle sends a
common instruction to each Proc_i to execute. Examples of SIMD parallelism include the
vector operations on a single processor Cray T
90 and earlier Cray vector machines; the
Thinking Machines CM
2; and a single pipelined floating point unit in the RS6000/590,
where the adder and multiplier must operate in a pipeline controlled by a single fused
add instruction. Examples of MIM
D parallelism include almost every
commercial parallel machine, where each processor may also be programmed as a
standard sequential machine, running sequential jobs. Note that a particular machine can
exhibit both SIMD and MIMD parallelism at different le
vels, such as a multiprocessor
Cray T90. MIMD parallel is more flexible and more common than SIMD parallelism,
which now usually appears within individual floating point or memory units.

Motivation for Parallel Computing

The traditional scientific paradig
m is first to do theory (say on paper), and then lab
experiments to confirm or deny the theory. The traditional engineering paradigm is first
to do a design (say on paper), and then build a laboratory prototype. Both paradigms are
being replacing by numeri
cal experiments and numerical prototyping. There are several
reasons for this.

Real phenomena are too complicated to model on paper (eg. climate prediction).

Real experiments are too hard, too expensive, too slow, or too dangerous for a

Laboratory (e.g. oil reservoir simulation, large wind tunnels, overall aircraft

Design, galactic evolution, whole factory or product life cycle design and

Optimization, etc...).

Scientific and engineering problems requi
ring the most computing power to simulate are
commonly called "Grand Challenges, like predicting the climate 50 years hence, are
estimated to require computers computing at the rate of 1 Tflop = 1 Teraflop = 10^12
floating point operations per second, and
with a memory size of 1 TB = 1 Terabyte =
10^12 bytes.:

NASA is putting up weather satellites expected to collect 1TB/day for a period of years,
totaling as much as 6 PB of data, which no existing system is large enough to store. The
Sequoia 2000 Global C
hange Research Project is concerned with building this database.

Why parallelism is essential

The speed of light is an intrinsic limitation to the speed of computers. Suppose we wanted
to build a completely sequential computer with 1 TB of memory running
at 1 Tflop. If the
data has to travel a distance r to get from the memory to the CPU, and it has to travel this
distance 10^12 times per second at the speed of light c=3e8 m/s, then r <= c/10^12 = .3
mm. So the computer has to fit into a box .3 mm on a sid

Now consider the 1TB memory. Memory is conventionally built as a planar grid of bits,
in our case say a 10^6 by 10^6 grid of words. If this grid is .3mm by .3mm, then one
word occupies about 3 Angstroms by 3 Angstroms, or the size of a small atom. It i
s hard
to imagine where the wires would go!

Is parallel computing really dead?

At the very least, it is undergoing a major transition. With the end of the cold war, there
is less funding for defense
oriented supercomputing, which has been the traditional
mainstay of the high
end market. If parallel computing is to survive in the new
environment, a much larger fraction of sales must be to industry, which seems to be
substantially less concerned with high
end performance.

Factors affecting commercial sales
of parallel computer

Two factors bear on the size of the industrial market for parallel computing. First, most
engineering firms have recently made the transition away from mainframes to
workstations. These companies believe that if they need more

computational power than
they have on a single workstation, they should be able to get it by using a network of
such machines. Whether or not this is true, it has substantially affected sales of tightly
coupled parallel systems and must be taken into acco
unt when analyzing the needs of
industry users.

A second factor affecting commercial sales of parallel computer systems has been the
reluctance of independent software vendors like MacNeal

Schwendler to move their
applications to parallel machines. I bel
ieve the primary reason for this reluctance has
been the absence of an industry standard interface that supports machine
parallel programming. Without such a standard, a software vendor's investment in
conversion to parallel machines is not pro
when a new parallel computing
architecture with a new programming interface emerges, the application would need to be

Steps for the development of parallel computing:

So is parallel computing on its last legs? Although good news has b
een very limited over
the past few months, there are several reasons why it is too soon to give up:

First, no matter how powerful the workstation processor is, if it is possible to
write programs that scale from one processor to thousands, there will be p
lenty of
applications that can take advantage of the additional computational power due to
parallelism. Even high
end workstation companies acknowledge this by providing
multiprocessors and support for clusters.

Second, the software and algorithms problem
s that must be solved for scalable
parallel systems to be usable (e.g., development of scalable parallel programming
interfaces and standard programming languages) also need to be solved for
networks of workstations if they are to succeed as high
end compu
ting engines.

Finally, I believe independent software vendors will adopt parallelism when
parallel programming interfaces become industry standards. Recently, J.S. Nolan
and Associates, a vendor of reservoir analysis codes, has undertaken a major
to implement their very popular code VIP in PVM. This has been made
possible by the portability of PVM, which has yielded implementations on most
common parallel computing systems, including networks of workstations.


Applications are classi
fied into 5 problem classes:

Synchronous Applications

Such applications tend to be regular and characterized by algorithms employing
simultaneous identical updates to a set of points

Loosely Synchronous Applications

Such applications are iterative o
r time
stepped but unlike the synchronous case,
employ different evolution (update) procedures which synchronize

Embarrassingly Parallel Applications:

Such applications can employ complex algorithms but can be parallelized because
the evo
lution of different points is largely independent

Asynchronous Problems:

These are hard to parallelize problems with no natural algorithmic synchronization
between the evolutions of linked irregular data points.


Such problems are

hybrid integration of several sub problems of the other four
basic application classes.


From the outset, we have known that a lot of work was needed to make scalable parallel
computing truly useful. Projects at the Center for Research on Para
llel Computation and
throughout the computational science community are now beginning to bear fruit.
Standards like High Performance Fortran and Message Passing Interface, along with
portable systems like PVM, are beginning to make parallel computing more
palatable to
commercial firms.

So we assert that parallel computing is not dead
it is simply suffering through a period
of growing pains. We are coming to realize that
parallel computing is really a software


Therefore, we should not be too distr
essed when a company drops out of the
crowded hardware market. The industry needs software and standards to make writing
parallel applications easier. If it gets them, the whole industry will grow, creating more
business for everyone.


Internet & Journals of Ken Kennedy;





http:/Nan’s parallel page

Presented By:


& PBK.Prashanth

II year;

Bandari Srinivas Institute of Technology;