MPI & OpenMP Mixed

Hybrid Programming
Berk ONAT
!T Bili"im Enstits
22 Haziran 2012
Outline
22.06.2012 2/36
- lnLroducuon
- Share & ulsLrlbuLed Memory rogrammlng
- Ml & CpenM AdvanLages/ulsadvanLages
- Ml vs. CpenM
- Why Mlxed/Pybrld rogrammlng?
- Ml Calls
- LlmlLauons and roblems
- Mlxed/Pybrld rogrammlng Model
- Labs: Lxamples
Hybrid Programming:
OpenMP + MPI & Apps.
Shared & Distributed Programming
22.06.2012 3/36
- Shared Memory (Lhread based)
- Muluple Lhreads sharlng daLa ln Lhe same address
space and expllclLy synchronlzlng when needed
- ulsLrlbuLed Memory (process based)
- ulsuncL processes, expllclLly parLaklng ln Lhe
palrwlse and collecuve exchange of conLrol and daLa
messages
- no way Lo dlrecLly access Lhe varlables ln Lhe
memory of anoLher process
Hybrid Programming:
OpenMP + MPI & Apps.
MPI Case
22.06.2012 4/36
- AdvanLages
- orLable Lo dlsLrlbuLed and shared memory machlnes
- Scales beyond one node
- no daLa placemenL problem
- ulsadvanLages
- ulmculL Lo develop and debug
- Plgh laLency, low bandwldLh
- LxpllclL communlcauon
- Large granularlLy
- ulmculL load balanclng (dynamlc/sLauc)
Hybrid Programming:
OpenMP + MPI & Apps.
OpenMP Case
22.06.2012 3/36
- AdvanLages
- Lasy Lo lmplemenL parallellsm
- Low laLency, hlgh bandwldLh
- lmpllclL Communlcauon
- Coarse and ne granularlLy
- uynamlc load balanclng
- ulsadvanLages
- Cnly on shared memory machlnes
- Scale wlLhln one node
- osslble daLa placemenL problem
- no speclc Lhread order
Hybrid Programming:
OpenMP + MPI & Apps.
MPI vs. OpenMP Case
22.06.2012 6/36
- Ml and CpenM are boLh sulLable for coarse graln
parallellsm (muluple asynchronous processor)
- CpenM can be eecuvely used for ne graln
parallellsm (vecLorlzauon)
- 8oLh Ml and CpenM can be used Lo parallellze
appllcauons ln daLa parallel and Lask parallel fashlon
- Lven Lhough CpenM ls based upon sharlng ouL work, lL
ls posslble Lo asslgn daLa Lo lndlvldual Lhreads
- WlLh some care when asslgnlng work, a daLa parallel
mode can be closely approxlmaLed ln CpenM
Hybrid Programming:
OpenMP + MPI & Apps.
Why Hybrid Programming?
22.06.2012 7/36
- Pybrld model ls an excellenL maLch for Lhe
domlnanL Lrend ln parallel archlLecLures whlch
are made of clusLers of mulu-core shared
memory or SM (SymmeLrlc Mulu-rocessor)
nodes. Such as quad or dual core xeon
rocessor
- Avold exLra communlcauon overhead wlLh Ml
wlLhln node
- Could have beuer scalablllLy boLh pure Ml
and CpenM
Hybrid Programming:
OpenMP + MPI & Apps.
Why Hybrid Programming?
22.06.2012 8/36
- AnAuCLu, kA8AuLnlZ, LCL, .
- lnLel xeon LM641 arch.
- dual & quad core nodes (as smp)
- lronL Slde 8us !
Hybrid Programming:
OpenMP + MPI & Apps.
Hybrid Programming
22.06.2012 9/36
- Muluple CpenM Lhreads under
each Ml process
- CpenM Lhreads can be used wlLhln
each shared memory node and Ml can
be used Lo communlcaLe across nodes
- LllmlnaLes message passlng wlLhln a
slngle shared memory node
- nesLed parallellsm ls posslble ln a hybrld
model
- 8lghL approach for uSM archlLecLures
comprlslng of a large number of shared
memory SM nodes
Hybrid Programming:
OpenMP + MPI & Apps.
Hybrid Programming Models
22.06.2012 10/36
!"#$%& ()
*++,%-./01
h
u
p
:
/
/
w
w
w
.
c
o
m
p
u
n
l
L
y
.
o
r
g
/
e
v
e
n
L
s
/
e
w
o
m
p
0
3
/
o
m
p
L
a
l
k
s
/
1
u
e
s
d
a
y
/
S
e
s
s
l
o
n
7
/
1
0
1
p
.
p
d
f

Hybrid Programming:
OpenMP + MPI & Apps.
MPI Routines for Threads
22.06.2012 11/36
- Ml_lnl1_1P8LAu
- lL allows Lo requesL a level of Lhread supporL
- 8LCul8Lu argumenL can be one of Lhe Ml_1P8LAu_[SlnCLL
| lunnLLLu | SL8lALlZLu | MuL1lLL]
- 8eLurned 8CvluLu argumenL may be less Lhan 8LCul8Lu by
Lhe appllcauon
MPI_INIT_THREAD(INTEGER REQUIRED, INTEGER PROVIDED, INTEGER IERROR)
int MPI_Init_thread(int *argc, char **argv,
int required, int *provided);
C
lorLran
Hybrid Programming:
OpenMP + MPI & Apps.
MPI Routines for Threads
22.06.2012 12/36
- Ml_lnl1_1P8LAu
- 1he slmplesL CpenM+Ml model ls Ml_1P8LAu_SlnCLL
- Ml_lnl1 could be equal Lo Ml_lnl1_1P8LAu wlLh requlred
Ml_1P8LAu_SlnCLL ln M sysLems and
Ml_1P8LAu_lunnLLLu ln hybrld sysLems (nLC Sx-3, Sx-6,
Sx-8)
Hybrid Programming:
OpenMP + MPI & Apps.
MPI Routines for Threads
22.06.2012 13/36
- Ml_lnl1_1P8LAu
- 2()34!56*738)9:;6< 1here ls only one Lhread ln appllcauon.
1here ls no CpenM muluLhreadlng ln program.
- 2()34!56*73=>996;67< 1here ls only one Lhread LhaL
makes Ml calls. All of Lhe Ml calls are made by Lhe masLer
Lhread. 1hls wlll happen lf all Ml calls are ouLslde CpenM
parallel reglons or are ln masLer reglons.
Hybrid Programming:
OpenMP + MPI & Apps.
MPI Routines for Threads
22.06.2012 14/36
- Ml_lnl1_1P8LAu
- 2()34!56*73865)*;)?67< Muluple Lhreads make Ml calls,
buL only one aL a ume. 1hls can be enforced ln CpenM by
CpenM SlnCLL dlrecuve.
- 2()34!56*732>;4)(;6< Any Lhread may make Ml calls aL
any ume.
- All Ml lmplemenLauons of course supporL Ml 1P8LAu
SlnCLL.
See for deLall lnformauon: hup://www.mcs.anl.gov/~lusk/hybrldpaper.pdf
Hybrid Programming:
OpenMP + MPI & Apps.
MPI Routines for Threads
22.06.2012 13/36
- Ml Calls lnslde CM_MAS1L8
- Ml_1P8LAu_lunnLLLu requlred
- SupporL also Ml ouLslde parallel reglon
- CM_8A88lL8 ls needed slnce Lhere ls no
synchronlzauon wlLh CM_MAS1L8
- lL lmplles all oLher Lhreads are sleeplng!
!$OMP BARRIER
!$OMP MASTER
call MPI_xxx()
!$OMP END MASTER
!$OMP BARRIER
(SLraLegy: ln a Lhreaded barrler, masLer process synchronlzes processes)
Hybrid Programming:
OpenMP + MPI & Apps.
MPI Routines for Threads
22.06.2012 16/36
- Ml Calls lnslde CM_MAS1L8 .
hup://www.hlrs.de/
organlzauon/amL/
servlces/models/
openmp
Hybrid Programming:
OpenMP + MPI & Apps.
MPI Routines for Threads
22.06.2012 17/36
- Ml Calls lnslde CM_SlnCLL | MAS1L8
- Ml_1P8LAu_SL8lALlZLu requlred
- CM_8A88lL8 ls needed slnce CM_SL8lAL only guaranues
synchronlzauon aL Lhe end
- lL also lmplles all oLher Lhreads are sleeplng!
!$OMP BARRIER
!$OMP SINGLE
call MPI_xxx()
!$OMP END SINGLE
Hybrid Programming:
OpenMP + MPI & Apps.
MPI Routines for Threads
22.06.2012 18/36
- Cverlap communlcauon and compuLauon
- AL leasL Ml_1P8LAu_lunnLLLu requlred. Whlle masLer or
slngle Lhread ls maklng Ml calls, oLher Lhreads are compuung
!$OMP PARALLEL
if (my_thread_rank < 1) then
call MPI_xxx()
else
do some computation
endif
!$OMP END PARALLEL
Hybrid Programming:
OpenMP + MPI & Apps.
MPI Routines for Threads
22.06.2012 19/36
- Cverlap communlcauon and compuLauon
- AL leasL Ml_1P8LAu_lunnLLLu requlred. Whlle masLer or
slngle Lhread ls maklng Ml calls, oLher Lhreads are compuung
!$OMP PARALLEL
if (my_thread_rank < 1) then
call MPI_xxx()
else
do some computation
endif
!$OMP END PARALLEL
Hybrid Programming:
OpenMP + MPI & Apps.
Advantages & Disadvantages
22.06.2012 20/36
- 1P8LAu_SlnCLL
- Lasy Lo program
- oLher Lhreads are sleeplng whlle masLer Lhread calls Ml
rouunes
- 1P8LAu_lunnLLLu
- load balance ls necessary
- useful for dynamlc Lask dlsLrlbuuon and problemauc for
domaln decomposluon programmlng
Hybrid Programming:
OpenMP + MPI & Apps.
Advantages & Disadvantages
22.06.2012 21/36
- 8esulLs
- 1here ls no opumal CpenM+Ml model
- depends on your appllcauon needs for communlcauon
- capablllLy for SM parallellzauon
- your avallable worklng hours for hybrld programmlng
- !" $%& '(! )*+,-,. $%,&-/ "-0&1
- 2,34 5%*6% 63/& 7*8& ! 6-8 6-77 '(! ,39:8&"1
- !8 5%*6% 63/& +736; ! 6-8 9"& <-,-77&7 ,&=*38"1
Hybrid Programming:
OpenMP + MPI & Apps.
Thread Info for MPI Process
22.06.2012 22/36
- Ml_CuL8?_1P8LAu
- 8eLurns provlded level of Lhread supporL
- 1hread level may be seL vla envlronmenL varlable!
- Ml_lS_1P8LAu_MAln
- 8eLurns Lrue lf Lhls ls Lhe Lhread LhaL lnvoked Ml_lnl1 or
Ml_lnl1_1P8LAu
- lndlcaLes LhaL whlch Lhread ls masLer
MPI_IS_THREAD_MAIN(LOGICAL FLAG, INTEGER IERROR)
Hybrid Programming:
OpenMP + MPI & Apps.
Limitations & Problems
22.06.2012 23/36
- CpenM has less scalablllLy due Lo lmpllclL
parallellsm whlle Ml allows mulu-
dlmenslonal blocklng.
- All Lhreads are ldle excepL one whlle Ml
communlcauon
- need Lo overlap compuLauon and
communlcauon for beuer performance.
- Crlucal Secuon necessary for shared varlables.
Hybrid Programming:
OpenMP + MPI & Apps.
Limitations & Problems
22.06.2012 24/36
- Cache coherence, daLa placemenL
- naLural one level parallellsm problems.
- ure CpenM code performs worse Lhan pure
Ml wlLhln node
- Lack of opumlzed CpenM compllers and
llbrarles
- noL necessarlly fasLer Lhan pure Ml or pure
CpenM
Hybrid Programming:
OpenMP + MPI & Apps.
Limitations & Problems
22.06.2012 23/36
- arLly depends on code, archlLecLure, parLly on
how Lhe programmlng models lnLeracL
- >35 4-8. :4&" *" - 8&5 $&-4 30 $%,&-/" 6,&-$&/1
- ?3&" 4-6%*8&@AB <9$ 8&5 $%,&-/" 5%&,& $%&. +&"$ ,&9"&
/-$- *8 6-6%&1
- >35 496% 30 $%& '(! 63/& *" 497:$%,&-/&/1
- >35 5&77 /3&" $%& 63/& 9"& $%& 6344C %-,/5-,&1
Hybrid Programming:
OpenMP + MPI & Apps.
Lab: Example 1
22.06.2012 26/36
- l Calculauon
- 1o demonsLraLe Lhe bouom Lo Lop approach ln
Ml/CpenM program developmenL, we
provlde a slmple program for Lhe calculauon of
Lhe n number uslng an easlly parallellzable
Lrapezoldal rule lnLegral evaluauon
- Ml (mpl_pl.c) / CpenM (omp_pl.c) and
Pybrld (hybrld_pl.c) verslons
dx
x
2
+1
=
!
4
0
1
"
Hybrid Programming:
OpenMP + MPI & Apps.
Lab: Example 1
22.06.2012 27/36
- )946; @2()A
8BC B1D%$01EB1C
F0E+%,B -0&B
source /RS/progs/intel/ict/3.0.1/mpi/3.0/bin64/mpivars.sh
mpiicc mpi_pi.c o mpi_pi.x
5G1 -0&B
mpirun np 8 ./mpi_pi.x
Hybrid Programming:
OpenMP + MPI & Apps.
Lab: Example 1
22.06.2012 28/36
- )946; @H+B12(A
8BC B1D%$01EB1C
F0E+%,B -0&B
export OMP_NUM_THREADS=8
icc openmp omp_pi.c o omp_pi.x
5G1 -0&B
./omp_pi.x
Hybrid Programming:
OpenMP + MPI & Apps.
Lab: Example 1
22.06.2012 29/36
- )946; @H+B12( I 2()A
8BC B1D%$01EB1C
F0E+%,B -0&B
export OMP_NUM_THREADS=4
source /RS/progs/intel/mpi/3.1/bin64/mpivars.sh
mpiicc openmp hybrid_pi.c o hybrid_pi.x
5G1 -0&B
mpirun np 2 hybrid_pi.x
J.$1%1K< lnLel Ml 3.0 has some bugs and glves error
messages when submlmng 'mlxed programed' codes.
6$$0$< sysLem error(22): __kmp_runume_desLroy:
pLhread_key_deleLe: lnvalld argumenL
CM aborL: faLal sysLem error deLecLed.
LMN
Hybrid Programming:
OpenMP + MPI & Apps.
Lab: Example 2
22.06.2012 30/36
- 2u PeaL Lquauon Calculauon
- WrlLe Pybrld Ml + CpenM program uslng Lhe
glven PeaL2u Ml program
- WrlLe correcL CpenM pragma llnes
- WrlLe Ml 1hread SLraLegy wlLh
Ml_1P8LAu_lunnLLLu
- WrlLe Pybrld code wlLh aL leasL one of Lhe
Ml_lnl1_1P8LAu
! MuL1lLL
! SL8lALlZLu
Hybrid Programming:
OpenMP + MPI & Apps.