Institute for computational modeling in civil engineering

Platzhalter fr Bild, Bild auf Titelfolie hinter das Logo einsetzen

A multi-core implementation of the lattice Boltzmann method for non-uniform grids

Kostyantyn Kucher, Sren Freudiger, Martin Geier, Maik Stiebler, Manfred Krafczyk 16.03.2010

The multi-scale lattice Boltzmann equation

The lattice Boltzmann equation

BGK (Bhatnagar-Gross-Krook) - relaxation term

D2Q9 model
16.03.2010 | Kostyantyn Kucher | Page 2

Qian, dHumires, Lallemand 1992

Institute for computational modeling in civil engineering

The multi-scale lattice Boltzmann equation

Interpolation cells
Geier, Greiner, Korvink, 2009
interpolation cell (coarse fine)

coarse

interpolation cell (fine coarse)

fine
16.03.2010 | Kostyantyn Kucher | Page 3

Institute for computational modeling in civil engineering

Geometric Partitioning of the Simulation Domain

domain block

patch2

patch1
lattice node

16.03.2010 | Kostyantyn Kucher | Page 4

Institute for computational modeling in civil engineering

Implementation
Synchronization
coarseP:SpTbPatch g:Synchronizer calculate() calculate()
exchangeBlockData() Tf

fineP1:SpTbPatch

n:Synchronizer

fineP2:SpTbPatch

calculate()
exchangeBlockData()

wait()
Tc

calculate()
exchangeBlockData() exchangeBlockData() Tf

wait() calculate()

wait() wait() interpolation()

exchangeBlockData()

wait()

wait() wait()

interpolation()

16.03.2010 | Kostyantyn Kucher | Page 5

Institute for computational modeling in civil engineering

Validation
The flow past a cylinder
Schfer und Turek 1996

U=V=0 4.2r inlet 4.0r 1.95r 4.0r 40.0r U=V=0 1.95r U=V=0 fine grid 128 x 688 lattice nodes 256 x 1376 lattice nodes 512 x 2752 lattice nodes coarse grid 128 x 688 lattice nodes 256 x 1376 lattice nodes 512 x 2752 lattice nodes outlet

16.03.2010 | Kostyantyn Kucher | Page 6

Institute for computational modeling in civil engineering

Validation
Re = 20
Parameter
resolution 128 x 688 Cd Cl 5.5972526 0.0042797 5.5807648 0.0073446

SBB

QBB

resolution 256 x 1376

Cd Cl resolution 512 x 2752 5.5870459 0.0162391 5.5731531 0.0101338

Cd
Cl Parameter Cd Cl
16.03.2010 | Kostyantyn Kucher | Page 7

5.5778068
0.0107031 Crouse 5.585-5.627 0.017-0.0119

5.5745483
0.0102097 Schfer and Turek 5.57-5.59 0.0104-0.011

Institute for computational modeling in civil engineering

Domain Decomposition for Load Balancing

METIS - Serial Graph Partitioning and Fill-reducing Matrix Ordering Karypis, Kumar 1998 Patch1 : 2 Threads

x = 1.0 t = 1.0

Patch2 : 6 Threads x = 0.5 t = 0.5

16.03.2010 | Kostyantyn Kucher | Page 8

Institute for computational modeling in civil engineering

Performance Analysis
Parallel Efficiency

100 90 80 70 60 50 40 30 20 10 0
0

coarse

parallel efficiency [%]

fine

Intel Xeon E5520 2.26GHz (uniform)

Intel Xeon E5520 2.26GHz (non-uniform) Intel Core2 Quad Q8200 2.38GHz (uniform) Intel Core2 Quad Q8200 2.38GHz (non-uniform)

number of threads

10

12

14

16

16.03.2010 | Kostyantyn Kucher | Page 9

Institute for computational modeling in civil engineering

Performance Analysis
Scale up

16
scaleup * number of threads

Intel Xeon E5520 2.26GHz (uniform) Intel Xeon E5520 2.26GHz (non-uniform)

14 12 10 8 6 4 2 0 0 2

Intel Core2 Quad Q8200 2.38GHz (uniform)

Intel Core2 Quad Q8200 2.38GHz (non-uniform) Linear Scaling

coarse fine

10

12

14

16

number of threads
16.03.2010 | Kostyantyn Kucher | Page 10

Institute for computational modeling in civil engineering

Performance Analysis
Limitation by Memory Bandwidth:
120 100 29.9 80 60 40 20 0 Intel Core2 Quad Intel Core2 Quad 2xIntel Xeon 2xIntel Xeon Q8200 2.38GHz, Q8200 2.38GHz, E5520 2.26GHz, E5520 2.26GHz, DDR2-6400 DDR2-6400 DDR3-1066 DDR3-1066 4 threads 4 threads 8 threads 2 threads 53.7 75.2 52.4 73.6 40.6 44.8 35.2 FLOPS peak performance, % Load of BW, %

16.03.2010 | Kostyantyn Kucher | Page 11

Institute for computational modeling in civil engineering

Outlook
Support of MPI
Support of varying hierarchic setups

Non-uniform Computational Fluid Dynamics in 3D

Dynamic load balancing

16.03.2010 | Kostyantyn Kucher | Page 12

Institute for computational modeling in civil engineering

Acknowledgments
German BMBF funding the SKALB (Lattice-Boltzmann-Methoden fr skalierbare Multi-Physik-Anwendungen) project (reference ID 01IH08003E).

Funding by the DFG under grant GE 1990/2-1 (Konsistente Multiskalenstrmungsmechanik mit der kaskadierten Lattice Boltzmann Methode).

16.03.2010 | Kostyantyn Kucher | Page 13

Institute for computational modeling in civil engineering