Schedule (1)

„ Classification of Partial Differential

Computational Fluid Dynamics Equations
„ Discretization in space and time
„ Finite Difference Method
Takayuki Aoki
„ Diffusion Equation
Global Scientific Information and Computing Center
Tokyo Institute of Technology „ Advection Equation
„ Poisson Equation
2

In XXXX, Manhattan, New York

Schedule (2)
„ 1-D and 2-D Burgers Equations
„ Shallow Water Equation
„ Incompressible Navier-Stokes
Equation
„ Visualization

3
 NATURAL AND HUMAN-MADE
GREENHOUSE EFFECT CYCLE OF CARBON

ppm
CO2 CONCENTRATION //ppm
Human-
Human-made CO2
Infra-red thermal energy from Calculation by unmitigated scenario
the ground is partially reflected Ice cores observation
by the glass, and some is 25%
Visible energy from the sun Absorbed in
passes through the glass trapped inside the greenhouse. the vegetation 47%
47%
Mauna Loa observation
and heat the inside. remain in the
atmosphere
28%
Absorbed in
the ocean

CO2 transfers in GtC natural and human-made

1.5 5.5 ATMOSPHERE 3

2 2
90 90
60 60
Vinyl house
OCEAN
LAND
Source: IPCC

Fastest Supercomputer
in the world
Environmental Problems No.1 Earth Simulator Japan (Jamstec
(Jamstec)) (NEC)
In 2002 June
35.86 TFlops
No.2 ASCI White USA (LLNL) (IBM Power 3) 7.23 TFlops
No.3 Alpha Server USA (Pittsburg SC) (Alpha) 4.46 TFlops

Human activity Natural Phenomena

■ Climate Change ■ Meteorological

global warming Disasters
Heavy rain, Drought
■ Air pollution
Acid rain
■ Diastrophism
Ozone hole Earthquake
Volcanism
 Latest Top500
2005 November TSUBAME Grid Cluster
No.1 BlueGene/L
BlueGene/L USA (DOE) (IBM) 280.60 TFlops • Tokyo-tech
No.2 BlueGene/L
BlueGene/L USA (IBM Watson) (IBM) 91.29 TFlops
No.3 ASCI -Purple USA (DOE) (IBM) 63.39 TFlops • Supercomputer and
No.4 Colombia USA (NASA) (SGI) 51.87 TFlops • UBiquitously
No.5 Thunderbird USA (Sandia) (Dell) 38.27 TFlops
No.6 Red Storm d USA (Sandia) (Cray) 36.19 TFlops • Accessible
No.7 Earth Simulator Japan (Jamstec
(Jamstec)) (NEC) 35.86 TFlops
No.8 MareNostrum Spain (Balserona
(Balserona)) (IBM) 27.91 TFlops • Mass-storage
• Environment
BlueGene/L
BlueGene/L - eServer (IBM) Power 4 × 131,072 CPU TSUBAME means swallow in Japanese,
Tokyo-tech symbol is swallow.

TSUBAME configuration details

SunFire (Galaxy4) Currently 85 Teraflops

655nodes 80+ cabinets ~900KW Power (1.2MW w/cooling)
16CPU/node
10480CPU/50TFlops (Peak) Planned ClearSpeed
extension to 655 nodes, (+ a Small SX-8i)
Memory: 21.4TB
> 100 TeraFlops (50 Scalar + 60 SIMD-Vector
ClearSpeed CSX600 Approx. 140,000 execution units,
Initially 360nodes
96GFlops/Node
35TFlops (Peak) DDR Capable Infiniband card, dual
lane per node (20Gbps current,
Titech TSUBAME
40Gbps with DDR capable switch) ~80+ racks
InfiniBand Network Voltaire ISR 9288 x 6
350m2 floor area
2-1

1400Gbps Unified & 1.2 MW (peak)

Redundunt Interconnect
200+200Gbps
bidirectional
24+24Gbps
External bidirectional
10Gbps Switch
Fabric
・・・
42 units FileServer FileServer
External
External 500GB 500GB 500GB
Grid
Grid
48disks 48disks 48disks
C C
Connectivity
Connectivity
SX-8 Storage Server A
Sun Storage (TBA), all HDD Storage B
NEC iStorage S1800AT
Physical Capacity 1PB, 40GB/s Phys. Capacity 96TB RAID6
All HDD, Ultra Reliable
Total 1.1PB
 Local Infiniband Switch
TSUBAME is #7 Fastest in World
(288 ports) Linpack is used as a performance measure in ranking
T/V N NB P Q Time Gflops
Node Rear Currently
-----------------------------------------------------------------
2GB/s / node WR10R2R4 1334160 240 36 144 41462.22 3.818e+04
Easily scalable to -----------------------------------------------------------------
8GB/s / node • N : 1334160
• NB : 240
• PMAP : Row-major process map
• P : 36
• Q : 144
• PFACT : Right
• NBMIN : 4
• NDIV : 2
• RFACT : Right
• BCAST : 1ring
Cooling Towers (~20 units) • DEPTH : 1
• SWAP : Mix (threshold = 240)
• L1 : transposed form
• U : transposed form
• EQUIL : yes
~500 TB out of 1.1PB • ALIGN : 8 DP words

38.18 TeraFlops
• Opteron Only, 648
nodes
• ~14 Terabytes memory
• 76.56% Efficiency

High-resolution Cloud Physics Process

Typhoon Simulation
Bulk method of cold rain.
CReSS: Cloud Resolving Storm Simulator Prognostic variables for mixing ratios :
Non-hydrostatic and compressible equation
„ water vapor
Terrain-following in three dimensional geometry „ cloud water
„ rain water
Prognostic variables:
„ cloud ice
„ three-
three-dimensional velocity components „ snow
„ perturbation of pressure „ graupel
„ perturbation of potential temperature
„ subgrid-
subgrid-scale turbulent kinetic energy, TKE
„ mixing ratios for water vapor and several hydrometeors
 Improved Dynamic Process Typhoon Simulation
#18-
#18-typhoon Sep 2004, Sever Damage in Kyushu

∂ ρu ⎛ ∂u ∂u ∂u ⎞ ∂p '
+ ρ ⎜⎜ u + v + w ⎟⎟ = − + ρ ( f s v − f c w) + Turb.u
∂t ⎝ ∂x ∂y ∂z ⎠ ∂x

4thFDM+Leap-
4thFDM+Leap-flog method Semi-
Semi-Lagrangian Scheme
(with Asselin filter) Cubic-
Cubic-Lagrange, CIP Method

Domain H: 1000 km × 1000 km × V: 18 km

H-gird size 1000 m
V-grid size 200 ~ 300 m (stretched)
Grid numbers H: 1003 × 1003× V: 63
Calculation Integration time 48 hours
Conditions: Micro-
Micro-physics the bulk cold rain type
Initial condition JMA Regional Spectral model output
Boundary JMA Regional Spectral model output
Surface real topography and observed SST
CFL = 0.15 CFL= 0.6 Earth Simulator 100 nodes (800 CPU)

Explosion Simulation

CH3 CH3

O2N NO2

3HONO2 3H2O

NO2

1kg-TNT explosion : 3.22 MJ

30kg-TNT explosion : 100 MJ

Dangerous and difficult

experiment
Numerical Simulation
 400 m
Blast wave just after explosion 5-ton TNT
in Cylindrical 2-D geometry
Shock Propagation on
Complex Terrain

Shock pressure front

Pressure history
on the ground surface

32kg-TNT EXPLOSION
ON THE GROUND
 Comparison with Experiment
(a) (b)
32kg-TNT Explosion Experiment
(a)
(b)
Safety test for 32kg-
32kg-TNT
Explosion on the ground.
(c)
In 2002, December 3,
In Ooita Pref., Hijuu-
Hijuu-dai,
dai, at Japan
Ground Self-
Self-Defense Force,
(d) (c) (d) The experiment was done by the
Research Center for Explosion
and Safety, National Institute of
Advanced Industrial Science and
Pentolite 100g Technology.

Recorded by High Speed Camera.

Blood Flow Simulation System

Extraction of Blood Data Compression
Clinical MRI CFD Visualization
vessel structure
call ido(u(j),u(j+1),u(j-),ux(j)
ux(j+1),ux(j-1),dp, dm,
cuxx, cuxxx)
call ido(p(j), p(j+1), p(j-1),
px(j), px(j+1), px(j-1),
and Animation
MRI Facility
dp,dm,cpxx,cpxxx)
cqxxx = 0.0
pt = - u(j)*px(j) - g(j) *p(j)
ptx = - ux(j)*px(j) - u(j)*pxx

Division of Medical Imaging,

Network Collaboration : National Institute of Radiological Sciences,
Ikehira Lab.

Gyroscan Intera
1.5T Philips
Head MRI Slice

4-D MRI processing synchronizing

electrocardiograph
 Extraction of Blood Vessel Structure from Voxel data
Blood Flow With Complex Branches
Pressure Profile

9.31% 6.12% 6.11%

Flow Velocities

2.82%
1.88%
3.42%

11.44%
7.86% 13.76%

71.92% 19.53%
55.56%
82.60%
32.60%
4.69%
100.0%
5.67% 25.41%
14.10%
3.24% 4.59%

9.98% 9.45%
Flow branch ratio

Bubbly Flow Simulation Computational Methods

„ Channel Width (H) 10 mm g
„ Channel Length (L) 250 mm „ Gas – Liquid Unified Solver : CIP (CCUP) method
„ Mesh Size : 0.25 mm
„ Mesh Number : 40× 40×40×
40×1000
„ 3-dimentional Compressible / Incompressible fluid
drier
„ Equal grid spacing „ Surface Tension : CFS model
„ Incoming Flow Velocity : 0.5 m/sec
Wall Boundary condition: non-
non-slip
„ Contact angle between wall and bubbles
18 m

„ separator
L (250mm)

„ Periodic in the gravity direction „ Surface tracking method : improved VOF method
„ Room Temperature
„ No thermal process
∂ρl φl Dui 1 ∂p 1 ∂τ ij
+ ∇ ⋅ (ρl φl u ) = 0
core
„ Initial Condition =− + + gi + σ i
－Average Void Ratio : 0.1 ∂t Dt ρ ∂xi ρ ∂x j
control rod
∂ρ g φ g
+ ∇ ⋅ (ρ g φ g u ) = 0
－Diameter of bubble : 2 mm
FLOW

－Number of bubbles : 594 De p ∂ui 1 ∂ ⎛ ∂T ⎞

φ8m ∂t =− + ⎜λ ⎟+q
（=66 stages X 9） 9） Dt ρ ∂xi ρ ∂xi ⎜⎝ ∂xi ⎟⎠
ρ = ρl φl + ρ g φ g φl + φ g = 1
BWR
H
 Ray Tracing Visualization
Free Software
Light
POV-Ray 3.6
Tracing light ray correctly
and reproducing optics.

Screen
Reflection Refraction
View Point

Void Ratio data Voxel DF3 format

Sloshing
Navigating Ship Simulation
Computational Ship Hydrodynamics
Moving Body interacting ocean flow, free surface

Prof. Takehiro Himeno (Univ. of Tokyo)

Non-linear wave

Prof. F. Xiao
(Tokyo Tech)

NMRI Japan
Dr. Takizawa
 Some techniques
Simulation for InterGrid (Cut Cell) Overset grid
Falling Leaves Two values at the same position,
representing the
front and the rear
surface pressure.
Major difficulties:
● Fluid-
Fluid-structure interaction
● Complex shape of leaves
● Very thin structure

Shape of the leaf :

modeled by
Geometry data
200 polygons
DXF or STL CAD format

Computational Mother Domain: 50x50x80

Computational Sub-
Sub-Domain: 40x40x30

DO Experiment

A Preliminary Simulation:

A Falling Piece of Paper

for example, a Name Card
 Mechanism of Flattering 3D Visualization
■ Start to fall with gravity acceleration
High
Pressure
■ Transverse acceleration by the lift
force due to an attack angle

■ Deceleration by the drag due to

pressure and friction

■ Torque Generation
■ Inverse attack angle High
Pressure

■ Transverse acceleration in the

inverse direction
 Anaglyph Particle Simulation for Disasters
#2
Structure destruction by Landslide, Boulder flow,
Crater formation by a meteorite impact

Fluid like behavior of

Granular

Structure Destruction
（Continuous model is not applicable）
applicable）
・Crack growth
・Amputation
・Shear
・Smashing

eＤＥＭ Compression Test

（Extended Discrete Element Method）
of Concrete Structure
Model for Granular (Sand, etc)

Model for Structure (building)

tangential Check for Destruction model
component

Discrete
normal elements
component

Shear Band (Dilatancy

(Dilatancy))
Pore
Material

pore-
pore-spring
 Visual Science Festa 2002
Grand Prix

Rock Fall Simulation Rock Fall Simulation

Digital Elevation Data
Rock Fall : One of slope disasters

Prevention Sakurajima
1000 Rocks

Control of Sediments
exp : fence, wall, net 1100m 200m
Improvement of Hazard Map

Numerical Simulation :
effective tool

400m
Aug 25, 1997
 Non-Spherical Rock Model Rock Fall Simulation
DEM Discrete Element Method Digital Elevation Data

component
tangential
Sakurajima
1000 Rocks

1100m 200m

slider
Summation for all the particles normal
damper

damper
spring
component

Translational and
Rotational motion of rocks
Quaternion
% =Q +Qi +Q j+Q k spring 400m
Q ss tt uu vv

Guard Fence for Rock Fall

Digital Elevation Data

Sakurajima
1000 Rocks

1100m 200m

Slit dam

400m