ASCI
VIRTUAL PROTOTYPING OF SOLID
PROPELLANT ROCKETS
Researchers seek a detailed, whole-system simulation of solid propellant rockets under
normal and abnormal operating conditions. A virtual prototyping tool for solid propellant
rocket motors based on first principles models of rocket components and their dynamic
interactions meets this goal.
S
afety and reliability are paramount con-
cerns in rocket motor design because of
the enormous cost of typical payloads
and, in the case of the Space Shuttle and
other manned vehicles, for the crew’s safety. In
the spring of 1999, for example, a series of three
consecutive launch failures collectively cost more
than US$3.5 billion. The most notorious launch
failure, of course, was the tragic loss of the Space
Shuttle Challenger and its seven crew members.
Thus, there is ample motivation for improving
our understanding of solid rocket motors
(SRMs) and the materials and processes on
which they are based, as well as the methodol-
ogy for designing and manufacturing them.
The use of detailed computational simulation
in the virtual prototyping of products and de-
vices has heavily influenced some industries—
for example, in automobile and aircraft design—
but to date, it hasn’t made significant inroads in
rocket motor design. Reasons for this include the
market’s relatively small size and the lack of suf-
ficient computational capacity. Traditional de-
sign practices in the rocket industry primarily
1521-9615/00/$10.00 © 2000 IEEE
MICHAEL T. HEATH AND WILLIAM A. DICK
Center for Simulation of Advanced Rockets, UIUC
21
use top-down, often one-dimensional modeling
of components and systems based on gross ther-
momechanical and chemical properties, com-
bined with engineering judgement based on
many years of experience, rather than detailed,
bottom-up modeling from first principles.
Moreover, there has been a tendency to study in-
dividual components in isolation, with relatively
little emphasis on the often intimate coupling
between the various components. For example,
SPP1—an industry-standard code for analyzing
solid propulsion systems—includes a fairly de-
tailed model of the propellant thermochemistry,
but no structural analysis and no detailed model
of internal flow.
One of our primary goals at the Center for
Simulation of Advanced Rockets (CSAR) is to
develop a virtual prototyping tool for SRMs
based on detailed modeling and simulation of
their principal components and the dynamic in-
teractions among them. Given a design specifi-
cation—geometry, materials, and so on—we
hope to be able to predict the entire system’s re-
sulting collective behavior with sufficient fidelity
to determine both nominal performance char-
acteristics and potential weaknesses or failures.
Such a “response tool” could explore the space
of design parameters much more quickly,
cheaply, and safely than traditional build-and-
test methods. Of course, we must validate such a
COMPUTING IN SCIENCE & ENGINEERING
Figure 1. Ide- Solid propellant Interior cavity
Igniter Combustion interface
alized solid
propellant
rocket. Major
components
are indicated
in red, our ini- Combustion-injection
tial simulation boundary layer model
models in Compressible Thermo-mechanical
green. turbulence LES model foam model
Thermo-visco-elastic Thermo-elastic model
model
capability through rigorous and extensive com-
parison with data for known situations to have
confidence in its predictions for unknown situa-
tions. Although it is unlikely that simulation will
ever totally replace empirical methods, it can po-
tentially dramatically reduce the cost of those
methods by identifying the most promising ap-
proaches before building actual hardware.
Challenges in rocket simulation
Solid propellant boosters are the “heavy
lifters” of the space launch industry. Most of the
world’s large, multistage launch vehicles—in-
cluding the Ariane, Delta, Titan, and Space
Shuttle—employ two or more SRBs in the ini-
tial stage to provide 80% or more of the im-
mense thrust needed to lift a payload in excess
of 10,000 pounds off the launch pad and propel
it the first few tens of miles above Earth. Beyond
this point, subsequent stages—typically liquid-
fueled—take over into orbit and beyond.
SRMs are notably simpler than liquid rocket
engines.2 The latter have far more moving parts
(pumps, valves, and so on) and require storing
and handling of liquids that might be cryogenic
or potentially hazardous. SRMs, though, have
almost no moving parts (often only a gimballed
nozzle for thrust vector control), and the com-
posite solid propellant (containing both fuel and
oxidizer) forms the combustion chamber. The
main disadvantage of SRMs is that once ignited,
combustion is essentially uncontrollable: the
propellant burns at maximum rate until ex-
hausted. Thus, solid motors are ideal for the ini-
tial stages of flight, when raw power is more im-
portant than finesse, and then liquid-propellant
rockets take over for the portions of flight re-
quiring more delicate maneuvering. Despite
their relative simplicity, SRMs are still fiendishly
complex in terms of the chemical and thermo-
mechanical processes that take place during fir-
22
Case Nozzle Exhaust
Insulation plume
Elasticity/ LES
ablation model model
ing, as well as the design and manufacturing
processes required to make them reliable, safe,
and effective.
Figure 1 shows a schematic drawing of a typi-
cal SRM—the major parts are indicated along
with the types of mathematical models that
might be used to represent them. Reality, of
course, is considerably more complex than this
2D picture, and we note the following major
challenges in achieving our goals.
• The complex behavior of SRMs requires
fully 3D modeling to capture the essential
physics adequately. Examples include the
combustion of composite energetic materi-
als; the turbulent, reactive, multiphase fluid
flows in the core and nozzle; the global
structural response of the propellant, case,
liner, and nozzle; and potential accident sce-
narios such as pressurized crack propaga-
tion, slag ejection, and propellant detona-
tion.
• The coupling between components is
strong and nonlinear. For example, the load-
ing due to fluid pressure deforms the solid
propellant, which changes the geometry of
the fluid flow, which in turn affects pressure,
and so on. Similarly, the burn rate increases
with pressure and vice versa.
• The geometry is complex and changes dy-
namically as the rocket consumes propel-
lant. The inclusion of slots and fins, which
forms a star-shaped cross-section, enhances
the amount of burning surface area. What-
ever its initial shape, the propellant surface
regresses at a pressure-dependent rate as the
propellant burns, and discrete representa-
tions of the solid and fluid components, as
well as the interface between them, must
adapt accordingly.
• The spatial and temporal scales are ex-
tremely diverse. For example, processes
COMPUTING IN SCIENCE & ENGINEERING
 such as combustion and crack propagation
occur on micron length scales and mi-
crosecond time scales, or less, which are en-
tirely infeasible to treat a two-minute burn
of a 125-foot-long rocket.
• Manufacturing and transportation con-
straints necessitate the use of numerous
joints, including field joints where motor
segments are assembled at the launch site.
This significantly complicates the geometry
and structural response of the motor and in-
troduces potential points of failure.
• Modeling and simulating each component
is challenging both methodologically and
computationally. Although there is consid-
erable experience in the field in modeling
the various rocket motor components, a
more fundamental understanding of the
constitutive and energetic properties of ma-
terials and of the processes they undergo re-
quires much greater detail along with teras-
cale computational capacity.
• Modeling and simulating component cou-
pling is even more demanding because it re-
quires not only still greater computational
capacity, but it also demands that the corre-
sponding software modules interact in a
manner that is physically, mathematically,
and numerically correct and consistent.
When data are transferred between compo-
nents, they must honor physical conserva-
tion laws, mutually satisfy mathematical
boundary conditions, and preserve numeri-
cal accuracy, even though the correspond-
ing meshes might differ in structure, reso-
lution, and discretization methodology.
• Integrated, whole-system SRM simulation
requires enormous computational capac-
ity, currently available only through mas-
sively parallel systems that have thousands
of processors. Thus, the software integra-
tion framework, mesh generation, numer-
ical algorithms, input/output, and visual-
ization tools necessary to support such
simulations must be scalable to thousands
of processors.
In September 1997, CSAR embarked on an
ambitious plan to tackle these daunting chal-
lenges and produce a virtual prototyping tool for
SRMs.3 This article is a progress report almost
two years into our five-year plan.
Our initial plans seemed audacious, but the
substantial resources our sponsor (the US De-
partment of Energy’s Accelerated Strategic
MARCH/APRIL 2000
Figure 2. Our
Accidents
code develop-
Geometrical complexity
Joints GEN2 ment follows
family
this staged
Star grain
approach with
3D GEN1 increasing
family complexity in
2D
component
GEN0
1D models and
coupling.
Weakly Fully Detailed
coupled coupled
Physical complexity
Computing Initiative program) provided let us
assemble a team of over 100 researchers, in-
cluding roughly 40 faculty, 40 graduate stu-
dents, and 20 staff (research scientists, pro-
grammers, and postdoctoral associates), that
represented 10 departments across our univer-
sity. This diverse group provided the broad ex-
pertise needed in combustion, fluid dynamics,
structural mechanics, and computer science,
but it also presented the additional challenge of
coordinating a large collaborative project that
cuts across traditional departmental boundaries.
We organized our effort along basic discipli-
nary lines without regard to the academic de-
partments of the participants. This has had the
salutary effect of inducing collaboration among
faculty and students in a given discipline, such
as fluid dynamics, regardless of which depart-
ment they might occupy. Cross-cutting
teams—such as System Integration and Valida-
tion and Specification—draw members from all
four disciplinary groups and require an addi-
tional level of collaboration.
Staged approach
We realized from the outset that a project of
this complexity would require a staged approach:
we would need to learn to walk before we could
run (much less fly). The primary axes of com-
plexity in our problem are physical and geomet-
ric (see Figure 2). Physical complexity refers to
the detail and sophistication of physical models
employed and the degree of coupling among
them. Geometric complexity refers to the di-
mension of the problem and the degree of detail
and fidelity in representing a real SRM. In
essence, we wish to move along the diagonal of
this diagram over time. In this spirit, we defined
three successive generations of integrated rocket
simulation codes:
23
 Space Shuttle Reusable Solid Rocket Motor
We chose the Space Shuttle Reusable Solid Rocket Motor as
our primary simulation target for a variety of reasons, includ-
ing its national importance, its public visibility, its fairly typical
design, and the availability of detailed specifications and
extensive test data. We outline here the basic technical facts
about the Space Shuttle RSRM.1,2 Figure A shows a composite
drawing of the RSRM.
Height: 126.11 ft
Diameter: 12.16 ft
Weight: 149,275 lb empty; 1,255,415 lb full
Case: High-strength D6AC steel alloy, 0.479 in. to 0.506 in.
thick
Nozzle: Aluminum nose-inlet housing and steel exit cone,
with carbon-cloth phenolic ablative liners and glass-cloth
phenolic insulators. Nozzle is partially submerged and is
movable for thrust vector control.
Insulation: Asbestos-silica-filled nitrile butadiene rubber
Propellant (material percent by weight):
Ammonium perchlorate oxidizer: 70
Powdered aluminum fuel: 16
Polybutadiene polymer (PBAN) binder: 12
Epoxy curative agent: 2
Ferric oxide burn rate catalyst: trace
Propellant grain: 11-point star-shaped perforation in head
end of forward segment, aft-tapered cylindrical perforation
in remaining segments. Liquid and solid ingredients are first
thoroughly mixed into a thick paste, then curative agent is
added before mixture is vacuum-cast into a mold and then
cured in a “slow” oven for several days. Consistency of
• GEN0: 2D ideal rocket with steady-state
burning at chamber pressure, power law for
propellant regression, Euler equations for
fluid flow, a rigid case, linearly elastic pro-
pellant, and one-way coupling from fluid to
solid. We based its physical parameters on
the Space Shuttle reusable solid rocket mo-
tor (see sidebar). GEN0 was intended pri-
marily as a warm-up exercise.
• GEN1: Fully 3D whole-system simulation
code using relatively simple component
models, two-way coupling, and reasonably
realistic geometry approximating that of the
Space Shuttle RSRM. Star grain of Shuttle
RSRM is included, but not joints, inhibitors,
or cracks. Solid components include vis-
coelastic propellant and linearly elastic case.
Fluid component is an unsteady, viscous,
compressible flow, with a large-eddy simu-
lation turbulence model but with no parti-
cles, radiation, or chemical reactions in the
24
resulting composite solid is similar to that of a pencil eraser.
Igniter: Solid rocket pyrogen igniter mounted in forward end,
47.5-in. long and containing 134 lb of TP-H1178 propellant
Total launch weight: 4.5 million lb (including two SRBs, ex-
ternal tank, orbiter, and payload)
Maximum thrust: 3,320,000 lb. force (each SRB)
Acceleration: Lift-off 1.6 g (maximum 3.0 g)
Launch timeline:
Liquid engines fire: –6.0 sec
SRB igniter initiated: 0.0 sec
Lift-off pressure: 564 psia reached at 0.23 sec
All exposed propellant ignited: 0.3 sec
Maximum operating pressure: 914 psia reached at 0.6 sec
Roll program begins: 10 sec
Star grain burnout: 21 sec
Liquid engines throttled down: 30 sec
Mach 1 reached: 40 sec
Solid propellant burnout: 111 sec
SRB separation: 126 sec
Velocity at separation: 3,100 mph
Altitude at separation: 25 nmi
References
1. Design Data Book for Space Shuttle Reusable Solid Rocket Motor, Thiokol
Space Operations, Publication No. 930480, TRW-16881, Revision A,
Brigham City, Utah, 1997.
2. A.J. McDonald, “Return to Flight with the Redesigned Solid Rocket Motor,”
Proc. AIAA/ASME/SAE/ASEE 25th Joint Propulsion Conf., AIAA Paper No. 89-
2404, AIAA Press, Reston, Va., 1989, pp. 1–15.
flow. Combustion model assumes homoge-
neous surface burning and pressure-depen-
dent regression rate. There is full, two-way
aeroelastic coupling between fluid and solid
components. Development of GEN1 was
expected to span the first three years of the
five-year project.
• GEN2: Fully capable rocket simulation tool
with detailed component models, complex
component interactions, and support for
subscale simulations of accident scenarios
such as pressurized crack propagation, slag
accumulation and ejection, and potential
propellant detonation. GEN2 includes
more detailed geometric features, such as
joints and inhibitors, and also includes more
detailed and accurate models for materials
and processes based on separate subscale
simulations. GEN2 was expected to span
the last three years of the five-year project,
overlapping with the final year of GEN1.
COMPUTING IN SCIENCE & ENGINEERING
Figure A. The Reusable Solid Rocket Motor (RSRM) is a primary booster for NASA’s Space Transportation System (STS). Section A-A shows
11-point slot and fin star grain structure in the RSRM’s forward segment. Propellant in forward-center and aft-center segments form straight-
walled cylinders; aft-segment propellant tapers outward to submerged nozzle. Inhibitors between segments are asbestos-filled carboxyl-
terminated polybutadiene used to tailor burning surface to meet the motor’s thrust requirements.

Progress to date increase the propellant’s initial burning surface

We assembled the integrated GEN0 code
from existing in-house modules for fluids, solids,
and combustion components, and we completed
it in May 1998. We ran it with modest levels of
parallelism on a shared-memory SGI Power
Challenge. Computed results agreed reasonably
well with predictions of classical 1D theory, but
we didn’t extensively validate it because we never
intended GEN0 as a realistic, high-fidelity sim-
ulation; we saw it simply as a start-up system in-
tegration exercise to get our team accustomed
to working together. Visit www.csar.uiuc.edu to
view animations of our results.
We based the subsequent GEN1 code on
newly written or substantially modified in-house
codes for the various modules. We completed a
simplified, serial, but fully 3D version of it in Oc-
tober 1998. Its principal simplifications included
our use of a strictly cylindrical geometry (no star
grain, which is a slot-and-fin geometry used to
area), no support for interface regression due to
burning (not a significant factor for short burn
times, but obviously necessary for longer runs),
the requirement that the solid and fluid meshes
must match at the interface to simplify data trans-
fer, and our use of a linearly elastic (rather than
viscoelastic) model for the propellant.
Computed results without the star grain in the
propellant gave a head-end pressure at the on-
set of steady burning of roughly half the empir-
ically measured value for the Space Shuttle
RSRM. This was not surprising, as the whole
point of the star grain is to increase the exposed
surface area of propellant early in the burn,
which increases pressure and thrust accordingly.
Thus, implementing the star grain became high
priority. Another high priority was a fully paral-
lel implementation, not only because this was an
important general goal, but also for the practi-
cal reason that we could not otherwise make

MARCH/APRIL 2000 25
Figure 3. Solids and fluids codes
have different approaches to
multicomponent simulation.
Rocsolid
• Finite element
• Linear elastodynamics
• Unstructured hexahedral meshes
• ALE treatment of interface regression
• Implicit time integration
• Multigrade equation solver
• F90, MPI parallelism
Rocflo
• Finite volume
• Unsteady, viscous, compressible flow
• Block-structured meshes
• ALE moving boundaries
• Explicit time integration
• 2nd order upwind total variation
• F90, MPI parallelism

runs of significant duration at a reasonable reso-

1,000
lution. By May 1999, we had completed a fully
parallel implementation of GEN1.4 With the
star grain geometry, computed head-end pres-
Scaled speedup

100 sure was now within a few percentage points of

the Space Shuttle RSRM’s measured value.
Recent efforts have focused on implementing
10 interface regression in both fluid and solid mod-
T3E
O2K ules and on allowing for nonmatching meshes at
SP2 the interface. Both fluid and solid modules now
1 support interface regression using an ALE (Ar-
1 10 100 1,000
(a) Processors bitrary Lagrangian-Eulerian) approach in which
the mesh moves to allow for the dynamically
1,000 changing geometry. We also devised general-
mesh-association and conservative data-inter-
polation schemes that permit accurate data
Scaled speedup

100 transfer between components with nonmatch-

T3E ing meshes at the interface.
O2K The main features of the solids codes (Rocsolid)
10
CLU include finite elements, unstructured hexahedral
meshes, linear elastodynamics, ALE treatment of
regression, implicit time integrations, multigrid
equation solvers, and Fortran 90 with MPI paral-
1
1 10 100 1,000 lelism. The main features of the fluids codes
(b) Processors (Rocflo) include finite volume; block-structured
meshes; unsteady, viscous, compressible flow; ALE
1,000 treatment of moving boundaries; second-order
upwind total variation diminishing schemes; ex-
plicit time integrations; and Fortran 90 with MPI
Scaled speedup

100 parallelism. Figure 3 shows how the two compare.

Parallel scalability of Rocsolid, Rocflo, and the
GEN1 code that integrates them has been excel-
10
T3E
lent. We’ve run Rocflo on up to 2,048 processors,
O2K
and we’ve run Rocsolid and GEN1 on up to 512
processors on a variety of platforms, including the
1 SGI Origin at NCSA, the Cray T3E at Pittsburgh
1 10 100 1,000 Supercomputer Center, and all three ASCI plat-
(c) Processors
forms at the US Department of Energy laborato-
ries. Figure 4 shows “scaled speedup,” meaning
Figure 4. Graphs show scaled speedup of separate (a) Rocsolid and that the problem size per processor is constant.
(b) Rocflo modules and of (c) integrated GEN1 code for Space Shut- The largest mesh sizes have about four million el-
tle RSRM. Problem size is fixed at 14,625 fluid grid points and 8,192 ements for the solid and seven million zones for
solid elements per processor as number of processors grows. Com- the fluid. Visualizations of some computational re-
puters used include Cray T3E, SGI Origin 2000 (02K), IBM SP2, and sults are shown in Figures 5 and 6.
Intel Pentium cluster (CLU). The features we still have to implement to

26 COMPUTING IN SCIENCE & ENGINEERING

Figure 5. Fully coupled 3D simulation of star grain in Space
Shuttle RSRM showing stress in propellant and gas pressure
isosurfaces in slots and core region. Executed on 256-proces-
sor SGI Origin 2000, visualized with Rocketeer, our in-house
visualization tool.

complete the full GEN1 code include viscoelas-

ticity, large deformations, and thermal effects in
the solid; large-eddy simulation turbulence
model in the fluid; more detailed model of com-
bustion and interface regression; and a flame-
spreading model to capture ignition transients.

System integration issues
A number of technical issues arise in building
an integrated, multicomponent code such as
GEN1. First is the overall integration strategy,
where the fundamental choice is between mod-
ular and monolithic approaches. In building the
GEN1 code, we chose a modular or partioned
approach in that we used separately developed
component modules and created an interface to
tie them together. In such an approach, sepa-
rately computed component solutions might re-
quire subiterations back and forth between com-
ponents to attain self-consistency. This approach
contrasts with a more monolithic strategy in
which all the physical components are incorpo-
rated into a single system of equations and all the
relevant variables are updated at the same time,
thereby obviating the need to iterate to self-con-
sistency. Although it has some theoretical ad-
vantages, a monolithic approach impedes sepa-
rate development and maintenance of individual
Figure 6. Gas temperature computed by Rocflo in star grain region
of Space Shuttle RSRM near onset of steady burning, visualized by
Rocketeer. Values range from 3,364 K (magenta) to 3,392 K (red).
Temperature is represented as (a) tint on interior surface of propel-
lant and as (b) a series of translucent colored isosurfaces in interior
at slightly later time. Rocket is cut in half along lateral axis to
improve visibility.
component modules by specialists in the respec-
tive areas. The modular approach not only ex-
pedites separate development and maintenance,
it also allows swapping of individual modules
without replacing the entire code or even affect-
ing the other modules. The modular strategy
seemed to offer clear practical advantages in our
somewhat dispersed organizational setting, as
well as potentially letting users include com-
mercial modules when appropriate.
However, even less coupled approaches are
commonly used in practice, in which entirely in-

MARCH/APRIL 2000 27
 dependent codes interact only offline (often with
human intervention), perhaps through exchange
of input and output data files. By contrast, in our
modular GEN1 code, the component modules
are compiled into a single executable code and
they exchange data throughout a run with sub-
routine calls and interprocessor communication.
Another important issue is the physical, math-
ematical, and geometric description of the inter-
face between components, which in our case in-
cludes the jump conditions combustion induces.
A careful formulation of the interface boundary
conditions is necessary to satisfy the relevant con-
servation laws for mass and linear momentum, as
well as the laws of thermodynamics.
Time-stepping procedures are another signifi-
cant issue in component integration. Here, the
time steps for the fluid are significantly smaller
than those for the solid. Thus, we employ a pre-
dictor-corrector approach in which the fluid is ex-
plicitly stepped forward by several (say, 10) time
steps, based on the current geometry the solid de-
termines. The resulting estimate of fluid pressure
at the future time is then available for taking an
implicit time step for the solid. However, the re-
sulting deformation and velocity of the solid
change the fluid’s geometry, so the time-stepping
of the fluid repeats and so on, until we attain con-
vergence, which usually requires only a few subit-
erations. Unless iterated until convergence, this
scheme is only first-order accurate, and it is “ser-
ial” in that only one component computes at a
time. Parallel time-stepping schemes that are sec-
ond-order accurate without subiterations are pos-
sible,6 and we plan to investigate these. But be-
cause we map both fluid and solid components
onto each processor, our current scheme does not
prevent us from utilizing all processors.
As we mentioned briefly earlier, data transfer
between disparate meshes of different compo-
nents is another highly nontrivial issue in com-
ponent integration. In our approach, we let the
meshes differ at the interface in structure, reso-
lution, and discretization methodology, and in-
deed this is the case in GEN1, because the fluid
mesh is block-structured, relatively fine, and
based on cell-centered finite volumes, whereas
the solid mesh is unstructured, relatively coarse,
and based on node-centered finite elements. Al-
though in principle the two interface meshes
should abut because they discretize the same sur-
face, in practice we can’t assume this because of
discretization or rounding errors. Thus, we have
developed general mesh association algorithms
that efficiently determine which fluid points are
28
associated with each element (facet) of the solid
interface mesh;7 we then use the local coordinates
of the associated element to interpolate relevant
field values in a physically conservative manner.
Yet another thorny issue in component inte-
gration is partitioning the component meshes
for parallel implementation in distributed mem-
ory. The block-structured fluid mesh is relatively
easy to partition in a highly regular manner, but
the unstructured solid mesh is partitioned by a
heuristic approach, currently using Metis, which
often yields irregular partitions (visit www-
users.cs.umn.edu/~karypis/metis for further in-
formation). Moreover, because we partition the
two component meshes separately, there is no
way to maintain locality at the interface—adja-
cent partitions across the interface may not be
placed on the same or nearby processors. In our
current approach, this effect complicates the
communication pattern and might increase com-
munication overhead, but it has not been a seri-
ous drag on parallel efficiency so far. Neverthe-
less, we plan to explore more global, coordinated
partitioning strategies that will preserve locality
and perhaps simplify communication patterns.
Software integration framework
Our overarching goal in CSAR is not only to
develop a virtual prototyping tool for SRMs, but
also to develop a general software framework and
infrastructure to make such integrated, multi-
component simulations much easier. Toward this
end, we have initiated research and development
efforts in several relevant areas of computer sci-
ence, including parallel programming environ-
ments, performance monitoring and evaluation,
parallel input/output, linear solvers, mesh gen-
eration and adaptation, and visualization.
Our work in parallel programming environ-
ments has focused on creating an adaptive soft-
ware framework for component integration
based on decomposition and encapsulation
through objects. This environment provides au-
tomatic adaptive load balancing in response to
dynamic change or refinement, as well as high-
level control of parallel components. It is pur-
posely designed to maintain compatibility with
component modules in conventional languages
such as Fortran 90, and it provides an automated
migration path for the existing parallel MPI code
base. This work is in part an extension of the pre-
viously developed Charm++ system, which has
successfully built a large parallel code for molec-
ular dynamics, NAMD.8 Although this frame-
COMPUTING IN SCIENCE & ENGINEERING
 Rocketeer
by Robert A. Fiedler and John Norris

We need a powerful scientific visualization tool to analyze

the large, complex 3D data sets our whole system and sub-
scale rocket simulations generate. After an extensive review
of existing packages, we decided to develop our own tool,
which we call Rocketeer. (Visit www.csar.uiuc.edu/F_
software/rocketeer to download a user guide and
software.) The tool has a number of features that make it
ideal for visualizing data from multicomponent simulations,
including its support for both structured and unstructured
grids, cell-centered and node-centered data, ghost cells,
seamless merging of multiple data files, automated anima-
tion, and a smart reader for HDF (hierarchical data format).
A particularly useful feature for visualizing field data in the
interior of an SRM is Rocketeer’s ability to depict translucent
isosurfaces, which lets us view isosurfaces of temperature or Figure B. Rocketeer visualization of temperature profile computed
pressure, for example, without blocking the view of other by Rocflo in the star grain region of RSRM at the onset of steady
isosurfaces deeper inside (see Figure B). Although our need burning. Values range from 3,364 K (magenta) to 3,392 K (red).
to visualize data from SRM simulations specifically Temperature is represented by tint on the propellant fin’s surface and
motivated many of these features, Rocketeer is a broadly by a series of translucent colored isosurfaces inside the slot. The image
useful, general-purpose visualization tool. is cut in half along the rocket’s axis to enhance visibility. The hottest
Rocketeer is based on the Visualization Toolkit (VTK),1 point in this imaged flow is in the stagnation region in the middle of
which is in turn based on OpenGL to take advantage of the rocket’s head end; the coolest point is in the middle of the star
graphics hardware acceleration. It currently runs on grain region’s open end.
Microsoft Windows and Unix/Linux. Planned enhancements
include implementing a client-server model so that we can
perform most compute-intensive operations (in parallel) on Reference
a remote supercomputer while interactive control and ren- 1. W.Schroeder, K. Martin, and W.E. Lorensen, The Visualization Toolkit: An
dering are performed locally on a graphics workstation. Object-Oriented Approach to 3D Graphics, Prentice Hall, New York, 1997.

work is not yet mature enough to serve as the ba-
sis for the current GEN1 code, results for pilot
implementations of the GEN1 component mod-
ules using the framework show great promise.
In performance evaluation, we are leveraging
ongoing development of the Pablo performance
environment,9 which provides capabilities for dy-
namic, multilevel monitoring and measurement,
real-time adaptive resource control, and intelli-
gent performance visualization and steering of dis-
tributed, possibly geographically dispersed, appli-
cations. We used these tools to instrument the
GEN1 code, view the resulting performance data,
and relate it back to the application code’s call
graph to identify performance bottlenecks. We
also updated the popular ParaGraph performance
visualization tool10 to display the behavior and per-
formance of parallel programs using MPI, and we
used it to analyze the GEN1 component modules.
Parallel I/O is often a bottleneck in large-scale
simulations, both in terms of performance and
of programming effort required to manage I/O
explicitly. For large-scale rocket simulations, we
need good performance for collective I/O across
many processors for purposes such as periodi-
cally taking snapshots of data arrays for visual-
ization or checkpoint/restart. Because we use ge-
ographically dispersed machines, we also need
efficient and painless data migration between
platforms and back to our home site. Panda11
provides all these services—it runs on top of
MPI and uses the standard file systems provided
with the platforms we use. Its library offers
server-driven collective I/O, support for various
types of data distributions, self-tuning of perfor-
mance, and integrated support for data migra-
tion that exploits internal parallelism. We’ve al-
ready incorporated Panda into Rocflo, and are
doing the same in Rocsolid. Again, early results
are promising, and we plan to use Panda to han-
dle parallel I/O within our main visualization
tool, Rocketeer (see sidebar).

MARCH/APRIL 2000 29
Figure 7. The propellant surface model employs two sizes of AP particles imbedded in a fuel/binder matrix. Mixture mod-
els for this matrix account for smaller AP particles. (a) 3D flames supported by AP and binder decomposition product
gases for configuration appear at midline region. (b) Primary flame results from burning binder/AP mixture.
Validation
Comparison of simulation results with known
test cases and laboratory data is essential to es-
tablish this approach’s validity for science and en-
gineering. With our GEN1 integrated code
rapidly maturing, we have begun an aggressive
series of computational experiments to verify and
validate its efficacy and fidelity. Fluid-solid inter-
action problems we use for validation include
flow over an elastic panel, flow over a wing
(Agard Wing 445.6), and a model of inhibitor de-
formation in an SRM. We hope also to be able
to make comparisons with test data for small lab-
oratory-scale rockets through collaboration with
various government laboratories. A larger-scale
test we are pursuing is to try to predict the pro-
pellant “slumping” that led to failure for the Ti-
tan IV SRB’s original design. Our ultimate test
will be comparison with the immense amount of
test data for the Space Shuttle RSRM taken dur-
ing static firing tests after its redesign. These data
include literally thousands of readings from strain
gauges, pressure curves, and so on for a liberally
instrumented test version of the Shuttle RSRM.
New research directions
Our second-generation rocket simulation
code, GEN2, will require significantly more de-
tailed and sophisticated component models than
those in GEN1. Moreover, GEN2 will also sup-
port accident scenarios that will require even
greater detail and finer resolution. To have these
new models ready by the time we need them in
GEN2, we have already begun extensive re-
30
search into these modeling issues, some of which
we outline here.
Heterogeneous propellant flames
The propellant in the Shuttle SRM consists of
a high-density packing of ammonium perchlo-
rate (AP) oxidizer particles embedded in a ma-
trix of powdered aluminum fuel and polymeric
binder. The aluminum particles burn in the gas-
phase products of AP-binder combustion. The
propellant uses an initial bimodal distribution of
AP particle sizes of 20 and 200 microns, and the
primary combustion field is located within a few
tens of microns of the propellant surface. Flow
disturbances originating in the chamber due to
acoustic-wave or mean-flow interactions and
turbulence can affect this field, leading to what is
known as erosive burning.
Some of the heat generated in the combus-
tion field is conducted to the propellant surface.
This heat is responsible for the surface regres-
sion and the conversion of solid propellant to
combustible gases. The resulting burning sur-
face is not flat because the instantaneous re-
gression rates of AP and binder differ, and if
cracks form in the propellant, the increase in
propellant surface can lead to a sharp increase
in combustion intensity.
To describe the surface regression and to gen-
erate boundary conditions for chamber flow, we
must resolve the 3D combustion field and couple
it with physical processes such as heat conduc-
tion in a thin surface layer in the propellant and
allow for pressure and thermal feedback from
the chamber flow (see Figure 7).
COMPUTING IN SCIENCE & ENGINEERING
Crack propagation in solid propellant
The initiation and propagation of one or more
cracks in the solid propellant or along the grain-
case interface can dramatically affect rocket per-
formance. Cracks create additional burning sur-
faces in the SP, so the propagation of cracks can
greatly affect the pressure history in the rocket
chamber, leading in some cases to the rocket’s
destruction. Modeling crack propagation in
burning SP is quite complex, because the prob-
lem is characterized by a tight coupling between
structural dynamics, combustion, and fluid me-
chanics, along with a rapidly evolving geometry.
Using a fully coupled explicit aeroelastic finite-
element, finite-volume code, we are investigat-
ing potential accident scenarios associated with
the presence of pre-existing cracks at various lo-
cations in the solid booster, with special empha-
sis on propellant-liner interfacial failures. We’re
using a novel cohesive-volumetric finite-element
scheme to capture the spontaneous motion of
the crack, allowing for the possibility of crack ar-
rest or branching. As the crack propagates and
the reacting gas pressurizes newly created crack
faces, the fluid domain undergoes complex
changes that an adaptive unstructured finite-vol-
ume scheme can capture. As illustrated in Fig-
ure 8, the reactive gas flow emanating from a
pre-existing radial crack in the SP interacts with
the core flow in the rocket motor. This gener-
ates a region of high pressure on the leeward face
of the crack and a region of low pressure in the
downstream vicinity of the crack that leads to
substantial deformation of the propellant grain.
Aluminum particle combustion
The solid propellants in modern rockets use
aluminum (Al) particles as fuel. As combustion
proceeds, these particles in the propellant melt
and agglomerate on the combustion interface. A
complex process follows as Al droplets detach
from the surface and are injected into the core
flow. The droplets, whose initial size varies from
20 to 300 microns, are injected into a strong
cross-flow, and they provide a significant source
of heat as they burn to form Al oxides. Near-wall
turbulence plays an important role in the dis-
persion of Al droplets, and as a result heat re-
lease is volumetrically distributed, although
dominant mainly in the near-wall region. Al2O3
is the primary product of combustion, and it ap-
pears either as fine powder of micron size or as
larger residual particles. The deposition of Al2O3
in the form of slag in the submerged nozzle can
adversely affect motor performance.
MARCH/APRIL 2000
Figure 8. Effect of
pre-existing
radial crack on
motor’s core
flow: Colored
contours denote
pressure in core
flow and hydro-
static pressure in
solid propellant.
Note region of
high pressure
(red) in deformed
crack and of low
pressure (blue)
downstream
from crack.
The combustion of Al droplets strongly cou-
ples the dispersed phase (droplets and Al2O3 par-
ticles) with the continuous phase (the surround-
ing core flow). We expect the GEN2 version of
Rocflo to include a Eulerian implementation of
the core flow and a Lagrangian implementation
of the Al droplets and the larger oxide particles.
The simulation will introduce tens of millions
of Al droplets into the flow at the combustion
interface according to a specified probability dis-
tribution and local mass injection rate. The po-
sition, velocity, temperature, and species con-
centration of each droplet will be tracked in the
simulation over time by solving a set of ordinary
differential equations. The effect of the sur-
rounding flow will be parameterized in terms of
lift and drag coefficients, heat and mass transfer
coefficients, and droplet burn rate. So far we
have developed a detailed time-dependent 3D
subsystem simulation of the flow, evaporation,
and burning of an isolated Al droplet to obtain
accurate state-of-the-art parameterizations (see
Figure 9).
T
he individual problems just discussed
are themselves examples of fluid−
solid interactions, albeit on finer
scales, for which we will be able to
use the same software integration framework as
for the global simulation of the SRM. In this
way, we hope to leverage much of our current
software development effort in component in-
tegration, and to be able to spin off these sub-
scale simulations from the larger-scale system
simulation in a highly compatible manner.
31

Figure 9. Results obtained from a time-dependent 3D simulation of
flow and heat transfer from a spherical droplet in cross-flow. The
simulation employs a resolution of 81 × 96 × 32 points along the ra-
dial, circumferential, and azimuthal directions at Reynolds number
Re = U∞D/ν = 350, based on free stream velocity (U∞) and droplet di-
ameter (D). At this Reynolds number, flow is unsteady with time-
periodic vortex shedding. (a) Azimuthal velocity contours at an in-
stant in time, where we can see an imprint of vortex shedding. (b)
Temperature contours, where approaching flow is hotter (red) than
the droplet (blue), which is considered isothermal.
Acknowledgments
We thank our many colleagues at CSAR for their research
contributions to this article. This program is truly a
collaborative effort based on the technical strengths of
many people. We thank Amit Acharya, Prosenjit Bagchi,
S. Balachandar, Dinshaw Balsara, John Buckmaster,
Philippe Geubelle, Changyu Hwang, Thomas L. Jackson,
and Biing-Horng Liou for their contributions to the “New
research directions” section. The CSAR research program
is supported by the US Department of Energy through
the University of California under subcontract B341494.
References
1. G.R. Nickerson et al., The Solid Propellant Rocket Motor Perfor-
mance Prediction Computer Program (SPP), Version 6.0, Tech. Re-
port AFAL-TR-87-078, US Air Force Materials Lab., Edwards Air
Force Base, Calif., 1987.
2. G.P. Sutton, Rocket Propulsion Elements, 6th ed., John Wiley &
Sons, New York, 1992.
3. M.T. Heath and W.A. Dick, “Virtual Rocketry: Rocket Science
Meets Computer Science,” IEEE Computational Science & Eng.,
Vol. 5, No. 1, Jan.–Mar. 1998, pp. 16–26.
4. I.D. Parsons et al., “Coupled Multi-Physics Simulations of Solid
Rocket Motors,” Parallel and Distributed Processing Techniques
and Applications Conf., Vol. VI, CSREA Press, 1999.
5. P.V.S. Alavilli, D. Tafti, and F. Najjar, “The Development of an
32
Advanced Solid-Rocket Flow Simulation Program ROCFLO,” Proc.
38th AIAA Aerospace Sciences Meeting and Exhibit, AIAA Press, Re-
ston, Va., 2000.
6. C. Farhat and M. Lesoinne, “Two Efficient Staggered Procedures
for the Serial and Parallel Solution of Three-Dimensional Nonlin-
ear Transient Aeroelastic Problems,” Computer Methods in Ap-
plied Mechanics and Eng., Vol. 182, Nos. 3 and 4, 2000.
7. X. Jiao, H. Edelsbrunner, and M.T. Heath, “Mesh Association:
Formulation and Algorithms,” Proc. Eighth Int’l Meshing Round-
table, Tech. Report 99-2288, Sandia Nat’l Labs., Albuquerque,
New Mexico, 1999, pp. 75–82.
8. L. Kale et al., “NAMD2: Greater Scalability for Parallel Molecular
Dynamics,” J. Computational Physics, Vol. 151, No. 1, May 1999,
pp. 283–312.
9. L. DeRose et al., “An Approach to Immersive Performance Visu-
alization of Parallel and Wide-Area Distributed Applications,”
Proc. Eighth IEEE Symp. High-Performance Distributed Computing,
IEEE Computer Soc. Press, Los Alamitos, Calif., 1999, pp.
247–254.
10. M.T. Heath and J.A. Etheridge, “Visualizing the Performance of
Parallel Programs,” IEEE Software, Vol. 8, No. 5, Sept. 1991, pp.
29–39.
11. Y. Cho et al., “Parallel I/O for Scientific Applications on Hetero-
geneous Clusters: A Resource-Utilization Approach,” Proc. 13th
ACM Int’l Conf. Supercomputing, ACM Press, New York, 1999,
pp. 253–259.
Michael T. Heath is the director of the Center for Sim-
ulation of Advanced Rockets at the University of Illinois,
Urbana-Champaign. He is also a professor in the De-
partment of Computer Science, the director of the
Computational Science and Engineering Program, and
a senior research scientist at the National Center for
Supercomputing Applications at the university. His re-
search interests are in numerical analysis—particularly
numerical linear algebra and optimization—and in par-
allel computing. He wrote Scientific Computing: An In-
troductory Survey (McGraw-Hill, 1997), and has served
as editor of several journals in scientific and high-per-
formance computing. He received a BA in mathemat-
ics from the University of Kentucky, an MS in mathe-
matics from the University of Tennessee, and a PhD in
computer science from Stanford University. Contact
him at CSAR, 2262 Digital Computer Lab., 1304 West
Springfield Ave., Urbana, IL 61801; m-heath@uiuc.edu;
www.csar.uiuc.edu.
William A. Dick is the managing director of the Center
for Simulation of Advanced Rockets at the University
of Illinois, Urbana-Champaign. He received a BS in me-
chanical engineering from the University of Delaware
and an MBA from the University of Illinois. Prior to
coming to the University of Illinois, he was a deputy
director and composites engineer in the National Sci-
ence Foundation’s Engineering Research Center for
Composites Manufacturing Science and Engineering.
Contact him at CSAR, 2266 Digital Computer Lab.,
1304 West Springfield Ave., Urbana, IL 61801; w-
dick@uiuc.edu; www.csar.uiuc.edu.
COMPUTING IN SCIENCE & ENGINEERING