Productivity Tools for HFSS: Increasing Speed and Size with DSO and HPC

2010 ANSYS, Inc. All rights reserved.

2010 ANSYS Regional Conferences

Objectives Introduction Brief overview of technology and acronyms Licensing options & Enabling technologies DSO HPC Case studies Conclusions

Introduction This talk will focus on additions to a standard HFSS solver to: Increase efficiency
Solve more models in less time Solve a single model in less time

Increase capacity
Solve larger models

Technology Overview

2010 ANSYS, Inc. All rights reserved.

2010 ANSYS Regional Conferences

Computing Terminology
HPC High Performance Computing

Uses supercomputers and computer clusters (connected groups of computers) to solve advanced computation problems [From Wikipedia]
Socket Part on computer motherboard to place Processor packaged device

Modern motherboard contain as few as 1 socket, and as many as 4 sockets

Core Single computing unit

Modern processors packages contain as few as 2 cores, and as many as 12

Common configuration
Dual-socket, Quad-core package 8 cores in a box

ANSYS Terminology On top of all these hardware choices, weve added license choices to improve productivity for different classes of problems DSO Distributed Solve Option DDM Domain Decomposition Method
Feature of HPC licensing

MPO Multi-processing option

Feature of HPC licensing

HFSS Adaptive Mesh

HFSS Solver Terminology

Each time HFSS solves the Volumetric Field Solution it must solve a matrix of unknowns. The solution describes the field behavior for that particular mesh This is done for each adaptive pass and directly solved frequency point.

Ax = b
HFSS offers 3 Solvers Options to apply to this matrix equation: 1. Direct Solver (Default)
Traditional solver used in HFSS Very stable Can be memory and time intensive for large matrices Added in HFSSv11 More memory efficient than the Direct Solver Can be more time efficient than the Direct Solver Added in HFSSv12

2. Iterative Solver

3. Domain Decomposition (more on this later)

HFSS Solver Terminology Direct Solver

The Direct Solver obtains an exact solution to the matrix equation

Ax = b
Common Direct Matrix Solver Methods: Gaussian Elimination LU Decomposition Best uses for the Direct Solver Moderately sized matrices Large number of excitations

a11 0 0 0

a12 a22 0 0

a13 a23 a33 0

a14 x1 b1 a24 x2 b2 = a34 x3 b3 a44 x4 b4

HFSS Solver Terminology Iterative Solver

How does it work?
The Iterative Matrix Solver works by guessing a solution to the matrix of unknowns, and then recursively updating the guess until an error tolerance has been reached Initial guess

Preconditioner

What is the advantage?

Reduced RAM and Simulation Time no

Update solution and search direction

Best uses for the Iterative Solver

Large Matrices (>30,000 Tets) Moderate Port Count (2 Ports Per Processor) For 1st, 2nd and Mixed Order Basis Functions only

Converges ? yes MPCG

Iterative Matrix Solver Increases Simulation Capacity

Compared direct and iterative matrix solvers for JSF example Iterative solver requires 3-4x less RAM than direct solver

Productivity Option #1 Distributed Solve (DSO)

2010 ANSYS, Inc. All rights reserved.

12

2010 ANSYS Regional Conferences

Distributed Solve Option (DSO)

Distributed Solve Option (DSO)

Design Variations Optimetrics / ANSYS DesignXplorer

Distributed Solve is a productivity enhancement option that accelerates solution times for frequency sweeps and model variations by leveraging a network of processors. Offers a near-linear speed-up over conventional single license simulation sweeps by distributing and simultaneously solving across a network of computers Increases throughput by speeding up turnaround time for individual simulations

Frequency Sweeps HFSS

Distributed Solve - Applications

Applications

Licensing
Hardware Independent Mix different CPU/Cores and RAM User defined machine selection Group setting for solver MP OS Independent Supports Windows and/or Linux LSF/PBS/SunGrid/HPC Enabled Solver independent Common license for supported solvers MP ready Flexible Share the licenses in the pool between multiple users or simulations License Options: Singles - 1 Design Point Saver Pack 10 Design Points

What-if studies Design of experiments (DOE) Dynamic circuit model generation Design for Six Sigma (DFSS) Broad-band frequency sweeps

DSO Examples: Parametric

Optimetrics analysis of circular waveguide phased array Parametric sweep over 45 scan angles 5X faster when distributed to 6 CPUs Optimetrics analysis of PIFA radiating element Parametric sweep of antenna geometry 7.5X faster when distributed to 8 CPUs

Ansoft Corporation

S11 for Element 1 Parametric Sweep

isolation

0 -5 dB(S(P1,P1)) -10 -15 -20 -25 -30

Curve Info dB(S(P1,P1)) Setup1 : Sweep1 extra_element_lengt dB(S(P1,P1)) Setup1 : Sweep1 extra_element_lengt dB(S(P1,P1)) Setup1 : Sweep1 extra_element_lengt dB(S(P1,P1)) Setup1 : Sweep1 extra_element_lengt dB(S(P1,P1))

Scan Impedance

-35 2.0

2.1

2.2

2.3

2.4 2.5 2.6 Freq [GHz]

2.7

2.8

2.9

3.0

DSO Example Investigating Solution Space

Distributed analysis used to quickly explore multi-dimensional design space
Wire radius Pitch spacing Helix radius

DSO distributes frequency and parametric sweeps to network of processors Approximately linear increase in simulation throughput Highly scalable to large numbers of processors Multi-processor nodes can be utilized

DSO distributes frequency and parametric sweeps to networked processors

Solution Space Exploration Helical Wire Antenna

Wire radius varied to determine impact on input impedance
Used DSO to solve 27X faster
Frequency

Return Loss
0.06 in 3GHz

Wire Radius

0.15 in

3D plots created in HFSS to easily visualize solution space

Return loss as function of frequency and wire radius

Acceptable Wire Radius

4GHz

DSO Example: Molex Connector Frequency Sweep Distribution

Adaptive process completed on one machine with frequency Sweep #N sweep sub-bands sent to multiple machines
Sweep #3 Sweep #2

Setup
Parametric Model Generation

Solution
Mesh Generation
Adaptive Mesh Refinement

Sweep #1
yes

Convergence

Frequency Sweep

no
#pts to Converge Reference DSO Interpolating 76 78 Clock Time Delta to Reference 1x 5.8x

22h26m 3h52m

DSO Discrete

NA

2h41m

8.4x

Productivity Option #2 High Performance Computing Option (HPC)

2010 ANSYS, Inc. All rights reserved.

19

2010 ANSYS Regional Conferences

Ansoft HPC Overview

Domain Decomposition (DDM)

Bigger

Domain Decomposition

A distributed memory parallel solver technique that distributes mesh subdomains to a network of processors. This method is a hybrid iterative and direct solver technique that significantly increases the simulation capacity by distributing the RAM usage across multiple computers. Enables the solution of higher fidelity and larger models

Multi-Processing (MP)

Faster

Multi-Processing

The MP option is used for solving models on a single machine with multiple processors/cores which share RAM. Increases throughput by speeding up turn-around time for individual simulations

Ansoft HPC - Applications

Cell Tower

Applications

Electrically Large RF/Antenna Designs Antenna Placement Radome Design Radar Cross-Section (RCS) EMC Analysis
Friend or Foe Antenna

Industries

Aerospace and Defense Wireless/Mobile Platforms Communications Healthcare

Satellite

RCS 24GHZ Medical

Domain Decomposition for HFSS

New feature in HFSS v12 Distributes mesh subdomains to network of processors Distributed memory parallel technique Significantly increases simulation capacity
64-bit meshing

HPC distributes mesh subdomains to networked processors and memory

Highly scalable to large numbers of processors Multi-processor nodes can be utilized

Domain Decomposition Example Cellular Base Station Array

GSM base station tower with radome-enclosed antenna arrays
950 MHz Electronic downtilt

Domain solver used to predict installed antenna patterns

34 domains 3.5 GB average RAM per domain 16M unknowns 119 GB Total Effective RAM used

Base Station Printed Dipole Arrays

Technology Comparison
HPC for MP
Geometric Complexity Excitations/RHS

HPC for DDM

Solver Fitness

Direct

Iterative

DDM

Electrical Size/Fidelity

Ansoft HPC - Licensing

Allocation
Each Simulation consumes one or more HPC packs Each individual pack enables 8 Parallel Parallel count increases quickly with multiple packs
Parallel Enabled
2048 512 128 32 8

Flexible Technology Access

Enable MP or DDM or DDM with MP

2 3 4 5 Packs per Simulation

Scalable Licensing
HPC Packs HPC Workgroup (Volume access to parallel) 128 to 2048 Parallel shared across any number of Simulations Enterprise

HPC License count is determined by the larger of the two: 1. # of Simulations (Solvers) 2. # of Cores

Ansoft HPC Multi-Processor Option

Single workstation solution to increase simulation throughput
Takes advantage of multi-core and/or multi-processor computing resources

Capability introduced in HFSS v8 for direct matrix solver

Parallelized matrix solver for multiple processors with shared memory

Enhanced by addition of iterative matrix solver in HFSS

Parallelized matrix pre-conditioner Parallelized excitations

MP Option for Helix Design

Element model converges with 30k mesh elements Multi-processor option reduces direct solver time by factor of 2x
20 seconds vs 40 seconds

Array model converges with 330k mesh elements Multi-processor option reduces iterative solver time by factor of 2.5x
8 minutes vs 20 minutes

Multi-processor option significantly decreases design iteration time at element and array levels

Case Studies

2010 ANSYS, Inc. All rights reserved.

28

2010 ANSYS Regional Conferences

Finite Array on Spacecraft

Electrically very large model with high level of geometrical detail Historically beyond the realm of full-wave EM solvers
Typically analyzed using asymptotic approximations which may sacrifice accuracy Challenging but important design problem

Full-wave analysis now possible using HFSS v12

Array on Spacecraft
Efficiently solved using HPC Option
Domain solver

Solution profile
25M unknowns 34 domains 6 GB average RAM per domain 204 GB Total Effective RAM used

Automobile with GSM Antenna

Domain solver used to predict on-vehicle patterns
11 domains 1.7 GB average RAM per domain 19 GB Total Effective RAM used
Far-field Radiation Pattern

Surface Currents

Radiation Pattern Electric Fields Around Vehicle

Apache Helicopter RCS

Military rotary-wing aircraft
1 GHz Monostatic RCS

Domain solver used to predict scattering signature

12 domains 4.5 GB average RAM per domain 6M unknowns 54 GB Total Effective RAM used

Surface Currents Monostatic RCS

Ground Transport Vehicle with Covert Patch Antennas

Domain solver used to predict installed antenna patterns
Two L-band patch elements mounted on Humvee roof

Solution profile
6 domains 0.75 GB average RAM per domain 4.5 GB Total Effective RAM used

RFID System in Loading Dock

Domain solver used to solve RFID system in industrial dock door environment
900 MHz system 2 patch antenna readers on pedestals 12 tags distributed throughout pallet of packaged items

Field due to reader

Solution profile
7 domains 2 GB average RAM per domain 14 GB Total Effective RAM used
Field due to tag

How big can you go? HumVee with 1 L-band patch antenna In proximity to cement wall with rebar Freq = 1.8 GHz 14,424 3 64 Domains 2 procs / domain 128 cores 50M Unknowns 409 GB Total RAM

Summary

2010 ANSYS, Inc. All rights reserved.

36

2010 ANSYS Regional Conferences

Summary: DSO Many parametric variations / design space exploration DoE & ANSYS DesignXplorer Broadband Frequency Sweeps Signal Integrity / EMI problems In both cases, each parametric variation or frequency point will be limited by the available RAM per core in the Distributed machine list

Licensing: DSO DSO licensed in either single-task or multitask bundles License enables MP functionality for each task Typical bundle is 10 tasks

Summary: HPC Solving the previously unsolvable models electrically HUGE Solving the existing problems on one machine, faster than before

Licensing: HPC
HPC is licensed in either Packs or Workgroup/Enterprise 1 Pack enables 8 cores for 1 problem
Up to 8 cores for domains OR MP

2 Packs enables
8 cores each for 2 problems OR 32 cores for 1 problem

3 Packs enables
8 cores each for 3 problems OR 32 cores for 2 problems AND 8 cores for 1 problem OR 128 cores for 1 problem

Count the cores!