PROGRAMMING THE INFINIBAND NETWORK ARCHITECTURE FOR HIGH

PERFORMANCE MESSAGE PASSING SYSTEMS

Vijay Velusamy$*, Changzheng Rao$ Srigurunath Chakravarthi*, Anthony Skjellum*$##

Mississippi State University$
Department of Computer Science
and Engineering
Box 9637
Mississippi State, MS 39762, USA
{vijay, cr90}@hpcl.cs.msstate.edu
Jothi Neelamegam*,
Weiyi Chen*, Sanjay Verma*,
MPI Software Technology, Inc. *
101, S. Lafayette St., Suite 33
Starkville, MS 39759, USA
{ecap, jothi, wychen, sanjay}@mpi-
softtech.com
Department of Computer and
Information Sciences
115A Campbell Hall,
1300 University Boulevard
University of Alabama at
Birmingham,
Birmingham, Alabama 35294,
USA
tony@cis.uab.edu

Abstract enhancement opportunities. Using these APIs to create

The InfiniBand Architecture provides availability,
reliability, scalability, and performance for server I/O and
inter-server communication. The InfiniBand specification
not only describes the physical interconnect and the high-
level management functions but also a wide range of
functions from simple unreliable communication to
network partitioning, with many options. The specification
defines line protocols for message passing as a collection
of proto verbs. This paper discusses various programming
options that available to benefit from the capabilities of
InfiniBand Network Architecture. An analysis is presented
of the various strategies and programming models that
would be enabled and/or enhanced to benefit from the
InfiniBand infrastructure, for message passing systems.
Conclusions are offered about benefits, trends, and further
opportunities for low-level InfiniBand programming.
Keywords: InfiniBand, Verbs, Message Passing Interface,
Middleware, RDMA
1. INTRODUCTION
The InfiniBand Architecture is an industry standard
developed by the InfiniBand Trade Association [3], to
provide availability, reliability, scalability, and
performance for server I/O and inter-server
communication. Design principles enlisting
communication strategies to virtualize resources among an
arbitrary number of processes form the core of the
standard. The InfiniBand specification defines only the line
protocols for message passing, and not any APIs. This
paper discusses InfiniBand Verb APIs, the VI Provider
Library API [1], and certain other portable network APIs,
while offering perspective on trends, convergence, and
fast MPI-1.2 implementation is also discussed. It is
envisioned that features in MPI-2, including MPI-I/O will
be able to benefit from the support of InfiniBand
Architecture for storage networks and this in aggregation
with its benefits for message passing could be used for
high performance computing systems.
The remainder of this paper is organized as follows.
Section 2 briefly describes the various programming
interfaces that are available to utilize InfiniBand.
Programming models applicable to InfiniBand are
described in Section 3. Section 4 encapsulates some
experiences in adopting InfiniBand to high performance
Message Passing, and Section 5 summarizes the paper.
2. PROGRAMMING INTERFACES
The HCA Verbs Provider Driver is the lowest level of
software in the operating system, and interfaces with the
hardware directly. The HCA driver consists of two parts,
a loadable kernel mode driver, and a user mode library
(most often a dynamically loadable library module), as
shown in Figure 1.
Programming interfaces for InfiniBand may be broadly
classified into two main categories, namely, Channel
Access Interfaces and Portable Interfaces. Programming
interfaces are generally provided by the adapter vendor
and communicate directly with the drivers of the Channel
Access Interfaces. The interfaces layer on top of these
channel access interfaces, thereby hiding the
implementation specifics, belong to the class of Portable
Interfaces.

#
Corresponding author, to whom inquiries should be sent.
2.1 Channel Access Interfaces
The InfiniBand Architecture incorporates many of the
concepts of the Virtual Interface Architecture [1], which in
turn drew from Shrimp [11] and Cornell University’s U-
Net [12]. The Virtual Interface (VI) Architecture is a
specification for high-performance networking interfaces,
which reduces the overhead of ubiquitous transports such
as TCP/IP. VI supports both, the traditional two-sided
send/receive model of communication, as well as a one-
sided remote memory access model. Some of the
characteristics of VI have been found to have significant
impact on the scalability, and performance of the systems
that support the Message Passing Interface (MPI) [6,7].
These include the connections that are required by a large
application, time taken to dynamically open/close
connections, and support for unexpected messages. The VI
model also does not support the sharing of a single pool of
buffers between multiple VI's that limits scalability of VI
based systems.
The IBM InfiniBlue Host Channel Adapter Access
Application Programming Interface is a vendor-
independent interface that allows the development of both
kernel space and user-space applications based on the
InfiniBand transport model [4]. It has the ability to work
with different verbs interfaces, and supports multiple
channel adapters from the same or different vendors. It also
recommends that the implementation ensure that multiple
threads may safely use the APIs provided they do not
access the same InfiniBand entity (such as queue pair or
completion queue).
Other verbs API include those defined by Mellanox and
VIEO. The Mellanox IB-Verbs API (VAPI) is Mellanox's
software programmer's interface for InfiniBand verbs [5].
The Mellanox VAPI interface provides a set of operations
that closely parallel the proto verbs of the InfiniBand
standard, plus additional extension functionality in the
areas of enhanced memory management and adapter
properties specifications.
The VIEO InfiniBand Channel Abstraction Layer (CAL)
Application Programming Interface provides a vendor-
independent interface for InfiniBand channel adapter
hardware [9]. The CAL API evidently lies under the verbs
abstraction, between the Channel Interface driver software
and adapter hardware. It isolates specific hardware
implementation details, providing both a common function
call interface and a common data structure interface for the
supported InfiniBand chipsets. An advantage of such an
abstraction is the simultaneous support for heterogeneous
channel adapters. This potentially enhances path selection.
However, CAL is not widely used at present.
2.2 Portable interfaces
In addition to the above interfaces that can be used to
communicate with the Host Channel Adapters' provider
drivers directly, there also exist certain portable
interfaces that hide the channel access interface from the
user. These include the Direct Access Transport and
Sandia Portals interface.
Direct Access Transport (DAT) [2] is a standard set of
transport-independent, platform independent APIs that
are designed to benefit from the remote direct memory
access (RDMA) capabilities of interconnect technologies
such as InfiniBand, Virtual Interface Architecture, and
RDMA over IP. DAT defines both user level (uDAPL)
and kernel level (kDAPL) provider libraries (DAPL). The
current version 1.0 of the DAT provides minimal
functionality to exploit RDMA performance capabilities.
It falls short at present of completely supporting IB
functionality, such as datagrams, and multicast groups.
This short-term limitation has adverse effects on the
scalability of large applications. However, DAT also
targets systems other than InfiniBand, which enhances its
long-term value.
The Interconnect Software Consortium – Transport API
(ICSC-Transport API) defines specifications related to
programming interfaces and protocols that enable direct
interaction with RDMA-capable transports. The
specifications includes extensions to the UNIX Sockets
API, an API that provides direct user application access
to interconnect transport, and APIs that provide
application access to interconnect fabric management
infrastructure. The Phase I Specification covers VI
Figure 1: High Level Architecture (Adapted from Linux
InfiniBand Project [10])
Architecture networks, and the Reliable Connection and Table 1. Layering strategies
Unreliable Datagram services of IB.
Layering Advantage/Potential Use(s)
Sandia Portals Interface [8] is an interface for message
passing between nodes in a system area network. The VIPL over Makes legacy VIPL applications
Portals interface is primarily designed for scalability, with DAPL usable on IB, RDMA over IP
the ability to support a scalable implementation of the MPI DAPL over Useful to architect DAPL
standard. To support scalability, the Portals interface VIPL applications on legacy VIPL based
maintains a minimal amount of state. Portals supports hardware (such as Giganet)
reliable, ordered delivery of message between pairs of
DAPL over IB Portable programming across
processes, without explicit point-to-point connection
Verbs multi-vendor IB HCA’s
establishment between them. Portals combines the
characteristics of both one-sided operations and two-sided Portals over Portable programming across
send/receive operations. Since this interface was designed IB multi-vendor IB HCA’s,
originally to support MPI, implementations of Portals over particularly suitable for MPI
the IB infrastructure could enable rapid deployment of high applications
performance message passing systems, however it could Portals over Makes Portals applications usable
also provide for an alternative option for other DAPL on VIPL, IB, RDMA over IP,
communication paradigms. particularly suitable for MPI
applications
2.3 Interesting Scenarios
Portals over Suitable for Portals based
Though the IBM HCA access API and the Mellanox Verbs VIPL applications on legacy VIPL based
API are primarily targeted at IBM and Mellanox chipsets hardware (such as Giganet)
based adapters respectively, an interesting scenario would
be to use IBM verbs on Mellanox adapters or vice versa. first time the function is invoked with attr_size=0; the
Evidence of such planned cross-support exists. routine returns the actual size of the buffer. The caller is
then able to allocate the buffer and pass it to the routine.
Another interesting scenario would be a port of VI This enables efficient memory utilization, as the call
Provider Library (VIPL) over the DAT library. Even cannot estimate how much memory should be allocated.
though DAT provides with a portable layer over VI
architecture, this scenario is ideal for applications that have There is also certain additional functionality provided by
been developed over VI Provider Library, and need to be some Programming Interfaces. For example, the
ported to a layer that is already supported by DAT. Current Mellanox API supports querying the HCA’s gid, and
studies that incorporate such an idea are geared towards pkey table. The Mellanox API also as a single interface
enumerating the shortcomings of DAT for support of the for both simple data transfer, and RDMA based data
VI Provider Library. The various layering strategies transfer functions. In the Post Send Interface, the opcode
possible, along with their advantages are enlisted in Table parameter specifies the type of operation to be performed.
1. It also supports a set_ack bit flag. This enables the last
packet to have its ack_required bit. This can be used to
2.4 Interface Comparison indicate to the corresponding end point whether the
initiator expects an acknowledgement or not.
The various Verb APIs discussed above are based on the
Verbs specification of the InfiniBand Architecture. Almost DAPL provides many features that make it attractive to
all APIs have many similarities in the semantics used. The applications. Since the API hides the internals of the
many functions are generally classified as HCA Access, implementation, it provides a user-friendly platform for
Memory Management, Address Management, Completion application developers. DAPL provides a common event
Queue and Completion Queue domains, Queue Pair model for notifications, connection management events,
related, End-to-end contexts, Multicast groups, Work data transfer completions, asynchronous errors, and all
Request Processing, and Event handling. However, they other notifications. It also supports combining OS wait
have some subtle differences in the parameters that are objects, and DAT events. However, in certain cases, this
associated with certain calls. For example, in the prevents the application from benefiting from certain
Query_hca function provided in the IBM Verbs API, the features that may be available to developers using the
caller is not required to pre-allocate memory for the underlying Verbs/VIPL interface directly. Also
attributes structure. The caller calls the function twice. The presence of an additional layer, has an adverse affect on
the performance of the system.
3. PROGRAMMING MODELS
InfiniBand supports the following transport models:
Reliable Datagrams (RD); Remote Operations (RO);
Unreliable Datagrams (UD); Reliable Connection (RC);
Unreliable Connection (UC); and Unreliable Multicast data
transfer paradigms (Fig. 2). Though InfiniBand provides
support only for Unreliable Multicast, applications
requiring Reliable multicast, such as MPI implementations
could use a model that uses Reliable Datagrams coupled
with a selective retransmission strategy to achieve Reliable
Multicast (RMC) capability. The Reliable Datagrams,
Remote Operations, and Unreliable Datagram models may
be used to implement the Sandia Portals API.
Implementations of MPI-1 and MPI-2 standards may use
the Portals implementation to benefit from InfiniBand.
DAT 1.0 and VIPL will benefit from Reliable Connection
oriented data transfers. Implementations of MPI-1 and
MPI-2 standards may be layered over DAT 1.0 or VIPL.
MPI-2 One Sided Communications will benefit from
Remote Operations supported by InfiniBand. Unreliable
Connection, Multicast, Reliable Multicast will be used to
implement MPI Collective Communications.
either be effectively implemented with a fully connected
graph. The coupling of such routing information to the
services of InfiniBand is also an important question that
impacts overall scalability. For instance, if RD is
exploited as a means to enhance scalability (for example
either directly or used in connection with Portals), then a
hidden unscalability from routing information must also
be identified and removed for large-scale networks, or
else the cost of higher scalability in terms of dynamic
caching must be estimated. Brightwell et al identify in
[8] reasons why connection-oriented protocols such as
VIPL limit scalability because each connection uses fixed
resources in the NIC, rather than each session. Similarly,
the lower-level routing information may effectively
reproduce such unscalable growth in resource
requirements. Then, one has to reason if these limitations
are binding on systems of size to be created practically
(e.g., up to 65,536 nodes in current generation
superclusters). From the emphases of this paper, it is
important to mention that there may be coupling issues to
existing verb layers that require new verbs, both for
message passing and also impacts on connection
management.

Networks like Myrinet offer non-deadlocking routes MPI I/O, the I/O chapter of the MPI-2 standard is a
through the Myrinet mapper [13]. Fully connected potential beneficiary of the RC and UD capabilities
networks that can be composed from either Myrinet, offered by InfiniBand. Collective communication, and
InfiniBand fabrics, or similar technologies should offer non-blocking I/O calls could take advantage of the low
access to such non-interfering routes in order to allow latency, high bandwidth capabilities offered by
programs to exploit full bisection bandwidth and to avoid InfiniBand to simultaneously offer extended overlapping
unexpected bandwidth collapse for cases that the fabric of not only computation and communication, but also
should be able to handle without performance compromise. I/O.
While this is generally well understood in older
technologies such as Myrinet, it is an issue to be resolved
for InfiniBand. Issues include access to appropriate sets of MPI – 1/2
routes for static/clique programs that assume all-to-all
MPI – 2 PORTALS
virtual topologies (such as MPI-1 programs), access to One Sided
appropriate sets of routes for dynamic programs that add MPI - RD
1 RD Remote
RemoteOPs
OPs UD
and remove cliques during computation and may or may
not need full virtual topology support (such as MPI-2 MPI – 1/2 RMC
RMC MC
MC
programs that use DPM). There are several kinds of MPI Collective
DATDAT
1.0,1.0,
VIPLVIPL Communication
issues that arise, which we raise here, rather than solving. RC UC
First, there has to be an API that gives access to static
Figure 2. Layering Options for Network Programming
routes, if they are precomputable for a fabric. Second,
Models
there has to be an API that allows modification of such
routes, and finally, if multiple route options should be
available for more complex InfiniBand networks, the 4. HIGH PERFORMANCE MESSAGE
ability to use these multiple routes should be possible. PASSING
Finally, if there is a resource limitation that prevents
precomputed routes from being stored for large systems, a
The following are some of our experiences in porting
systematic understanding of the costs for dynamically
MPI/Pro (a commercial implementation of the MPI-1
caching such routes is important. While formally requiring
standard) and ChaMPIon/Pro (a commercial
all-to-all virtual connectivity, most real applications will
implementation of the MPI-2 standard) to benefit from
use rather more limited runtime graphs, except those that
InfiniBand.
actually do data transposition (corner turns), which might
4.1 Impacts on efficient and scalable MPI
implementations
Some of the features provided by InfiniBand include
Reliable Connection and Reliable Datagram services. In
Reliable Connection oriented services, programmers using
IB Verbs need to explicitly change Queue Pair (QP) states
to initiate handshake between two QP’s. Applications
could benefit from this for increasing the performance by
overlapping handshaking and computation, using this fine-
grained control. This would also increase the scalability of
applications in a large-scale InfiniBand network, as QP’s
could be multiplexed/reused, and reducing the number of
connections required. (Perhaps making and breaking QP
connections can be used in MPI Communicator creation
and destruction to help us use QPs efficiently, especially
when the number of QPs are limited. When the number of
QPs are limited all QPs can be created in MPI_Init, and
lightweight state change operations can be used in
MPI_Comm_create/dup and MPI_Comm_free.)
4.2 Exploiting HCA Parallelism
IB HCAs typically exploit parallelism in hardware. A
direct consequence is that throughput can be maximized by
pipelining messages using multiple QPs to transfer data
whenever possible. MPI requires message ordering in
communicator and tag scope. This may limit unrestricted
use of multiple QPs between MPI ranks, because message
ordering is guaranteed by IB Reliable Connections using
the same QP connection. Naïve MPI implementations
would use a single QP for each MPI rank pair. However,
there are several ideas to help leverage concurrent
messaging bandwidth by MPI libraries through usage of
multiple QPs. They are as follows:
a. Use one QP for each process pair for each
communicator. This is a conservative exploitation of
parallelism that benefits MPI application using multiple
communicators. No special measures are required by the
MPI library to enforce message ordering since ordering
is limited to communicator.
b. Preallocate a fixed number (more than one) of QP
connections between each rank pair, and multiplex large
message transfers across multiple QPs. This exploits
HCA parallelism by pipelining message transfers better.
As bus limitations are removed over time, such methods
may increase in value.
c. Use separate QPs for control messages (handshake
messages, flow control messages) and data. This
insulates control messages from being blocked behind
long data transfers. An extension of this model would be
to use one QP each for small, medium, large and very
large sized data.
d. Use multiple QPs, one for each {tag,
communicator} pair. This model extends model (a) to
use even more QPs. It would benefit applications using
several communicators and/or several tags, and
inherently maps well to the tag+communicator
ordering rule for MPI. However, handling the
MPI_ANY_TAG wild card will require the MPI
implementation to add extra ordering logic.
e. Use a separate QP(s) for Collective
Communication and Point-to-point communication.
This model should be easy to implement because MPI
dictates no ordering constraints between point-to-point
and collective messages spaces.
On the other hand, when the number of QPs are
restricted, it limits liberal use of QPs. Performance has
to be traded off for scalability in light of this resource
constraint.
Still developers of high performance cluster
middleware face scalability vs. bandwidth trade-offs.
However, as the Reliable Datagram (RD)
communication service type is widely supported by
mainstream IB hardware vendors, the conflict may be
solved by the solution to provide a dynamic mapping
between application communication channels and End-
to-end (EE) Contexts. This strategy requires
sophisticated design and it may be application-oriented
to avoid introducing additional overhead and/or
latency. It is widely believed to be a promising solution
for high performance clustering middleware.
4.3 Timeouts and retries
IB allows a programmer to specify timeouts and number
of retries (RNR_TIMEOUT and RNR_RETRY flags).
This can help the MPI library eliminate adding explicit
flow control mechanisms that typically add overhead.
Using an appropriate combination of timeout and retry
can help achieve low or no-overhead flow control.
Applications that inherently do not require flow control
(applications that never exhaust system buffers) will
never incur extra overhead, whereas this is not true for
even simple credit based flow control mechanisms.
4.4 Selective event completion notification
IB allows QPs to be configured so that notification of
event completion can be specified when posting a send or
receive work queue entry. Typically generating an event
queue entry is reasonably expensive as it involves
programmed I/O (PIO) across the PCI bus by the driver.
The MPI library can request for notification for only every
Nth send work queue entry, rather than for each one.
4.5 Fine-grained control over IB enabling efficient
communications
Traditional flow control, such as credit-based flow-control,
is built on top of implicit message ordering requirements
often utilizing features such as fencing, immediate data,
etc. The receive-side needs to send messages with credit to
the send-side as flow control notifications. In the cases that
data is constantly transferred one way, this credit based
flow control is unnecessary. Credit notifications may also
waste bandwidth.
The Work Request Entry of IB verbs contain fields such as
retry, timeout, etc. These provide a more fine-grained
control of communication. When specifying a proper retry
count and timeout value, flow control can be more efficient
or even remove in some cases.
4.6 Atomic operations for MPI-2 implementations
The shared memory window paradigm of MPI with
Window-based communicator matches well the memory
region model offered by InfiniBand. The semantics for
updating such windows, versus required MPI
4.7 Some experimental results
Some of our initial experiments with porting MPI/Pro
over IBM Verbs API, uDAPL, and Mellanox VAPI
included ping-pong latency, and bandwidth
measurements. Back-to-back configurations were used
for the IBM Verbs testbed. For experiments with IBM
Verbs API, and uDAPL (implementation over IBM
Verbs API), the configuration included Supermicro
motherboards, with INTEL E7500 chipsets, Intel Xeon
processors, and Paceline 2100 HCA's. Experiments with
the Mellanox VAPI were performed on InfiniHost
MTEK23108 HCA's connected by Topspin 360 switch.
The configuration included Intel Server Board SHG2,
with ServerWorks Grand Champion LE chipset, Intel
Xeon processors. The IBM Verbs based tests were
performed with two microcode versions (1.5.1, and
1.6.4), while uDAPL tests were performed only with the
HCA firmware version 1.5.1.
Figure 3 shows the Latencies achieved with MPI/Pro for
two processes, in back-to-back configuration, and Figure
4 shows the one-way Bandwidths achieved with similar
Latency
DAPL
IBMV (Microcode Ver. 1.5.1)

synchronization semantics, is an attractive match. 200

IBMV (Microcode Ver. 1.6.4)

InfiniBand supports two Atomic operations: Compare &

150
Swap, and Fetch & Add. The Compare & Swap instructs
the remote QP to read a 64-bit value, compare it with the 100
us

compare data provided, and if equal, replace it with the

50
swap data, provided in the QP. The Fetch & Add operation
on the other hand, instructs the remote QP to read a 64-bit 0
value and replace it with the sum of the 64-bit value and 0 200 400 600 800 1,000 1,200
the add data value, provided in the QP. MPI-2 offers a rich, Message Length (Bytes)

high-level distributed shared memory paradigm with one-

sided semantics in addition to two-sided message passing. Figure 3: MPI/Pro Latency over IBM Verbs
These one-sided semantics appear to work well with the
availability of both RDMA constructs of InfiniBand, and Bandwidth DAPL
IBMV (Microcode Ver. 1.5.1)
also with atomic operations of InfiniBand. While the latter 600 IBMV (Microcode Ver. 1.6.4)
have not been explored extensively at all, we find that both 500
the read-modify-write and fetch-and-add can be exploited 400
MB/s

in support of certain 1-sided operations. The MPI 300

collective accumulate operation appears to be supported, at 200
least in part by such atomic operations. The direct use of 100
0
Distributed Shared Memory (DSM) for collective data
reorganization in third-party memory appears promising.
0

0
0

0
,0

,0

,0

,0

,0

,0

,0

,0

,0

Finally, the use of DSM and RDMA together with MPI-2's

0

00

00

00

00

00

00

00

00
50

5
1,

1,

2,

2,

3,

3,

4,

4,

parallel I/O chapter appears extremely promising, offering Message Length (Bytes)
to allow the removal of intermediate TCP/IP connections
between parallel I/O clients and parallel I/O servers, and Figure 4: MPI/Pro Bandwidth over IBM Verbs
also allow flexible choices for data reorganization at either
the (clique/parallel) client or server side of the parallel I/O.
configuration. The latencies and bandwidths achieved in
IBM Verbs (microcode version 1.6.4) were considerably
better than that with IBM Verbs (microcode version 1.5.1).
Although bandwidths were comparable for uDAPL with
IBM Verbs (microcode version 1.5.1), the observed
latencies were slightly higher. This is attributed to the
overhead associated with the uDAPL layer. Also, RDMA
writes were used for long messages, as opposed RDMA
reads, which would have yielded better performance. Peak
bandwidth of up to 500 MB/s was observed.
scale remaining to be explored in great depth.
InfiniBand is a useful technology for implementing
parallel clusters, and several of the verb APIs are
particularly efficient for supporting the requirements of
MPI-1 point-to-point message passing via RDMA, while
offering the potential to enhance implementations of
MPI-2 one-sided communication. Opportunities for
enhancing parallel I/O system implementations using
InfiniBand also exist.

Latency and Bandwidth measurements for two processes

MPI/Pro VAPI Latency
using MPI/Pro over Mellanox VAPI are shown in figures 5 Blocking

and 6 respectively. MPI/Pro over VAPI supports two Polling

50.00
modes, namely polling and blocking. The blocking mode is

(microseconds)
40.00
implemented by using VAPI event handling Verbs. Polling

Latency
30.00
latencies of 8.5us, and 11.8us were observed for the VAPI
20.00
layer and MPI/Pro respectively. Event latencies of 33us,
10.00
and 38us were observed in VAPI and MPI/Pro
0.00
respectively. Peak bandwidth of 827 MB/s was observed
0 200 400 600 800 1,000 1,200
for the VAPI layer. For MPI/Pro, peak bandwidth of 802
Message Size (Bytes)
MB/s was observed when the buffers are aligned on the
page size. However, when the buffers were aligned, and Figure 5: MPI/Pro Latency over Mellanox VAPI
the same buffers were used for sending and receiving in the
ping-pong tests, peak bandwidth of 815 MB/s was
observed. It was interesting to note that best results were MPI/Pro VAPI Bandwidth (Long Messages)
obtained for MTU 1024, whereas the maximum bandwidth Polliing (Not Aligned)
Polling (Aligned)
was only 560 MB/s using MTU 2048. Blocking (Not Aligned)
900.00
Blocking (Aligned)
800.00
5. CONCLUSIONS 700.00
600.00
This paper compares, contrasts, and maps APIs to 4X 500.00

InfiniBand hardware. Commentary on porting between 400.00

300.00
APIs, and simultaneously using multiple APIs is given.
200.00
This paper analyzes the various programming models that 100.00
might be used by applications to benefit from various 0.00
features provided by InfiniBand, while assessing potential
36

16

32

64
07

14

28

57

15

30

60

converges of low-level APIs. Outstanding scalability

,5

,2

,4

,8
1,

2,

4,

8,

7,

4,

8,
65

77

54

08
13

26

52

04

09

19

38

,7

,5

,1
concerns are also raised, notably that RC is not as scalable
1,

2,

4,

8,

16

33

as desired for large clusters, and that underlying route 67 Message Size (Bytes)

computation must be utilized. Opportunities to use RD and

other InfiniBand modes together with the Portals API
appear to be excellent paths to high scalability, but routing
issues will also be important here.
InfiniBand introduces many new programming options and
many new ways to build high performance clusters. In the
next few months, convergence of programming models,
programming options, and hardening of low-level
implementations will help identify how fully commercial
offerings for production computing will evolve. The
opportunity to explore maximum scalability of InfiniBand
(APIs, protocols, and routing issues) remains for
immediate work. Effective scalability up to commonly
used sizes is obviously possible, with issues at massive
Figure 6: MPI/Pro Bandwidth over Mellanox VAPI
6. ACKNOWLEDGEMENTS
Work at both Mississippi State University and MPI
Software Technology, Inc was enabled through loaner
hardware from the respective commercial vendors
mentioned herein, as well as Los Alamos National
Laboratory, as well as superb technical support from both
Paceline Systems and Mellanox concerning low-level
software issues. Work at MPI Software Technology, Inc
and Mississippi State University was funded, in part,
under a grant for the National Science Foundation, SBIR
Program, Phase II, IIb. Grants DMI-9983413 and DMI-
0222804 are acknowledged. MPI Software Technology, [13] Nanette J. Boden and Danny Cohen and Robert
Inc also was enabled through a subcontract with the US E. Felderman and Alan E. Kulawik and Charles
Department of Energy, Los Alamos National Laboratory. L. Seitz and Jakov N. Seizovic and Wen-King
Dr. Gary Grider of Los Alamos is specifically Su, “Myrinet: A Gigabit-per-second Local Area
acknowledged as well. Network,” IEEE Micro, 15(1):29--36, February
1995.
7. REFERENCES [14] Interconnect Software Consortium,“ICSC
[1] Compaq, Microsoft, and Intel, “Virtual Transport API Specification,”
Architecture Specification Version 1.0”, http://www.opengroup.org/icsc
Technical report, Compaq, Microsoft, and Intel,
December 1997.
[2] DAT Collaborative. uDAPL API Specification
Version 1.0. http://www.datcollaborative.org
[3] InfiniBand Trade Association, “InfiniBand
Architecture Specification, Release 1.0,”
http://www.InfiniBandta.org.
[4] International Business Machines (IBM)
Corporation, “InfiniBlue Host Channel Adapter
Access Application Programming Interfaces
Programmer's Guide,” October 2002
[5] Mellanox Technologies Inc., “Mellanox IB-Verbs
API (VAPI),” 2001.
[6] Message Passing Interface Forum, “MPI: A
Message Passing Interface standard,” The
International Journal of Supercomputer
Applications and High Performance Computing,
vol. 8, 1994.
[7] Ron Brightwell and Arthur B. Maccabe,
“Scalability Limitations of VIA-Based
Technologies in supporting MPI,” Proceedings of
Fourth MPI Developer's and User's Conference,
March 2000.
[8] Ron Brightwell, Arthur B. Maccabe, and Rolf
Riesen, “The Portals 3.2 Message Passing
Interface Revision 1.1,” Sandia National
Laboratories, November 2002.
[9] Vieo Inc., “Channel Abstraction Layer (CAL)
API Reference Manual V2.0,” January 2002.
[10] Intel Corporation, “Linux InfiniBand Project,”
http://InfiniBand.sourceforge.net.
[11] C. Dubnicki et al., “Shrimp Project Update:
Myrinet Communication,” IEEE Micro, Jan.-Feb.
1998, pp. 50-52.
[12] T. von Eicken et al., “U-Net: A User-level
Network Interface for Parallel and Distributed
Computing,” Proc.15th Symp. Operating System
Principles, ACM Press, New York, 1995, pp. 40-
53.