You are on page 1of 8

Open architecture in the core of AXE

Mats Jonback and Stefan Schultz

The APZ 212 40 is the first central processor of a new generation based on The APZ 212 40 introduces a new system
industry-standard microprocessors. It introduces a new warm-standby/hot principle. In place of the parallel synchro-
on-demand system principle: the CP has two independent sides, each of nous (lock-step) machine, we are introduc-
which contains two processors—one that serves as an instruction pro- ing a warm-standby/hot on-demand principle.
cessing unit and one that serves as a signal processing unit—which run The CP has two independent sides, A and
B, that are loosely coupled via a high-speed
as a two-way symmetrical multiprocessing computer. The APZ virtual
interconnect and a maintenance channel.
machine handles ASA execution and is the middleware that guarantees Each side contains two processors: one that
telecommunications-grade availability. serves as an instruction processor unit (IPU),
The authors describe the APZ 212 40 hardware and software, the and one that serves as a signal processor unit
warm-standby concept for high availability, and differences and similari- (SPU), running as a two-way symmetrical
ties between this and previous APZ processors. multiprocessing (SMP) computer. The
hardware and the operating system define a
two-node cluster of two-way SMP comput-
ers.
Introduction The APZ virtual machine (VM), a new
part of the system, handles the ASA execu-
Being based on industry-standard micro- tion and is the middleware required to
processors, the APZ 212 40 central proces- achieve telecommunications-grade avail-
sor (CP) is a milestone in Ericsson hardware ability. It provides recovery and repair for
implementation. The latest in a line of cen- hardware faults, software upgrades and the
tral processors, the APZ 212 40 is the first like while minimizing traffic disturbance.
central processor of a new generation, and At the same time, Ericsson is introducing
the platform for future multiprocessor solu- an industry-standard hardware equipment
tions (Figure 1). practice, cPCI, and a standard operating sys-
tem. Parts of the CP core have been re-
written in standard C++ code using com-
mercial development tools.
The APZ 212 40 will be deployed as a
high-capacity central processor for all kinds
Figure 1 of application. It offers processing capacity
APZ 212 40 central processor cabinet. Fan three times as great as the fastest
Note: CDU panel not shown. APZ 212 30, and is fully backward-
RPHM-A compatible—that is, it can run all existing
AXE application software.

APZ 212 40 hardware


Hardware compatibility and structure
RPHM-B
Processor technology is advancing rapidly,
and Ericsson has decided to make the most
of research and development in the main-
Fan stream of the computer industry by using
industry-standard microprocessors. This re-
duces Ericsson’s time-to-market, especially
for upgrades, which can be made available
CPU-A as improvements are achieved in third-party
development. At the same time, this makes
CPU-B it possible to concentrate in-house design ef-
forts on developing services, improving ro-
bustness, and integrating solutions.
Each APZ 212 40 central processor mag-
Fan azine (CPUM, see Figure 3) contains five
boards and a power module (Box A). The
CPU board contains the industry-standard
microprocessors. Connections are provided
by
• the regional processor handler magazine

24 Ericsson Review No. 1, 2001


interface (RPHMI)—through cable con-
nections;
• the base input-output (IO) board— Memory in four CPU chip with
banks heat sink
through access ports and some general
support functions; and
• the update bus board (UPBB)—through
interfaces such as the new Ethernet bus
and cable connections.
The new cPCI backplane is connected to the
CPU backplane via a PCI interface module
(PIM).

Ethernet and the interplatform network


To speed up central processor-to-APG40
communication, and thus reduce reload
time, boot time, backup time, and the like,
the APZ 212 40 uses direct communication Figure 2
CPU magazine.

BOX A, APZ 212 40 BOARDS

CPU board from the microprocessor system (CPU board) the IPNX Ethernet switch in RPHM
• 2 high-performance (GHz) microprocessors, • Some general support functions for the CPU • Cable connection to the central display unit
each with 8 MB level 2 cache board and power module (CDU) panel at the top of the cabinet
• 8 GW (16 GB) SDRAM memory • Cable connections to the CPU magazine fan
UPB board units for monitoring and controlling the fans
RPHMI board • One 1 Gbit/s Ethernet optical fiber connection
• Cable connections to the magazines (RPHB) to the other CP side for updating The PCI interface module (PIM) card (not visible
for the A- and B-sides RPH-A and RPH-B • Two 100 Mbit/s Ethernet cable connections from the front) includes functionality for connect-
• Cable connection to the RPHMI on the other to the adjunct processor over the interplat- ing the CPU backplane with the cPCI backplane.
CP side for error information and for control form network (IPN), connected via the IPNX
of system states (WSB). Ethernet switch in RPHM Power module (in bottom left part of the CP sub-
• One 10 Mbit/s Ethernet cable connection to rack)
Base IO board the adjunct processor, a processor test bus • Duplicated –48V external connection
• Access port (connected to UPBB) for obtain- (PTB) for access with the central processor • DC/DC converters for internal system volt-
ing detailed low-level system-error information test (CPT) command interface, connected via ages

Processor backplane Figure 3


CPU subrack.

DIMM
memory
PIM

Compact
PCI back-
plane

Heat
sink

CPU
Base IO
RPHMI
Top view UPBB Front view

Ericsson Review No. 1, 2001 25


via Ethernet. Figure 4 shows the Ethernet central processor unit magazine and the re-
switch used to bridge communication be- gional processor input-output (RPIO) board
tween the APG40, CP, AXD 301, and other in the RPHM. With the exception of the
equipment connected to the interplatform RPIO board, the two RPHMs are identical
network (IPN). Communication between to those developed for the APZ 212 30. The
the central processor and APG40 uses cross- RPIO board has been updated to support
connected 100 Mbit/s Ethernet links. The the cross-connection.
two CP sides are connected by end-to-end 1 Of the two CP sides, one is always the ex-
Gbit/s Ethernet link optical fiber. ecutive side; the other is on standby. The ex-
The central processor test (CPT), which is ecutive CP side has control over one or both
used for maintenance and repair activities, of the RPHMs. In a normal traffic situation,
is also connected to the Ethernet switch. All the executive side controls both RPHMs.
Ethernet connections to and from the cen- Because one of the two RPHMs is always ac-
tral processor are connected to the UPBB tive (used in traffic), if an error occurs, the
board via the Ethernet switch. The use of standby RPHM can take over immediately.
Ethernet for CP-to-IO (APG40) communi- The RPHMs can also be separated, and each
cation also decreases the load on the region- can be assigned to its own CP side.
al processor handler bus (RPHB). To avoid accidents, such as the inadver-
tent disconnection of the active RPHB, the
Cross-connected regional processor state of the RPHMs is shown on the CP con-
handler bus trol and display unit (CDU) panel at the top
As in previous models of the APZ, an RPHB of the cabinet.
runs between the signal processor unit and
the RPHM. However, unlike previous
APZs, the RPHB is cross-connected in the
APZ 212 40 software
APZ 212 40 for redundancy. This connec-
tion is between the RPHMI board in the Layered software structure
The structure of the APZ 212 40 can best
be described as layered (Figure 5). From the
bottom, we have the CPU hardware plat-
form, with microprocessor, memory,
firmware console, privileged architecture li-
BOX B, TERMS AND ABBREVIATIONS brary (PAL) code, and other functions. On
top of that, we find a suitable operating sys-
AMU Automatic maintenance unit IP Instruction processor tem that interfaces with the hardware and
AOT Ahead of time IPN Interplatform network the application—in this case, software de-
AP Adjunct processor (external IO IPNX IPN switch veloped by Ericsson.
system used for man-machine JAM Jump address memory
communication, external boot media JIT Just in time The APZ virtual machine is a PlexEngine
and charging output) MAU Maintenance unit that executes traffic. It is the interface be-
APG40 The latest generation of adjunct MW 16 Megaword (1,048,576 words) 16-bit tween the existing application software and
processor most suitable for the word length the hardware platform, similar to a micro-
APZ 212 40 OS Operating system (commercial)
ASA Programming language assembler, also OS API OS application program interface
program. The virtual machine communi-
machine language in earlier APZ CPs PAL Privileged architecture library cates with the CPU platform via two addi-
BOOTP BOOT protocol PCI Peripheral component interconnect tional thin layers: an operating system ap-
BUMS Backup in the main store PIM PCI interface module plication interface (OS API) and the hard-
CDU CP control and display unit PLEX Proprietary programming language ware abstraction layer (HAL). These two
CP Central processor (includes RPHM, for exchanges
CPUM and cabling) PS Program store layers are intended to make the APZ less de-
cPCI Compact peripheral component PTB Processor test bus pendent on a particular microprocessor ar-
interconnect RP Regional processor chitecture or operating system. The idea is
CPIO Central processor input-output RPHB RP handler bus that if the operating system or hardware
CPT Central processor test RPHM RP handler magazine (subrack)
CPU Central processor unit. In this RPHMI RPHM interface board
platform is replaced, the virtual machine
machine, same as CPUM RPIO Regional processor input-output and the layers above it can remain more or
CPUM CPU magazine (subrack), one CPU side RS Reference store less intact.
DIMM Dual in-line memory module SDRAM Synchronous dynamic random The APZ virtual machine accesses the re-
DS Data store access memory sources and services provided by the under-
FTP File transfer protocol SMP Symmetrical multiprocessing
GW 16 Gigaword (1,073,741,824 words) SP Signal processor lying CPU hardware and operating system
16-bit word length UPBB Update bus board via HAL and OS API. In Figure 5,
HAL Hardware abstraction layer VM Virtual machine PLEX/ASA indicates where in this layered
IO Input-output WSB Working state bus structure the APZ software units and the

26 Ericsson Review No. 1, 2001


rest of the application can be found. In the
actual target machine, only ASA code is ex-
ecuted. PLEX-to-ASA compilation is done APG40 IPNX APZ 21240 RPHM
off target. A A A
The APZ virtual machine employs the op-
erating system application interface to com-
municate with the operating system and to
100Mbit/s 1Gbit/s
use its functions when required. The OS API Ethernet Ethernet RPHB
layer is intended to make the system less de-
pendent on particular operating system plat-
forms. If, for some reason, it becomes benefi-
cial to change the platform, the main changes APG40 IPNX APZ 21240 RPHM
will be made in the OS API. The central B B B
processor design is based on a commercial
CPU, making it feasible to use a commercial
operating system that
Figure 4
• is available for the chosen microprocessor Ethernet links and cross-connected RPHM.
architecture;
• supports multiple processors;
• uses 64-bit processor architecture;
• supports dynamic-link libraries; and
• has memory protection between process- Ahead-of-time compiler
es and threads. The AOT program compiles entire software
units of ASA binary code in background ex-
Two ASA compilers ecution, including all currently loaded ASA
The APZ 212 40 CP compiles ASA binary corrections. The AOT compilation is per-
code using two methods: just-in-time (JIT) formed in a separate process on the SPU
and ahead-of-time (AOT). Thus, the APZ processor. Its purpose is to generate code
includes a JIT compiler and an AOT com- that is as efficient as possible. The AOT
piler to compile the ASA binary code into compiler is allowed longer execution time
native CPU code. This is transparent to the to produce code. Thus, unlike the JIT com-
user. piler, it can optimize over an entire ASA
Just-in-time compiler
The APZ virtual machine calls the JIT com-
piler when there is no compiled code—that
is, when neither AOT-compiled nor JIT-
compiled code is available. The JIT program IPU processor SPU processor Figure 5
compiles and stores a “compilation unit,” Central processor structure of the
which is a small part of the ASA code—a se- APZ 212 40.
PLEX/ASA AOT
quence of instructions out of which there are
no jumps (short compilation time is the
main focus of the JIT compiler). The next
time that piece of code is called, the com- JIT APZ VM APZ VM
piled code will be executed directly. If any-
thing happens that affects the ASA binary OS API
code, such as a correction being inserted or
a trace on an instruction address using the OS OS
test system, the generated code is invalidat-
ed and must be recompiled the next time it HAL
is called.
Compared to earlier versions of APZ, the Hardware Hardware
JIT-compiled code has full register co-
herency. Consequently, the JIT compiler is
the one to use during troubleshooting. It has IPU thread, execution process
full support for the test system and the cur-
SPU thread, execution process
rent correction handling—the code is JIT-
compiled until the AOT compilation is fin- AOT compilation process
(JIT shares thread with APZ VM)
ished.

Ericsson Review No. 1, 2001 27


software unit. Capacity is the main focus of In contrast to existing central proces-
the AOT. sors, modern microprocessors rely heavily
The AOT program compiles only the most on caching, so they spend much less time
frequently executed software units. These are reading data if they have just read the pre-
either selected by the system or decided ceding data. Thus, when the APZ 212 40
manually—that is, they are preset. The num- reads data, some data that follows imme-
ber of blocks is estimated to be 10-15% of diately after is also read and put in the cache
the total number, or around 150 blocks. This memory. The introduction of the record-
should be sufficient to cover at least 90% of oriented data-store architecture only af-
the most frequently executed code, the rest fects a few PLEX or ASA software units in
is compiled using the JIT compiler. the APZ operating system and not the ap-
The AOT compiler supports signal trace plication.
and some forlopp traces, but not trace on
out-signal. If any other traces are requested APZ virtual machine
by the test system, the JIT compiler will The APZ virtual machine is the compiled
take over program execution. The trace on and linked module of C++ code that exe-
every jump option using the test system is no cutes the application. The APZ virtual ma-
longer available. chine, which is compiled into native code
The switch between AOT-compiled and off-line, is the first module loaded after the
JIT-compiled code is transparent to the user. operating system when the system is pow-
For instance, when a trace is activated, the ered on. This microprogram substitute takes
system switches from AOT-compiled to care of the incoming signals from the re-
JIT-compiled code. When the trace is deac- gional processors (RP) and from the IPN
tivated, the system automatically switches network—it schedules, dispatches, and dis-
back to the AOT-compiled code. tributes the signals that result from the jobs.
The APZ virtual machine also offers special
Record-oriented data-store services or support functions to the software
architecture layers above it (Figure 5).
The APZ 212 40 introduces a record- The APZ virtual machine executes in a
oriented data store (DS) architecture. process with two threads, referred to as the
Record-oriented data allocation is based on instruction processor (IP) thread and the
the observation that after one piece of data signal processor (SP) thread. The instruction
has been read into memory, another variable processor thread handles all job scheduling
in the same record (with the same pointer (Figure 6). The main loop of the instruction
value) will likely be read soon after. processor thread starts by polling the job
buffers for incoming jobs. The jobs are then
dispatched and executed. The main parts of
the instruction processor thread are
Figure 6 • Exec—which is the virtual machine ker-
APZ virtual machine. nel, dispatcher, scheduler, and signal dis-
tributor;
• error-handling and recovery;
JIT comp
• the run-time log;
JIT code Support • the JIT compiler; and
compiler functions
• an interface to the AOT-compiled code.
The signal processor thread handles exter-
nal communication and controls the job
table. Basically, it runs in an infinite loop
AOT comp
checking the external and IPU interfaces for
External Transporter Exec
code new signals or messages to store in job
buffers—for subsequent execution in the in-
struction processor thread. The main parts
of the signal processor thread are
• job-table scanning;
Job table Job buffer Error handler • error-handling and recovery;
IP thread • communication support (transports and
SP thread external signaling);
Non-APZ VM • RP signaling; and
• IPN signaling.

28 Ericsson Review No. 1, 2001


Warm-standby concept Execution load

for high availability


End raw copy
The CPU sides do not run in parallel syn-
chronous (lock-step) mode. Instead, a warm- Raw copy
standby/hot on-demand principle is applied. In High-
normal traffic situations, one CP side exe- level
Traffic-oriented load copy
cutes and the other side is on standby (nor-
Basic load
mal). The same system that is executing on Figure 7
the executive side is preloaded into memo- Time “Heating up.”
ry on the standby side. The standby side is
also regularly updated with transient data.
Likewise, the standby side is updated
through the automatic backup function.
When a side is scheduled to be switched or
a function is scheduled to be replaced—for with previous APZs, planned events do not
example, when testing, performing mainte- result in any traffic disturbance.
nance, or adding hardware—the standby
side is “heated up”; that is, the entire mem-
ory is copied from the executive side to the
Previous APZ
standby side, so that the standby side be- processors—similarities
comes a mirror image of the executive side. and differences
A 1 Gbit/s Ethernet link (the updating bus)
between the two CP sides provides the ca- New concepts in APZ 212 40
pability to heat up the standby side. The The warm-standby concept, which is new to
copy procedure can be divided into two APZ design, obviates the need for mainte-
phases (Figure 7): background copy (raw nance hardware to detect matching errors
copy) and traffic-oriented copy (high-level between CP sides. Consequently, the
copy). Memory is copied page by page using APZ 212 40 has no separate maintenance
a fault-on-write mechanism. This means unit (MAU).
that a copied page is set to write protect. As explained above, the APZ central
Then, when a write order to that page is re- processor is based on a commercial micro-
ceived, the page is logged before it is writ- processor that requires a substitute for the old
ten. The page is copied again later
(Figure 8).
Measurements and estimates have been
made for copying data between CP sides.
With one 1 Gbit/s link, the capacity will
drop marginally for a period less than 15 sec- Figure 8
Data transfer to standby side.
onds. At the end of the copy interval, the
system is frozen for around 0.5 seconds, al-
lowing the CP states and the APZ virtual
Data store Executive Standby Data store
machine to be copied. During this interval,
traffic from the RPs is buffered, so no data
is lost. AOT/JIT code
executed by Data
If the standby side is put into operation APZ VM Raw copy
because of a debilitating hardware fault in
the executive side, a restart with reload is
1 Gbit/s
ordered. In the normal state, a system will Data is stored
already have been loaded, so what actually in the standby
occurs is a nearly instantaneous reload fol- High- side
level
lowed by a large restart. The command log copy
is later read from the adjunct processor with- References to
pages that have
out affecting downtime. changed after Delta buffer
The advantage of the warm-standby/hot on- raw copy
demand concept is in-service performance
during software recovery, which fully com-
pensates for any negative in-service perfor-
mance from CPU hardware recovery. As

Ericsson Review No. 1, 2001 29


microprogram. The substitute is found in the CPUM—that is, a complete CP side. For the
APZ virtual machine and the new compilers. RPHM, the smallest replaceable hardware
Hence, the information that the system pro- unit is the same as in previous versions: in-
vides in fault situations and other specific dividual boards. The CP working state logic
states differs from that of earlier CP versions. is implemented on the RPHMI board; a spe-
For example, the jump address memory cial working state bus (WSB) cable connects
(JAM) contains different information. the RPHMI boards in the two CPU sides.
An Ethernet switch (Figure 4) has been The memory can be configured in steps of
introduced to connect ports in the central 1 GB. The normal way of upgrading or in-
processor to the adjunct processor and other creasing physical memory is to update a con-
systems, such as the AXD 301. figuration file on the adjunct processor, re-
place the CPUM, add physical memory, and
Hardware dissimilarities then use regular size-alteration commands
The smallest replaceable hardware unit in to inform the applications that more mem-
the new central processor is the entire ory is available. As usual, this process is
ended with a backup.
A new CP control-and-display unit panel
indicates which side is executing and which
side is on standby. It also indicates which
RPHM is handling traffic at any moment.
The CDU panel indicates normal system
state when the executing side is switching
traffic and the standby side is ready to take
over control. The standby side is ready to
BOX C, SYSTEM CHARACTERISTICS AND TECHNICAL SPECIFICATIONS take over when
• it has the same system generation pre-
Equipment practice BYB 501 loaded in system memory as the executive
Footprint 600 x 400 mm side; and
(23.62 x 15.75 inches)
Height 1800 mm (70.87 inches)
• when it is receiving transient data.
A new state (normal, NRM on the CDU
Cabinet subracks 2 CPU subracks (one each for CPU-A panel) has been defined for the standby side
and CPU-B) to indicate when these criteria are satisfied.
2 RPH subracks (RPHM) (one each for Software dissimilarities
sides A and B)
The C and C++ parts of the system are treat-
Electromagnetic compatibility (EMC) class: Fulfills EMC Class B ed separately from the rest of the system.
These separate files on the adjunct proces-
Cooling type Fans
CPU subrack 2 x 3 packages with 2 fans each
sor are loaded using BOOTP and FTP. They
RPH subrack 1 package with 3 fans each are not part of the system backup.
CPU subrack To ensure short reload time, backup in
(one of the CP sides) 1 processor board including memory main store (BUMS) must be active. When
1 interface board to RPHM
1 interface board to other CP side
1 base IO board

RPH subrack 1 interface board to CPUM


Max. 16 RPH boards (parallel type) or
Max. 8 RPH boards (serial type)
1 IPNX Ethernet switch board

Memory size 8 GW SDRAM


Power supply –48 V
Power consumption ~1400 W

System limit APZ 212 20 APZ 212 30 APZ 212 40


Data store (DS) 1.5 GW 4 GW 7.7 GW
Program store (PS) 64 MW 96 MW 256 MW
Reference store (RS) 2 MW 16 MW 16 MW
32 bits 32 bits 32 bits
Number of RPs 1,024 1,024 1,024
Number of EMGs 1,024 1,024 1,024
Capacity increase
from the APZ 212 20 1.0 3.5 10

30 Ericsson Review No. 1, 2001


APZ 212 40 APZ 212 40 APZ 21240 APZ 212 40

ABC 123

Duplicated switched Gbit/s Ethernet

Job distributor

RPHM RPHM
APG40 APG40

Figure 9
APZ 21240 Example of a node in the future.

backup of the system data is first made, it is Conclusion


stored in the backup area of primary memo-
ry. As part of the backup function, the sys- The new APZ hardware architecture makes
tem is then transferred to external media— the most of increased processing speed
the APG40. The automatic backup function through the use of industry-standard micro-
is routed via the standby side to the adjunct processors and innovations such as internal
processor. In the standby side, an automatic Ethernet buses. The mainstream sourcing of
reload is initiated to keep the standby side certain processors allows AXE users to ben-
“warm.” efit from general advances in microproces-
When the APZ VM, JIT, HAL, OS API, sor technology.
AOT, or OS needs to be updated, the new The latest version of Ericsson’s APZ cen-
version is loaded as a normal file onto the tral processor introduces several innovative
adjunct processor. A boot is then initiated concepts, but backward compatibility with
in the CP standby side, and the new versions previous versions of AXE hardware and soft-
are loaded. The CP sides (roles) are then ware has been ensured, thereby safeguard-
switched and the sequence is repeated. ing operator investments.

BOX D, KEEPING PACE WITH FUTURE DEVELOPMENTS

With its architecture and use of commercial processor systems. In a distributed multi-
microprocessors, the APZ 212 40 allows us to processor system, replicated processors can
take advantage of rapid evolution in the com- improve in-service performance and simplify
puter industry. Development will keep pace with handling when adding capacity to the node. Fig-
the mainstream computer industry and enable ure 9 shows an example of a multiprocessor
improvements in characteristics, such as system.
capacity and robustness, with limited changes A powerful duplicated processor handles
in software. The APZ central processor design job distribution between a number of repli-
is also open enough to simplify moves between cated processors (call handlers). The APG40,
different microprocessor suppliers and the for IO communication, is connected to the
operating systems they support. Gigabit Ethernet used for communication.
The introduction of the interplatform network Other equipment can be attached to the net-
(IPN) via directly connected Ethernet makes the work based on the needs of applications.
APZ 212 40 the obvious platform for future multi-

Ericsson Review No. 1, 2001 31