Asymptotical Analysis and Comparison of

Two Coded Modulation Schemes Using PSK

Signaling { Part I
Gerd Beyer, Karin Engdahl (Student Member, IEEE),
Kamil Sh. Zigangirov (Fellow, IEEE)y
April 23, 2001

Abstract{ An asymptotical analysis and comparison of two coded modula-

tion systems using phase shift keying (PSK) signaling with the transmission
taking place over the additive white Gaussian noise (AWGN) channel is pro-
vided. In the rst part of the paper, bounds for the decoding error probability
of conventional coded modulation are derived. Both block and trellis coding
are considered. The state-complexity error exponent is introduced. In the
second part of the paper we will study a multilevel modulation scheme with
multistage decoding and we will compare the asymptotical performances of
the two systems, that is when the block length goes to innity (in the case of
block codes), or the code memory goes to innity (in the case of convolutional
codes).

Keywords{ Coded modulation, Multilevel modulation, PSK-signaling.

 The material of this paper was presented in part at ISIT 2000, Sorrento, Italy, June
25-30 2000.
y The authors are with the Dept. of Information Technology, Lund University, Box 118,
SE { 221 00 Lund, Sweden.

1
I Introduction
It was known already in the 60s, that by combining channel coding and
modulation it is possible to increase the data rate compared to that of an
uncoded modulation system with the same bandwidth and energy rate. The
combination of modulation, convolutional codes and soft Viterbi decoding
has an especially positive eect. Further progress in this area was done by
Ungerbck in his pioneering work [1], where he suggested a principle of set
partitioning of a signal constellation. The core of Ungerbck's scheme is
dividing a signal set into smaller subsets with increasing intra-set Euclidean
distance. Independently, Imai and Hirakawa [2] proposed a multilevel coded
modulation scheme based on binary codes. The multilevel scheme enables the
use of multistage decoding, which has a better complexity-reliability tradeo
than the decoders of conventional coded modulation schemes, and provides
a larger bandwidth eciency.
Several researchers have studied the theoretical performance of multi-
level modulation with multistage decoding. In [3],[4], the eective capacity
of multilevel schemes using quadrature amplitude modulation (QAM) and
PSK signaling was studied, and the concept of average distance prole for
dierent levels was introduced. The problem of propagation of decoding er-
rors at the highest coding level was studied in [5],[6]. In these papers the
application of suboptimal methods for decoding of component codes was
also studied. But, although multilevel modulation with multistage decoding
is widely used in practice, many theoretical problems are still not solved.
Among these problems are the mathematically correct comparison of con-
ventional coded modulation schemes to the corresponding multilevel coded
modulation schemes, and the optimal choice (in the sense of complexity-
reliability tradeo) of parameters of the component codes.
This paper continues the work of two of the authors of this paper, devoted
to theoretical analysis of multilevel modulation schemes [7]{[12] (see also

2
[13]). In these papers we analyzed the decoding error probability for dierent
multilevel modulation schemes [7]{[10] as well as introduced and analyzed
hexagonal multilevel QAM with multistage decoding [11]{[12]. The main
goal of this paper is a comparison of the asymptotical performances of a
conventional coded modulation system using PSK signaling and transmitting
over the AWGN channel, and the corresponding multilevel coded modulation
system.
In the rst part of the paper we analyze the conventional q-PSK signaling
over the AWGN channel. We study both block coding and trellis coding.
Partially, we reformulate known bounds for the decoding error probability in
the application to this system. In these cases we omit the proofs. We also
prove new bounds for the distance and for the decoding probability. Among
the new bounds are the Costello-type bound for the free distance of trellis
codes, and the expurgated bound for the error probability of trellis codes.
The last bound can not be proved by expurgation of code words, as it can be
done in the block coding case, but by the procedure of expurgating codes from
the ensemble of linear codes. The state-complexity error exponent, which is
essential in the analysis of the complexity-reliability tradeo, is introduced.
In the second part of the paper we derive bounds for the decoding error
probability of a multilevel coded modulation system with multistage decod-
ing. Correspondingly, both the block coding case and the trellis coding case
are considered. Multistage Viterbi decoding of trellis modulation is stud-
ied when either optimal or suboptimal metric is used. Then a comparison
of the asymptotical performances of the conventional and multilevel coded
modulation schemes is provided.

II Channel Model and Preliminaries

Let S , a set of q complex variables,
p  2 2 
 
S = Es cos q i + | sin q i ; i = 0; 1; :::; q 1 ; (1)

3
where Es is the signal energy per channel use, be the input alphabet of a
discrete memoryless channel with complex Gaussian noise e = e I +| e Q . We ( ) ( )

suppose that e I and e Q are independent zero-mean Gaussian variables with

( ) ( )

variance N =2. Given the channel input sn 2 S and the noise component en,
0

n = 0; 1; :::, at the n-th moment of time, the channel output is rn = sn + en.

Since the channel is memoryless, the complex random variables en, n =
0; 1; :::, are independent. This is the discrete-time analogue of the AWGN
channel with two-sided power spectral density N =2 and q-PSK signaling.0

Es=N is called the signal-to-noise ratio (SNR ) per channel use. We consider
0

only the case when q is a power of 2, q = 2L; L = 1; 2; :::.

We will study asymptotical performances of coded transmission over a
channel with discrete input and continuous output, when the memory of the
code goes to innity.
Consider block transmission over the channel. Let M be the number
of possible messages, N the block length and R = (log M )=N (in bits per 2

channel use) the transmission rate. Let

s = s ; s ; :::; sN ;
0 1 1 sn 2 S; n = 0; 1; :::; N 1 ; (2)
be the channel input sequence and let
r = r ; r ; :::; rN ;
0 1 1 (3)
where rn, n = 0; 1; :::; N 1, are complex variables, be the received sequence.
The conditional probability density function of the received sequence r, given
that the transmitted sequence is s, is
Y
N 1
1 exp
 d (r ; s ) 
E n n
f ( r j s) = N N ; (4)
n=0 0 0

where dE (r; s) is the squared Euclidean distance between the complex vari-
ables r and s. Application of known results from information theory [14],
gives the following expression for the capacity of the AWGN channel with
4
q-PSK signaling:
1 X
q 1 Z1 Z1 1
C = q N
i 0

1 (p1 E p 
=0

s cos (2i=q ) x) + ( Es sin (2i=q) y)

2 2

exp N
 pE
0
p 
i=q) x)2 +( Es sin(2i=q) y)2
q exp ( s cos(2
N0

log 2 qP  pE p  dx dy : (5)
s cos(2j=q) x)2 +( Es sin(2j=q) y)2
1
exp (
N0
j =0
We note that for PSK input, the AWGN channel capacity is reached for
uniform distribution of transmitted symbols over the input alphabet S .
As usual we dene a block decoding error as the event when the de-
coded message does not coincide with the transmitted message. Let Pe be
the probability of this event. Pb is the bit error probability, dened as the
mathematical expectation of the ratio of the number of erroneously decoded
bits to the number of transmitted bits. The function
EB (R) = Nlim log Pe
def 2
(6)
!1 N
is called the block reliability function, or block error exponent. Since
Pe  P  P ; (7)
b e
log M 2

the bit reliability function, limN !1(log Pb)=N , coincides with EB (R). There-
2

fore we will only study the asymptotical behavior of the decoding error prob-
ability Pe when N ! 1. Particularly, we establish lower and upper bounds
for EB (R),
E B (R)  EB (R)  E B (R) : (8)
The upper bounds E B (R) for the block reliability function, namely the
sphere-packing bound and the straight line bound, will be formulated for
arbitrary block codes. The lower bounds E B (R) for the block reliability
function, namely the random coding bound and the expurgation bound, will
5
be formulated for arbitrary block codes, but the proof of the existence of
codes, whose reliability functions satisfy the latter bound will be given for
codes constructed on the base of linear codes over the integer ring Zq modulo
q.
The necessary denitions for trellis coding, particularly the denition of
the convolutional reliability function EC (R) will be given in Section V. The
upper bounds E C (R) for EC (R), namely the Viterbi-type bound and the
straight line bound, will be proved for arbitrary trellis codes. The lower
bounds E C (R) for EC (R), namely the convolutional random coding bound
and the convolutional expurgated bound, will be proved for codes constructed
on the base of convolutional codes over Zq .

III Block Coding Upper Bounds for the Re-

liability Function
In this section we formulate lower bounds for the decoding error probability
in the case when block coding is used for error control. From these bounds
the upper bounds for the reliability function follow directly.
The Gallager function of the AWGN channel with q-PSK signaling (for
uniform distribution over the input alphabet, which is optimal in this case)
is dened as
Z1 Z1 " 1 X
q 
1
1

G(s) = log 2
q N (9)
1 1 i=0 0

 (pE cos (2i=q) x) p

+ ( Es sin (2i=q) y)
2 2
 1+1 s # 1+s
s
exp N dx dy ;
0

0s<1:
Theorem 1 (Sphere-Packing Bound) For any block code of rate R and
length N , that is used to communicate over the AWGN channel with q-PSK
signaling, the decoding error probability Pe is lowerbounded by the sphere-

6
packing bound
Pe  2 fEB; sp ( R o1 (1))+o2 (1)gN ; (10)
where
EB;sp (R) = sup [ G(s) sR ] ; 0< s< 1; (11)
s
and G(s) is given by (9). The quantities o1 (1) and o2 (1) go to zero with
increasing N and can be taken as o1(1) = O1 ((log2 N )=N ) and o2 (1) =
p
O2(1= N ). 2
The theorem follows from the famous theorem by Shannon, Gallager and
Berlekamp [15], see also [14], which gives the sphere-packing bound for the
memoryless channel with discrete input and discrete output. A sketch of the
proof is given in Appendix A. The sphere-packing error exponent EB;sp (R)
can be parametrically represented as
(E sG0(s) ;
B;sp (s) = G(s)
0<s<1: (12)
R(s) = G0 (s) ;
From Theorem 1 the bound
EB (R)  EB;sp (R) (13)
for the reliability function follows. For low rates the bound (13) can be
strengthened. The proof of this strengthening follows from the Plotkin-type
upper bound for the minimum squared Euclidean distance of the code, and
is given in Appendix B.
Theorem 2 (Straight-Line Bound for Block Codes) Consider trans-
mission over the AWGN channel with q-PSK signaling. For any block code of
rate R and length N , let RB be the rate at which there is a tangent EB;sl (R)
to the curve EB;sp (R), that also satises EB;sl (0) = Eex(0) = Es=(2N0 ln 2),

EB;sl (R) = Eex(0) Eex(0) REB;sp (RB ) R : (14)

B

7
Then, for N ! 1 and any R satisfying
0  R < RB ; (15)
the decoding error probability Pe is lowerbounded by the inequality
Pe  2 [EB;sl R o
( )+ (1) ]N ; (16)
where o(1) ! 0, when N ! 1. 2
The function
( E (R ) ; 0  R < RB ;
B;sl
E B (R ) = (17)
EB;sp (R) ; RB  R < C ;
is presented in Figure 1 for q = 32 and Es=N = 10dB. 0

IV Block Coding Lower Bounds for the Re-

liability Function
The best way to construct a block code B for the AWGN channel with q-
PSK signaling is to rst construct a linear block code V over the integer ring
Zq modulo q, and then to map the symbols of the code sequences V onto
symbols sn 2 S .
A linear rate R = log (q)K=N , length N code V over Zq , q = 2L, L =
2

1; 2; : : :, is dened by its K  N generating matrix

G = kgktk ; k = 0; 1; :::; K 1 ; t = 0; 1; :::; N 1 ; (18)
where gkt 2 Zq . Let
u = u ; u ; :::; uK ;
0 1 1 uk 2 Zq (19)
be the q-ary information sequence at the encoder input, and let
v = v ; v ; :::; vN ;
0 1 1 vt 2 Zq ; (20)
8
such that
v = uG ; (21)
be the q-ary code sequence. If we map the sequences v onto the code words
s = s ; s ; :::; sN ;
0 1 1 sn 2 S ; (22)
we get a code B over the alphabet S .
For an arbitrary code word s, let ad (s) be the number of code words s0 in
B , such that dE (s; s0) = d. We will call the set fad (s)gNd the spectrum of
=1

the code, since, as we will prove now, ad (s) does not depend on s.
Consider two arbitrary code words of V , v = v ; v ; :::; vN and v0 =
0 1

v0 ; v0 ; :::; vN0 and the corresponding code words s and s0 of B . The sum of v
0 1

and v0 over Zq and the dierence of v and v0 over Zq are also code words.
Since dE (s; s0) depends only on the dierence of v and v0 over Zq , all code
words s in B have the same spectrum. Consequently the decoding error
probability does not depend on the transmitted code word. Therefore it is
sucient to study the spectrum of and the transmission of the allzero code
word.
Consider a random ensemble of linear codes V , for which the entries
of the generating matrix G are independent identically distributed (i.i.d.),
equiprobable q-ary random variables, and the corresponding ensemble S of
codes in the alphabet S . In this ensemble the code sequences s (except
for the code sequence corresponding to the allzero information sequence),
are random sequences with statistically independent symbols sn, uniformly
distributed over S or a subset of S (see below). Furthermore two code se-
quences corresponding to two nonzero information sequences are pairwise
independent. This pairwise independence is sucient for the derivation of
the random coding and expurgation bounds for block and trellis codes (for
details see [14] and [19]).
Theorem 3 (Random Coding Bound for Block Codes) For the
AWGN channel with q-PSK signaling, there exists a rate R block code of
9
length N , whose decoding error probability is upperbounded by the inequality
Pe < 2 [EB;rc(R)+o(1)]N ; (23)
where
EB;rc(R) = sup [G(s) sR] ; (24)
0 s1
G(s) is given by (9) and o(1) ! 0 if N ! 1. 2
The proof of this theorem almost coincides with the proof of the random
coding bound for a memoryless channel with discrete input and discrete out-
put [14], except at one point. In the ensemble of codes used in the discrete
memoryless channel a code word corresponding to any nonzero information
sequence is a sequence of N i.i.d. random variables uniformly distributed
on the input alphabet. In the q-PSK signaling case, the distribution of the
symbols of a code word depends on the corresponding information sequence.
If at least one of the symbols of the information sequence is odd, then the
symbols of the corresponding code word are i.i.d. random variables uni-
formly distributed on the whole alphabet S . But if all the symbols of the
information sequence are even, then the symbols of the corresponding code
word are uniformly distributed not on S , but on a subset of S . Therefore
it is necessary in this proof to consider several cases: rst when all symbols
of the information sequence are divisible by 2L , then when at least one of
1

the symbols is divisible not by 2L but by 2L , and so on. This makes

1 2

the proof more complicated, but does not change the result, since the case
determining the result is the case when at least one of the symbols of the
information sequence is odd. A similar consideration should be done also for
some of the following theorems.
Analogously to the discrete channel case, the value G(1) = R is called
0

the computational cuto rate, the value dG(s)=dsjs = Rcr is called the
=1

critical rate. Then the random coding error exponent can be represented as
(R R ; 0  R < Rcr ;
EB;rc(R) = (25)
0

EB;sp (R) ; Rcr  R < C ;

10
 For small rates the random coding bound can be strengthened. To prove
the stronger bound we need to prove a lower bound for the minimum distance
dE;min . Both this lower bound and the proof of it are given in Appendix C,
as well as a corollary.
We will now present the expurgated bound for the decoding error probabil-
ity. There are two ways of proving this bound. The rst way is to expurgate
\bad" code words from the code, see [14]. The second way is to expurgate
\bad" codes from the ensemble. In the rst case we violate the linearity of
the original code and the method is not applicable to convolutional codes.
We use the second method in our proof, given in Appendix C.
Let
1
 1 X
q 1

Gex(s) = s ' 4sN ln 2 = s log q 2

def (1 i=q))Es =(2N0 s ln 2)
cos(2
(26)
2
0
i =0

be the expurgation function of the channel, and let


Rex = G0ex(s)s =1
(27)
be the expurgation rate of the channel.
Theorem 4 (Expurgated Bound for Block Codes) For the AWGN
channel with q-PSK signaling, there exists a rate R block code of length N ,
whose decoding error probability is upperbounded by the inequality
Pe < 2 EB;ex (R)N ; 0  R < Rex + o(1) ; (28)
where the expurgated block error exponent EB;ex (R) is dened by the para-
metric equation
(E sG0ex(s) ;
B;ex (s) = Gex(s)
1<s<1; (29)
R(s) = G0ex(s) + o(1) ;
and o(1) ! 0 when N ! 1. 2

11
 The function
(E 0  R < Rex ;
B;ex (R)
;
E B (R ) = (30)
EB;rc(R) ; Rex  R < C ;
is presented in Figure 1 for q = 32, Es=N = 10dB. 0

V Trellis Coding Upper Bounds for the Re-

liability Function
Let
u = u ; u ; :::; un; ::: = u ; u ; :::; ui; :::
0 1 0 1 (31)
where
un = unb; unb ; :::; u n
+1 ( +1) b 1 ; ui 2 Zq ; (32)
be the q-ary information sequence at the input of a memory m rate R =
log (q)b=c (bits per channel use) trellis encoder, and let
2

s = s ; s ; :::; sn; ::: = s ; s ; :::si; ::: ;

0 1 0 1 (33)
where
sn = snc; snc ; :::; s n
+1 ( +1) c 1 ; si 2 S ; (34)
such that
sn = f (un; un ; :::; un m )
1 (35)
be the code sequence at the output of the trellis encoder. The received
sequence at the output of the AWGN channel with q-PSK signaling is
r = r ; r ; :::; rn; ::: = r ; r ; :::; ri; ::: ;
0 1 0 1 (36)
where
rn = rnc; rnc ; :::; r n c ;
+1 ( +1) (37) 1

and the decoded sequence is u^ = u^ ; u^ ; :::; u^n; :::. Analogously to block

0 1

transmission, the trellis transmission is characterized by two decoding error

12
probabilities, see [17]. Those are the burst error (or rst error event) prob-
ability P (burst) and the bit error probability P (bit). It can be shown that
the convolutional reliability function, also called convolutional error exponent
satises
EC (R) = mlim
def log P (burst) = lim log P (bit) ;
2 2
(38)
!1 cm m!1 cm
and we can limit our study to that of the asymptotical behavior of P (burst).
To simplify the analysis, we also consider terminated transmission of trel-
lis coded information sequences in this paper. We divide the information
1

sequence into frames u jK; j K = ujK ; ujK ; :::u j K of K b-tuples,

[ ( +1) ) +1 ( +1) 1

and each frame, followed by m dummy allzero b-tuples, is encoded by the

rate R = log (q)b=c memory m trellis encoder, described above. Then
2

N = (K + m)c is the frame length. The frame length should grow faster than
m, and we could for example choose K equal to m . Let Pe be the frame
2

error probability, i.e. the probability that at least one subblock of the frame
is decoded erroneously, and let Pb be the bit error probability in the frame,
i.e. the mathematical expectation of the fraction of erroneously decoded
bits per frame. If m ! 1 the rate of terminated transmission convergates
to the rate R of nonterminated transmission and the ratios log Pe=(cm) 2

and log Pb=(cm) both convergate to the convolutional reliability function

2

EC (R) = limm!1 log P (burst)=(cm).

2

Theorem 5 (Viterbi-Type Lower Bound) For any rate R = log (q)b=c 2

memory m trellis code, that is used to communicate over an AWGN chan-

nel with q-PSK signaling, the burst error probability is lowerbounded by the
inequality
P (burst) > 2 [EC; R o ]mc ;
sp ( )+ (1)
0R<C; (39)
1 The reason for introducing terminated transmission will be clear in the second part of
this paper, when we study the multistage decoding of multilevel trellis codes. Although
the proof of equality (38) has no principal problem for conventional trellis coding, this
is not the case for multistage decoding of multilevel codes due to the propagation of
decoding errors on high levels. Introducing termination of innite codes will limit the
error propagation to one frame.

13
where EC;sp (R) is the convolutional sphere-packing exponent, dened by the
parametric equation
( EC;sp (s) = G(s) ;
0s<1; (40)
R(s) = Gs(s) ;
G(s) is the Gallager function (9) and o(1) ! 0, if m ! 1. 2
The proof of Theorem 5 uses Theorem 1, and a sketch of the proof is given
in Appendix D.
Theorem 6 (Straight-Line Bound for Trellis Codes) For any rate
R = log (q)b=c memory m trellis code, that is used to communicate over an
2

AWGN channel with q-PSK signaling, let RC be the rate at which EC;sp (R) =
Eex(0). Then for m ! 1, the burst error probability of the trellis code is
lowerbounded by the inequality
P (burst)  2 [Eex (0)+o(1)]cm ; 0  R < RC ; (41)
where Eex (0) is the value of the expurgated exponent (71) at rate R = 0, and
o(1) ! 0, if m ! 1. 2
Analogously to the proof of Theorem 5, the proof of this theorem uses the
lower bound for the decoding error probability of block codes. It can also be
proved directly from (70).
The function
( E (0) ; 0  R < R ;
ex C
E C (R ) = (42)
EC;sp (R) ; RC  R < C ;
is presented in Figure 1 for q = 32 and ES =N = 10dB.
0

VI Trellis Coding Lower Bounds for the Re-

liability Function
The construction of a trellis code for the AWGN channel with q-PSK signaling
consists of two phases. First we construct a convolutional code over the
14
integer ring Zq modulo q and then map the symbols of the code sequences
onto symbols si 2 S .
A convolutional (time-varying) rate R = log (q)b=c memory m code over 2

Zq is dened by its generating matrix

0 G (0) G (1)


G (m) 1
m
G=@ A;
0 1
G (1)


Gm (m) Gm(m + 1)
0 1 (43)







where Gi(t), i = 0; 1; :::m, t = 0; 1; ::: are b  c matrices over Zq . We will
consider only periodically time-varying convolutional codes, i.e. codes for
which Gi (t) = Gi(t + T ), where T is the period. Let
u = u ; u ; :::; un; ::: = u ; u ; :::; ui; ::: ;
0 1 0 1 (44)
where
un = unb; unb ; :::; u n
+1 ( +1) b 1 ; ui 2 Zq ; (45)
be the q-ary information sequence at the encoder input. Then
v = v ; v ; :::; vn; ::: = v ; v ; :::; vi; ::: ;
0 1 0 1 (46)
where
vn = vnc; vnc ; :::; v n
+1 ( +1) c 1 ; vi 2 Zq ; (47)
such that
v = uG ; (48)
is the q-ary code sequence. We map symbols of v onto symbols si 2 S , as
we did in the case of block coding. We get the output code sequence
s = s ; s ; :::; sn; ::: = s ; s ; :::; si; ::: ;
0 1 0 1 (49)
where
sn = snc; snc ; :::; s n
+1 ( +1) c 1 ; si 2 S : (50)
This construction denes a trellis code with code words over the alphabet S .
15
 Consider now the random ensemble of periodically time-varying convolu-
tional codes, whose entries in the submatrices Gi(t) are i.i.d. equiprobable
q-ary random variables, and the corresponding ensemble C of trellis codes
with codewords s over the alphabet S .
Theorem 7 (Random Coding Bound for Trellis Codes) For the
AWGN channel with q-PSK signaling, there exists a rate R = log2 (q)b=c
trellis code of memory m, whose burst error probability resulting from maxi-
mum likelihood decoding is upperbounded by
8
<2 R0 +o(1)]cm
[
; 0R<R ;
P (burst)  :
0
(51)
2 [EC; R o
sp ( )+ (1) ]cm ; R R<C;
0

where EC;sp (R) is dened by (40), o(1) ! 0, when m ! 1, and R0 is the

computational cuto rate. 2
It follows from (51) that

lim log Pe = lim log Pb = lim log P (burst)

2 2 2
m!1 cm m!1 cm m!1 cm
(R ; 0R<R ;
= EC (R)  EC;rc (R) = (52)
0 0

EC;sp (R) ; R  R < C : 0

The proof of this theorem is analogous to the proof of the Yudkin-Viterbi

bound for the BSC (see, for example [17],[19]) and is therefore omitted.
For rates R < R the random coding bound can be strengthened. The
0

proof this strengthening uses the following corollary.

Corollary 1 (Costello-Type Bound) In the ensemble C , there exists a
rate R, memory m trellis code, for which R = R^ + o(1), dE;free  d^ + o (1), 2

where R^ and d^ satisfy (94), o (1) ! 0 and o (1) ! 0, when m ! 1.

1 2 2
In Appendix E the proof of this corollary as well as a sketch of the proof of
the following theorem are given.

16
Theorem 8 (Expurgated Bound for Trellis Codes) For the AWGN
channel with q-PSK signaling, there exists a rate R = log2 (q)b=c trellis code
of memory m whose burst error probability resulting from maximum likelihood
decoding is upperbounded by
P (burst) < 2 EC;ex (R)cm ; 0R<R ; 0 (53)
where the expurgated convolutional exponent is dened by the parametric
equation
(
EC;ex (s) = Gex(s) + o (1) ;
1
1<s<1: (54)
R(s) = Gex(s)=s + o (1) ; 2

Here Gex(s) is the expurgation function of the channel, R0 is the computa-

tional cuto rate, and o1 (1) ! 0, o2 (1) ! 0 when m ! 1. 2
From theorems 7 and 8 follows that there exist periodically time-varying
trellis codes for which the reliability function EC (R) is lowerbounded by
E C (R), where
(E 0<RR ;
C;ex (R)
;
E C (R ) = (55)
0

EC;rc(R) ; R R<C:
0

The function E C (R) is presented in Figure 1 for q = 32, Es=N = 10dB.

0

VII The State-Complexity Error Exponent

Now, let us dene the decoding state-complexity as the number of states 
of the maximum likelihood (Viterbi) decoder. This number is equal to 2Rcm.
The value  denes the complexity of a Viterbi decoder, rst, since it is
proportional to the memory size of the Viterbi decoder, and, second, since it is
proportional to the number of computations the Viterbi decoder performs in
each decoding step. In the case of transmission of terminated frames of length
m , the number of computations for a frame is O( log ). We can say that
2
2

17
the asymptotical complexity of the frame decoding is O( 2 2  = 2 ) = 1+(log log ) log

 o , where o(1) ! 0, when  ! 1.

1+ (1)

We also introduce the parameter = lim!1 log Pe= log , which we

def
2 2

will call the state-complexity error exponent . Since 2

Pe =  +o(1) ; (56)
where o(1) ! 0 if  ! 1, denes the asymptotical complexity-reliability
tradeo for our coded modulation scheme. It follows from equality (42) that
the upper bound for is
(E ; 0  R < RC ;
= E CR(R) =
ex (0)
R
def
(57)
s ; RC  R < C ;
where s satises R = G(s)=s. The lower bound for follows from inequality
(55),
= E CR(R) = s ;
def
(58)
where s satises R = Gex(s)=s for 0  R < R and R = G(s)=s for R  R <
0 0

C . The bounds (R) and (R) for q = 32, Es=N = 10dB are given together
0

with the corresponding bounds for the state-complexity error exponent of

the multilevel modulation scheme in the second part of this paper.

VIII Concluding Remarks

In this rst part of the paper we have studied the asymptotical perfor-
mances of a conventional coded modulation system with transmission over
the AWGN channel with PSK signaling. We omit the proofs of bounds in
those cases when the bounds can be considered as generalizations of known
bounds for the memoryless channel with discrete input and output. We stud-
ied the bounds for the code minimum and free distances, and the expurgated
2 The authors are thankful to Professor Jim Massey for suggesting the term state-
complexity error exponent, and the term eective capacity which is introduced in the
second part of this paper.

18
bounds both for block and trellis coding cases in more detail. The state-
complexity error exponent, which will be important in the comparison with
the multilevel modulation scheme provided in the second part of this paper,
was introduced.

Appendix A Sketch of proof of Theorem 1

To prove Theorem 1 we partition the channel output space into p regions,
R ; R ; :::; Rp. Then we apply the Shannon, Gallager and Berlekamp the-
1 2

orem [15] to the discrete memoryless channel with q-ary input alphabet
and p-ary output alphabet. If p increases the partitioning of the chan-
nel output space becomes increasingly ne. Letting p ! 1 completes the
proof. A complication in the proof appears, since for any xed p we have
p p
O (1= N ) = O ( log Pmin = N ), where Pmin is the smallest nonzero tran-
2 2 2

sition probability of a transmitted symbol to one of p channel output regions.

Therefore the partition of the channel output space and the value of p should
p
be chosen for each N , such that log Pmin = N ! 0, when N ! 1.
2

Appendix B Proof of Theorem 2

First we will derive the Plotkin-type bound for the Euclidean distance in
the case of PSK signaling. Let dE (x; y) be the squared Euclidean distance
between two points x and y in the complex plane, and let
X
N 1

dE (s; s0 ) = dE (sn; s0n) (59)

n=0
be the squared Euclidean distance between the code words
s = s ; s ; :::; sN
0 1 1 (60)
and
s0 = s0 ; s0 ; :::; s0N :
0 1 1 (61)
19
The minimum distance dE;min is dened as the minimum value of dE (s; s0)
over all code words s and s0 , such that s 6= s0 .
Consider the listing of M code words of length N , an array with M
rows and N columns, where the m-th code word is the m-th row of the
array. Take one column of this array and suppose that the symbol xi 2 S ,
i = 0; 1; :::; q 1, occurs ai times in this column. Then the contribution of
this column to the sum of distances between all ordered pairs of code words
is q X
X q 1 1

(a ; a ; :::; aq ) =
0 1 ai aj dE (xi; xj ) ;
1 (62)
i=0 j =0
where "   #
dE (xi; xj ) = Es cos 2q i cos 2q j + sin 2q i sin 2q j
2 2

 2 

= 2Es 1 cos q (i j ) : (63)
The function (62) of a ; a ; :::; aq is upper bounded by the maximum value
0 1 1
P
of the right-hand side of (62) under the constraints iq ai = M , 0  ai  1
=0

M , where ai , i = 0; 1; : : : ; q 1, are not necessarily integers. This maximum

value of (
) is achieved if and only if ai = M=q, i = 0; 1; :::; q 1. This can
be proved by application of the Karush-Kuhn-Tucker theorem [16]. Summing
the squared Euclidean distances over all N columns, we have
XX NM q X
X q 1 1

dE (xi ; xj ) = NM dE ;
2
d E ( s; s )  q
0
2
(64) 2

all s s
all 0 i j =0 =0

where
dE = 1
Xq X q 1 1

q 2
dE (xi ; xj ) = 2Es (65)
i=0 j =0
is the average squared Euclidean distance between symbols of the alphabet
S . It should be noted that dE does not depend on q.
Since dE;min is the squared Euclidean distance between a certain pair of
code words and since there are M (M 1) ordered pairs, we have
M (M 1) dE;min  NM 2Es ; 2
(66)
20
or
2NM E :
dE;min  M (67)
1 s
Thus, we have proved the following
Lemma 1 (Plotkin-Type Bound) In the alphabet (1), the minimum
squared Euclidean distance for any block code of M code words and block
length N satises inequality (67), where Es is the signal energy per channel
use. 2
From Lemma 1 follows that asymptotically, if M ! 1 when N ! 1,
dE;min  2N (Es + o(1)) ; (68)
where o(1) ! 0 when N ! 1.
Let us now dene Pe;min as the decoding error probability for the worst
code word in a code, minimized over all codes of given block length N and
number of code words M . Since the error probability of a code with two code
words, at squared Euclidean distance dE;min , is equal to
rd !  d 
E;min E;min
Q 2N = exp 4N+ o(dE;min ) ; (69)
0 0

when transmission is over an AWGN channel with two-sided noise power

density N =2, we get from (68) that for N ! 1
0

Pe;min  2 Eex (0)+o(1)]N

[
; (70)
where
Eex(0) = 2NEsln 2 (71)
0

is the expurgated error exponent, evaluated at R = 0. The term expurgated

originally comes from the name of the upper bounds for the decoding error
probability, whose proof uses the method of expurgating \bad" codes from
an ensemble.

21
 Equation (70) establishes an upper bound for the reliability function
EB (R)  Eex(0) (72)
for all rates, including rate zero. To strengthen the lower bound of Pe for
low rates, we use the following remarkable property [15]. If we have a point
on the sphere-packing upper bound exponent (13) and another on any other
asymptotic upper bound exponent, the straight line connecting these points
is an asymptotic upper bound exponent for all intermediate rates. This
suggests, that we connect the asymptotical bound point at rate zero with
the sphere-packing bound, by a tangent to the curve of the sphere-packing
bound exponent. Theorem 2 summarizes this bound.

Appendix C Proof of Theorem 4

In this appendix we will prove a lower bound for the minimum distance
dE;min, to be used in the proof of Theorem 4. Consider an arbitrary nonzero
code word s and its squared Euclidean distance dE (0; s) from the allzero
code word. Let F () = E [2 dE 0;s ],  > 0, where the expectation is over
( )

the random ensemble of codes, be the characteristic function of dE (0; s).

Then
(
 )

F () = E [2 dE 0;s ] = E [2 dE ;s ] N = 2'  N ; (73)
(0 ) ( )

where (see (63))

X
q
'() = log 1q 2 Es
1
i=q ;
2
>0:(1 cos(2
(74)
))
2
i =0

Using Cherno bounding arguments we get that the probability of the event
dE (0; s)  d^ is upperbounded by the inequality
 
P dE (0; s)  d^ < 2dF () = 2d ^ 'N
^+ ( )
;  > 0: (75)
The probability that, at least for one nonzero code word s, the squared
Euclidean distance from the allzero code word is less than d^, that is the
22
probability that the minimal squared Euclidean distance of the code dE;min
is less than d^ is upperbounded by the union bound
 
P dE;min < d^ < 2RN d^+'()N
+
: (76)
For any given d^, let us choose  =  to minimize the right-hand side of (76),
0

i.e.  satises the equality

0

d^ = '0 ( ) : (77)
N 0

Then from (76) follows that for rate R, and any ; 0   < 1, such that
R = '( ) +  '0( ) + N1 log (1 ) ;
0 0 0 2 (78)
we have  
P dE;min < d^ < 1  : (79)
From (79) follows that more than a fraction  of the codes in the ensemble
S have minimal squared Euclidean distance larger than d^. We proved:
Lemma 2 Let R, d^ and , 0   < 1, satisfy (77)-(78) for some  . Then
0

more than a fraction  of the rate R, block length N codes in the ensemble
S have minimal squared Euclidean distance dE;min > d^. 2
Corollary 2 (Gilbert-Varshamov-Type Bound) In the ensemble S
there exists a rate R, block length N code, whose minimal squared Euclidean
distance is lower bounded by the inequality dE;min > d^, where d^ and R satisfy
the parametric equation
( d^
N = '0 () ;
>0: (80)
R = '() + '0() ;
2
Now we give a proof of Theorem 4. Using Lemma 2, let  = 1=2 and
consider, in the ensemble S , the sub-ensemble S 0 of codes, whose minimum
23
distance dE;min > d^, and where d^ and R satisfy (77)-(78) with  = 1=2. This
sub-ensemble includes at least half of the codes in S . Let the allzero code
word be transmitted over the channel. We will say that the code word s
causes a decoding error, if
dE (s; r)  dE (0; r) : (81)
The probability of this event is
0s 1  d (0; s) 
P (error caused by s) = Q @ dE ( 0; s ) A < exp E : (82)
2N 0 4N 0

The conditional probability of the decoding error given that the code belongs
to the sub-ensemble S 0 is upperbounded by the inequality
  d (0; s) 
E
Pe(S ) < 2 ES exp
0 RN 0

4N
; (83)
0

where ES f
g denotes mathematical expectation over S 0 . Since the set of

0

code words s of the ensemble S , such that dE (0; s) > d^, include all the code
words of the codes of the sub-ensemble S 0 , and some code words that are not
in S 0 , we have
X  d
0
Pe(S ) < 2
2RN P (dE (0; s) = d) exp 4N ; (84)
0
d>d ^

where P (dE (0; s) = d) is the probability that a code word in the ensemble S
has squared Euclidean distance d from the allzero code word. Using Cherno
bounding arguments we get from (84) for  > 0 the inequality
X  d
Pe (S 0 ) < 2
2RN P (dE (0; s) = d) exp 4N 2 d ( d^)
d 1
all
 0

= 2
2RN F 4N ln 2  2 d^
0
1
= 2
2RN ' 4N0 ln 2  N d : (85)
+ ( ) ^

24
Inequality (85) is valid for all  in the interval 0 <   1=(4N ln 2), partic- 0

ularly for
 = 4N 1ln 2  ;
0 0   4N 1ln 2 ; 0 (86)
0 0

where  satises (77). We note, that for  = 1=(4N ln 2) we have '( )+

0 0 0 0

 '0( ) = Rex. For  =  we get from (77), (78) and (85)

0 0 0

^
!
Pe(S 0 ) < 2
21
exp 4Nd ; (87)
 1  1
 1  1
0

0
0  R  ' 4N ln 2 + 4N ln 2 ' 4N ln 2
0 N: 0 0

From (87) follows that, in the given interval of transmission rates R, there
exists at least one code in the sub-ensemble S 0 , for which
!
Pe < exp d^ : (88)
4N 0

From (77), (78), (26), (27) we get that

d^ = E = G (s) sG0 (s) ;
def
B;ex ex ex
4N N 0

R = G0ex(s) N1 ; (89)
for R  G0ex(s)js =1 1=N . This is equivalent to (29).

Appendix D Sketch of proof of Theorem 5

There are several ways of conducting the proof of Theorem 5 (see, for ex-
ample, [18],[19]). The simplest way is by contradiction and is given in [17]
for convolutional codes over a BSC. Let us assume that inequality (39) does
not hold, i.e. there exists a rate R memory m trellis code, whose burst error
probability does not satisfy (39) when m ! 1. Then we can always con-
struct a sequence of a block code of rate r, such that the block reliability
function and the block length satisfy equalities

25
 EC;sp (R) = R R r EB;sp (r) ; (90)
N = EEC;sp ((Rr)) mc : (91)
B;sp
One can check that the decoding error probabilities for this sequence of a
block code violates the bound (10). This contradiction proves Theorem 5.

Appendix E Proof of Corollary 1 and Theorem 8

First we will prove Corollary 1. Let dE (s; s0) be the squared Euclidean dis-
tance between two dierent code sequences s and s0 , and let
dE;free = min
def

s6 s
0
dE (s; s0 )
=
(92)
be the free distance of the code. Since all code words have the same spectrum
dE;free = min
s
dE (0; s) : (93)
Let '() be the characteristic function (74), and let R^ and d^ be connected
by the parametric equation
R^ = '() ;
(94)
d^ = '  cm ;
( )

where 0 <  < 1.

Lemma 3 Let R^ and d^ satisfy (94) and let R = R^ 1=m = R^ + o (1), 1

d = d^ (3=) log m = d^+ o (1)m, T = m , where o (1) ! 0 and o (1) ! 0,

0 2 2
2
1 2

when m ! 1. Then more than a fraction (1 ln 2=c) of the rate R memory

m codes in the ensemble C have free distance dE;free > d . 0 2
The methods for proving lower bounds for the free distance for periodi-
cally time-varying convolutional codes are described in [17], Section 3.7. To
prove the bound we have to nd, for each t, t = 0; 1; :::; T 1, where T is
26
the period of the periodically time-varying convolutional code, and for each
i, i = 1; 2; :::; T m, the probability in C that at least one of the 2Rci paths s
of length c(m + i) (restricted length paths), stemming from the t-th node will
have the distance dE (0; s) smaller than d . This probability is upperbounded
0

by (comp. with (76))

2Ric d0 '  m i c
+ + ( )( + )
(95)
and we have from (94), (95)
X
T m
ci
P (dE;free < d ) < T
0 2Ric ^
m +d^ 3 log 2 m+'()(m+i)c
i=1
c=m
< m1 1 2 2 c=m < lnc2 :
( )

( )
(96)
For a strong proof it is necessary in addition to (96) to upperbound the
probability that paths of length greater than T have a distance larger than
d from the allzero path. This probability is exponentially decreasing with
0

T , T = m , and we can neglect this probability for large m. This completes

2

the proof of the lemma.

Since the right-hand side of (96) is less than 1, Corollary 1 follows directly
from Lemma 3.
We will now use Lemma 3 to give a sketch of the proof of the expurgated
bound for trellis codes. To prove the theorem we have to consider the sub-
ensemble C 0 of the ensemble C of codes, whose free distance dE;free > d = 0

d^ (3=) log m and rate R = R^ 1=m, where R^ and d^ satisfy (94). Then
2

we upperbound the average burst error probability over the codes of the sub-
ensemble C 0. The proof is analogous to the proof of the expurgated bound
for block codes (Theorem 4), and is given in [17] for convolutional codes and
the BSC. Here we give a sketch of the proof.
For each t, t = 0; 1; :::T 1, where T is the period of the code, we nd
the upper bound for the probability that an error burst starts in the t-th
node. Then, we have to consider error bursts of two kinds. First, bursts with

27
restricted length, not larger than T , and second bursts which have a length
larger than T . The upperbounding of the probability of the restricted length
error burst is done below. The probability of error burst having length larger
than T , T = m , is upperbounded by a function exponentially decreasing
2

with T , and consecutively we can neglect this probability in comparison to

the rst, which is, as we will see, decreasing exponentially with m.
The conditional probability of restricted length error bursts, given that
the code belongs to the sub-ensemble C 0 is upperbounded by (compare with
(83)-(85) and (95), (96))

T X
T m
' 4N01ln 2 )(m+i)c d0
Pe (C 0 ) < 2Ric + (

1 ln 2
c i=1
m 2 X
T m ic 1 3
< 2Ric m +'( 4N0 ln 2 )(m+i)c d^+  log2 m ; (97)
^

1 ln 2
c i=1
where 0 <  < 1=(4N ln 2). Let us choose
0

 = 4N 1ln 2  ; (98)
0

where , 0 <   1=(4N ln 2), satises (94). We note that  = 1=(4N ln 2)

0 0

corresponds to '() = R . Then from (97) we get for 0 < R < R

0 0

3
Pe (C 0 ) < m  2 c=m
2
2+ 1
4N0 ln 2 d^
1 c 1 2 c=mln 2

= 2 '  o1 cm ;
[ ( )+ (1)]
(99)
where o (1) ! 0 when m ! 1. The bound
1

Pe (C 0 ) < 2 [Gex(s)+o1 (1)]cm ; (100)

which in combination with equality
R^ = '() = Gexs(s) (101)
is equivalent to (53), follows from (99) and (26).

28
References
[1] G. Ungerbck, "Channel coding with multilevel/phase signals," IEEE
Transactions on Information Theory, vol. IT-28, pp. 55{67, Jan. 1982.
[2] H. Imai and S. Hirakawa, "A new multilevel coding method using error-
correcting codes," IEEE Transaction on Information Theory, vol. IT-23,
pp. 371{377, May 1977.
[3] J. Huber, "Multilevel codes: Distance proles and channel capacity," in
ITG-Fachbereicht 130 (conf. rec.), 1994, pp. 305{319.
[4] J. Huber and U. Wachsmann, "Capacities of equivalent channels in mul-
tilevel coding schemes," Electronics Letters, vol. 30, pp. 557{558, no. 7
Mar. 1994.
[5] Y. Kofman, E. Zehavi, and S. Shamai, "Performance analysis of a mul-
tilevel coded modulation system," IEEE Transactions on Communica-
tions, vol. 42, pp. 299{312, Feb./Mar./Apr. 1994.
[6] H. Herzberg, "On the spectrum of distances of a multilevel code, decoded
by a multistage decoder," IEEE Transactions on Information Theory,
vol. IT-43, pp. 1736{1739, Sep. 1997.
[7] K. Engdahl and K. Sh. Zigangirov, "On the calculation of the error
probability for a multilevel modulation scheme using QAM-signaling,"
in Proceedings ISIT, 1997, p. 390.
[8] K. Engdahl and K. Sh. Zigangirov, "On the calculation of the error
probability for a multilevel modulation scheme using QAM-signaling,"
IEEE Transactions on Information Theory, vol. IT-44, pp. 1612{1620,
Jul. 1998.

29
 [9] K. Engdahl and K. Sh. Zigangirov, "An analysis of the error probability
for a multilevel modulation scheme using PSK-signaling," in Proceedings
ICT, 1998, pp. 450{454.
[10] K. Engdahl and K. Sh. Zigangirov, "The Cherno bounding parameter
for a multilevel modulation scheme using PSK-signaling," International
Journal of Electronics and Communication (AEU)  , pp. 18{24, no. 1
1999.
[11] K. Engdahl and K. Sh. Zigangirov, "A comparison analysis of two mul-
tilevel QAM Systems," in Proceedings ISIT, 1998, p. 209.
[12] K. Engdahl and K. Sh. Zigangirov, "A comparison analysis of hexago-
nal multilevel QAM and rectangular multilevel QAM," Probl. Peredach.
Inform., vol. 34, pp. 39{50, no. 4 Oct./Nov./Dec. 1998.
[13] K. Sh. Zigangirov, and R. Johannesson \A Trellis Coding Scheme based
on Signal Alphabet Splitting," Probl. Peredach Inform., vol. 28, pp 14{
23, no. 4 Oct./Nov./Dec. 1992.
[14] R. G. Gallager, Information Theory and Reliable Communication, Wi-
ley, New York, 1968.
[15] C. E. Shannon, R. G. Gallager, and E. R. Berlekamp, \Lower Bounds
to Error Probability for Coding on Discrete Memoryless Channels," In-
formation and Control, vol. 10, pp. 65{103 (Part I), pp. 522{552 (Part
II), 1967.
[16] H. Bazarna, H. Sherali, and C. Shetty, Nonlinear Programming Theory
and Algorithms, Second edition, Wiley, New York, 1993.
[17] R. Johannesson and K. Sh. Zigangirov, Fundamentals of Convolutional
Coding, IEEE Press, Piscataway N.J., 1999.

30
[18] A. J. Viterbi, \Convolutional Codes and an Asymptotically Optimum
Decoding Algorithm," IEEE Transactions on Information Theory, vol.
IT-13, pp. 260{266, Apr. 1967.
[19] A. J. Viterbi and J. K. Omura, Principles of Digital Communication
and Coding, McGraw Hill, 1979.

31
 LIST OF FIGURE CAPTIONS
Fig. 1: Lower and upper bounds for asymptotical block and convolutional
reliability functions for q = 32, Es=N = 10dB. The solid lines are the
0

bounds for convolutional coding E C and E C , and the dashed lines are
the bounds for block coding E B and E B . We also marked the rates
RB , Rex, RC , Rcr , R and C .
0

32
 LIST OF FOOTNOTES
1 The reason for introducing terminated transmission will be clear in the
second part of this paper, when we study the multistage decoding of
multilevel trellis codes. Although the proof of equality (38) has no
principal problem for conventional trellis coding, this is not the case
for multistage decoding of multilevel codes due to the propagation of
decoding errors on high levels. Introducing termination of innite codes
will limit the error propagation to one frame.
2 The authors are thankful to Professor Jim Massey for suggesting the term
state-complexity error exponent, and the term eective capacity which
is introduced in the second part of this paper.

33
 BIOGRAPHIES
Gerd Beyer was born in Aalen, Germany, in 1973. He received the M.Sc.
degree in computer science from the University of Ulm, Germany, in
1999. Working at his master thesis, during 1998-1999, he spent six
months at the Department of Information Technology at Lund Univer-
sity, Sweden. Since November 1999 he has been working as consultant
at Beck et al. in Munich, Germany.
Karin Engdahl (S'96) was born in Sweden in 1971. She received her
M.Sc. degree in Electrical Engineering from Lund University, Sweden,
in 1995. Since then she is with the Dept. of Information Technology,
Lund University, where she is working towards a Ph.D.
Kamil Sh. Zigangirov (M'95-SM'99-F'2001) was born in the USSR in
1938. He received the M.S. degree in 1962 from the Moscow Institute
for Physics and Technology and the Ph.D. degree in 1966 from the
Institute of Radio Engineering and Electronics of the USSR Academy
of Sciences, Moscow.
From 1965 to 1991 he held various research positions at the Institute
for Problems of Information Transmission of the USSR Academy of
Sciences, Moscow, rst as a Junior Scientist, and later as a Main Sci-
entist. During this period he visited several universities in the United
States, Sweden, Italy, and Switzerland as a guest researcher. He orga-
nized several symposia on information theory in the USSR. In 1994 he
received the Chair of Telecommunication Theory at Lund University,
Lund, Sweden. His scientic interests include information theory, cod-
ing theory, detection theory, and mathematical statistics. In addition
to papers in these areas he has published a book on sequential decoding
of convolutional codes. With R. Johannesson he coauthored the text-
book "Fundamentals of Convolutional Coding" (Piscataway, NJ: IEEE

34
Press, 1999).

35
 8

Eex(0) R =1.95
C
7

E (R)
C,sp
5
EB,sl(R)
E(R)

4 E (R)
C,ex

3
R =2.45
0

RB=1.39 EC,sp(R)
2
E (R)
B,sp
E (R)
1 B,ex EC,rc(R)

R =1.68
ex R =2.18
E (R) cr
B,rc C=2.75
0
0 0.5 1 1.5 2 2.5 3
R

Figure 1:

36