Coding Theory and Applications (I):

A Quick Introduction to Coding Theory

Pre-Conference Tutorial, AMC 2005
San Ling
Division of Mathematical Sciences
School of Physical and Mathematical Sciences
Nanyang Technological University, Singapore
0-0
1 Introduction
Information media not absolutely reliable
because of noise or interference.
Coding theory detects/corrects errors.
source coding and channel coding.
Source coding changing message source to
a suitable code to be transmitted through the
channel. E.g., ASCII code.
0-1
A simple communication model:
Message Source Receiver

Source Encoder Channel Source Decoder
0-2
Example 1.1
apple 00, banana 01,
cherry 10, grape 11.
Suppose apple (00) transmitted.
Assume 01 received.
apple banana

00 Channel 01

noise
0-3
Receiver may not realize message was cor-
rupted.
Channel coding encode message again after
source coding by introducing some form of re-
dundancy so that errors can be detected or
even corrected.
0-4
Message Source Receiver

Source Encoder Source Decoder

Channel Encoder Channel Channel Decoder
0-5
Example 1.2
In Example 1.1, introduce redundancy of 1
bit:
00 000, 01 011,
10 101, 11 110.
Suppose apple (000) is transmitted over a
noisy channel, and only one error.
Received word is 100, 010 or 001 none among
encoded messages.
0-6
Detects errors at the cost of reducing trans-
mission speed (transmit 3 bits for a message
of 2 bits).
Does not correct errors.
E.g., if 100 is received, then do not know whether
100 comes from 000, 110, or 101.
With more redundancy, can correct errors. E.g.,
00 00000, 01 01111,
10 10110, 11 11001.
0-7
Suppose apple is transmitted over a noisy
channel, and only one error.
Received word must be one of: 10000, 01000,
00100, 00010, 00001.
Assume 10000 received.
Can be sure that 10000 comes from 00000 be-
cause there are at least two errors between
10000 and each of the others: 01111, 10110
and 11001.
0-8
apple apple

00 00000

00000 Channel 10000

noise
0-9
Goal of channel coding to construct encoder
and decoder for
1. fast encoding of messages,
2. easy transmission of encoded messages,
3. fast decoding of received messages,
4. maximum transfer of information per unit
time, and
5. maximal detection or correction capabil-
ity.
0-10
Mathematically, the primary goal 4 & 5 (but
not compatible).
Therefore, need trade-o.
0-11
2 Error Detection, Correction and
Decoding
2.1 Some denitions
Denition 2.1
A = a
1
, a
2
, . . . , a
q
set of size q, code
alphabet; its elements code symbols.
(i) q-ary word of length n over A: sequence
w = w
1
w
2
w
n
or vector (w
1
, . . . , w
n
) in
A
n
.
0-12
(ii) q-ary block code of length n over A: non-
empty set C of A
n
.
(iii) Element of C: codeword in C.
(iv) Number of codewords in C, denoted by [C[
size of C.
(v) Code of length n and size M: (n, M)-code.
0-13
Example 2.2
Code over F
2
= 0, 1 binary code.
(i) C
1
= 00, 01, 10, 11: (2,4)-code.
(ii) C
2
= 000, 011, 101, 110: (3,4)-code.
Code over F
3
= 0, 1, 2 ternary code.
0-14
2.2 Hamming Distance
Denition 2.3
x, y: words of length n over A.
(Hamming) distance from x to y, de-
noted by d(x, y) number of places at which
x and y dier.
0-15
Example 2.4
Let A = 0, 1 and let x = 01010, y =
01101, z = 11101, then
d(x, y) = 3
d(y, z) = 1
d(z, x) = 4
0-16
2.3 Nearest Neighbour/Minimum Distance
Decoding
Suppose codewords fromC are being sent over
a communication channel.
If x is received, nearest neighbour de-
coding rule (or minimum distance de-
coding rule) will decode x to c
x
such that
d(x, c
x
) = min
cC
d(x, c). (1)
0-17
For received word x, if two or more codewords
c
x
satisfy (1), then
complete decoding rule arbitrarily selects
one to be most likely word sent,
incomplete decoding rule requests for re-
transmission.
0-18
Remark 2.5
Assume that it is much more likely that a
symbol is sent correctly than if an error occurs.
Example 2.6
Suppose codewords from binary code
C = 0000, 0011, 1000, 1100, 0001, 1001
are being sent. Assuming x = 0111 is re-
ceived, then
0-19
d(0111, 0000) = 3
d(0111, 0011) = 1
d(0111, 1000) = 4
d(0111, 1100) = 3
d(0111, 0001) = 2
d(0111, 1001) = 3
By using nearest neighbour decoding, we de-
code x to 0011.
0-20
Example 2.7
C = 000, 011: binary code. Incomplete
decoding table for C is as follows, where
means that a retransmission is sought:
0-21
received x d(x, 000) d(x, 011) decode to
000 0 2 000
100 1 3 000
010 1 1
001 1 1
110 2 2
101 2 2
011 2 0 011
111 3 1 011
0-22
2.4 Distance of a Code
Denition 2.8
[C[ 2. (Minimum) distance of C,
denoted by d(C)
d(C) = mind(x, y) : x, y C, x ,= y.
Denition 2.9
Code of length n, size M, and distance d:
(n, M, d)-code.
n, M, d: parameters of the code.
0-23
Example 2.10
C = 00000, 00111, 11111: binary code,
then d(C) = 2 since
d(00000, 00111) = 3,
d(00000, 11111) = 5,
d(00111, 11111) = 2.
C: binary (5,3,2)-code.
0-24
Denition 2.11
u: positive integer.
C: u-error-detecting if, whenever a code-
word incurs at least one but at most u errors,
the resulting word is not a codeword.
C: exactly u-error-detecting if it is u-
error-detecting but not (u+1)-error-detecting.
0-25
Example 2.12
Binary C = 00000, 00111, 11111: 1-error-
detecting since:
00000 00111 needs to change 3 bits,
00000 11111 needs to change 5 bits,
00111 11111 needs to change 2 bits.
C exactly 1-error-detecting: changing the rst
two positions of 00111 will result in another
codeword 11111 (so C not 2-error-detecting).
0-26
Theorem 2.13
C: u-error-detecting if and only if d(C)
u + 1, i.e.,
a code with distance d is an exactly (d1)-
error-detecting code.
0-27
Denition 2.14
v: positive integer.
C: v-error-correcting if minimum dis-
tance decoding is able to correct v or fewer er-
rors, assuming that incomplete decoding rule
is used.
C: exactly v-error-correcting if it is v-
error-correcting but not (v+1)-error-correcting.
0-28
Example 2.15
Binary C = 000, 111.
Using minimum distance decoding,
if 000 is sent and one error occurs, then re-
ceived word (100, 010 or 001) will be decoded
to 000;
if 111 is sent and one error occurs, then re-
ceived word (110, 101 or 011) will be decoded
to 111.
Hence, C is 1-error-correcting.
0-29
If 2 errors occur, the decoding rule may
produce the wrong codeword.
E.g., if 000 is sent and 011 is received, then
011 will be decoded to 111 using minimum
distance decoding.
Hence, C is exactly 1-error-correcting.
0-30
Theorem 2.16
C: v-error-correcting if and only if d(C)
2v + 1, i.e.,
code with distance d is an exactly (d
1)/2|-error-correcting code.
0-31
3 Linear Codes
3.1 Basic Notions
F
q
: nite eld of order q
Denition 3.1
Linear code C of length n over F
q
: sub-
space of F
n
q
.
0-32
Example 3.2
Examples of linear codes:
(i) C = (, , . . . , ) : F
q
. (Repeti-
tion code)
(ii) (q = 2) C = 000, 001, 010, 011.
(iii) (q = 3) C = 0000, 1100, 2200, 0001,
0002, 1101, 1102, 2201, 2202.
0-33
Denition 3.3
C: linear code in F
n
q
.
(i) Dual code of C:
C

= v F
n
q
: v c = 0 for all c C.
(ii) Dimension of linear code C: dim(C).
0-34
Theorem 3.4
C: linear code of length n over F
q
, then:
(i) [C[ = q
dim(C)
;
(ii) C

is a linear code and

dim(C) + dim(C

) = n;
(iii) (C

= C.
0-35
Remark 3.5
q-ary [n, k]-code or simply [n, k]-code.
Also (n, q
k
)-linear code.
If distance d of C is known, then [n, k, d]-
linear code.
0-36
Denition 3.6
C: linear code.
(i) C is self-orthogonal if C C

.
(ii) C is self-dual if C = C

.
Example 3.7
(q = 2) C = 0000, 1010, 0101, 1111 is
self-dual.
0-37
3.2 Hamming Weight
Denition 3.8
x F
n
q
. (Hamming) weight of x, de-
noted by wt(x): number of nonzero coordi-
nates in x, i.e.,
wt(x) = d(x, 0).
Lemma 3.9
x, y F
n
q
. Then
d(x, y) = wt(x y).
0-38
Denition 3.10
C: code (not necessarily linear).
Minimum (Hamming) weight wt(C) of
C: smallest of the weights of the nonzero code-
words of C.
0-39
Theorem 3.11
C: linear code over F
q
.
Then d(C) = wt(C).
Example 3.12
Binary linear C = 0000, 1000, 0100, 1100.
wt(1000) = 1,
wt(0100) = 1,
wt(1100) = 2.
Hence, d(C) = 1.
0-40
3.3 Bases for Linear Codes
Algorithm A
Input: nonempty subset S of F
n
q
.
Output: A basis for C =<S > and
C

.
Description: Form the matrix Awhose
rows are the words in S.
Use elementary row operations to place
A in RREF.
0-41
Let G be the k n matrix consisting
of all the nonzero rows of the RREF:
A

G
O

.
Rows of G give a basis for C.
G contains k leading columns.
Permute the columns of G to form
G
/
= (I
k
[X) .
0-42
Form a matrix H
/
as follows:
H
/
=

X
T
[I
nk

.
Apply the inverse of the permutation
applied to the columns of G to the
columns of H
/
to form H.
Rows of H form a basis for C

.
0-43
Example 3.13
Let q = 3. Find a basis for C

if the RREF
of A is
G =

1 2 3 4 5 6 7 8 9 10
1 0 2 0 0 2 0 1 0 2
0 0 0 1 0 1 0 0 0 1
0 0 0 0 1 0 0 2 0 0
0 0 0 0 0 0 1 0 0 1
0 0 0 0 0 0 0 0 1 2

.
Leading columns of G: 1, 4, 5, 7 and 9. Per-
0-44
mute columns of G into the order 1, 4, 5, 7, 9,
2, 3, 6, 8, 10 to form matrix G
/
= (I
5
[X), i.e.,
G
/
=

1 4 5 7 9 2 3 6 8 10
1 0 0 0 0 0 2 2 1 2
0 1 0 0 0 0 0 1 0 1
0 0 1 0 0 0 0 0 2 0
0 0 0 1 0 0 0 0 0 1
0 0 0 0 1 0 0 0 0 2

.
Form H
/
and nally rearrange columns of
0-45
H
/
using the inverse permutation to get H:
H
/
=

1 4 5 7 9 2 3 6 8 10
0 0 0 0 0 1 0 0 0 0
1 0 0 0 0 0 1 0 0 0
1 2 0 0 0 0 0 1 0 0
2 0 1 0 0 0 0 0 1 0
1 2 0 2 1 0 0 0 0 1

,
0-46
H =

1 2 3 4 5 6 7 8 9 10
0 1 0 0 0 0 0 0 0 0
1 0 1 0 0 0 0 0 0 0
1 0 0 2 0 1 0 0 0 0
2 0 0 0 1 0 0 1 0 0
1 0 0 2 0 0 2 0 1 1

.
By Algorithm A, the rows of H form a basis
for C

.
0-47
3.4 Generator and Parity-Check Matrices
Denition 3.14
(i) Generator matrix for linear code C: ma-
trix G whose rows form a basis for C.
(ii) Parity-check matrix H for linear code C:
generator matrix for dual code C

.
0-48
Denition 3.15
(i) A generator matrix of the form (I
k
[X): stan-
dard form.
(ii) A parity-check matrix in the form (Y [I
nk
):
standard form.
0-49
Lemma 3.16
C: [n, k]-linear code over F
q
, with gen-
erator matrix G.
v F
n
q
belongs to C

v orthogonal to every row of G;

i.e., v C

vG
T
= 0.
0-50
In particular, given (n k) n matrix
H, then
H: parity-check matrix for C

rows of H linearly independent

and HG
T
= O.
0-51
Theorem 3.17
C: linear code
H: parity-check matrix for C. Then
(i)
C has distance d

any d 1 columns of H
are linearly independent;
0-52
(ii)
C has distance d

H has d columns
that are linearly dependent.
0-53
Corollary 3.18
C: linear code with parity-check matrix
H.
C has distance d

any d 1 cols of H : lin indep

and
H has d cols : lin dependent.
0-54
Example 3.19
C: binary linear code with parity-check ma-
trix
H =

10100
11010
01001

.
There are no zero columns and
no two columns of H sum to 0
T
,
so any two columns of H are linearly indepen-
dent.
0-55
However, columns 1, 3, and 4 sum to 0
T
, and
hence are linearly dependent.
Therefore, the distance of C is d = 3.
0-56
Theorem 3.20
If G = (I
k
[X) is the standard form gen-
erator matrix of an [n, k]-code C,
then a parity-check matrix for C is H =
(X
T
[I
nk
).
Remark 3.21
Not every linear code has a generator ma-
trix in standard form.
0-57
3.5 Equivalence of Linear Codes
Denition 3.22
Two (n, M)-codes over F
q
are equivalent
if one can be obtained from the other by a
combination of following operations:
(i) permutation of the n digits of the codewords;
and
(ii) multiplication of the symbols appearing in a
xed position by a nonzero scalar.
0-58
Example 3.23
q = 3 and n = 3.
C = 000, 011, 022.
Permuting rst and second positions,
followed by multiplying the third position by
2,
obtain equivalent code
C
/
= 000, 102, 201.
0-59
Theorem 3.24
Any linear code C is equivalent to a lin-
ear code C
/
with a generator matrix in stan-
dard form.
0-60
4 Some Interesting Codes
A
q
(n, d) the largest possible size for all codes,
dened over any alphabet of size q, of length
n and of minimum distance d.
0-61
4.1 Hamming Bound and Hamming Codes
Theorem 4.1 (Hamming or sphere-packing
bound)
q > 1: integer
n, d: integers s.t. 1 d n
A
q
(n, d)
q
n
(d1)/2|
i=0

n
i

(q 1)
i
.
Any code meeting the Hamming bound is
called a perfect code.
0-62
Denition 4.2
r 2. Binary linear code, n = 2
r
1,
parity-check matrix H: columns all nonzero
vectors of F
r
2
binary Hamming code of
length 2
r
1. Ham(r, 2).
Remark 4.3
(i) Order of columns of H not xed.
(ii) Rows of H linearly independent since H con-
tains all r columns of weight 1 words.
Hence, H is indeed a parity-check matrix.
0-63
Example 4.4
Ham(3, 2): length 7 with parity-check ma-
trix
H =

0 0 0 1 1 1 1
0 1 1 0 0 1 1
1 0 1 0 1 0 1

.
0-64
Proposition 4.5 (Properties of binary
Hamming codes)
(i) All binary Hamming codes of a given length
are equivalent.
(ii) Dimension of Ham(r, 2): k = 2
r
1 r.
(iii) Distance of Ham(r, 2): d = 3, hence Ham(r, 2)
is exactly 1-error-correcting.
(iv) Binary Hamming codes are perfect codes.
0-65
4.2 Golay Codes
Binary Golay Codes
Denition 4.6
G: 12 24 matrix G = (I
12
[A),
where I
12
: 12 12 identity matrix
A: 12 12 matrix
0-66
A =

0 1 1 1 1 1 1 1 1 1 1 1
1 1 1 0 1 1 1 0 0 0 1 0
1 1 0 1 1 1 0 0 0 1 0 1
1 0 1 1 1 0 0 0 1 0 1 1
1 1 1 1 0 0 0 1 0 1 1 0
1 1 1 0 0 0 1 0 1 1 0 1
1 1 0 0 0 1 0 1 1 0 1 1
1 0 0 0 1 0 1 1 0 1 1 1
1 0 0 1 0 1 1 0 1 1 1 0
1 0 1 0 1 1 0 1 1 1 0 0
1 1 0 1 1 0 1 1 1 0 0 0
1 0 1 1 0 1 1 1 0 0 0 1

.
Binary linear code with generator matrix G
extended binary Golay code G
24
.
0-67
Remark 4.7
(i) Used in Voyager 1 and 2 spacecraft launched
towards Jupiter and Saturn in 1977 encod-
ing and decoding of the general science and
engineering (GSE) data for the missions.
(ii) Any code equivalent to the linear code with
generator matrix G extended binary Golay
code.
0-68
Proposition 4.8 (Properties of extended
binary Golay code)
(i) Length of G
24
: 24, dimension: 12.
(ii) Parity-check matrix for G
24
:
H = (A[I
12
).
(iii) G
24
is self-dual, i.e., G
24

= G
24
.
(iv) Another parity-check matrix for G
24
:
H
/
= (I
12
[A)(= G).
0-69
(v) Another generator matrix for G
24
:
G
/
= (A[I
12
)(= H).
(vi) Weight of every codeword in G
24
: multiple
of 4.
(vii) No codeword of wt 4, so d = 8.
(viii) G
24
: exactly 3-error-correcting code.
0-70
Denition 4.9
Binary Golay code G
23
code obtained from
G
24
by deleting last coordinate of every code-
word.
0-71
Proposition 4.10 (Properties of binary
Golay code)
(i) Length of G
23
: 23, dimension: 12.
(ii) Parity-check matrix for G
23
: 1123 matrix

H = (

A
T
[I
11
).
(iii) Distance of G
23
is d = 7.
(iv) G
23
: perfect exactly 3-error-correcting code.
0-72
Ternary Golay Codes
Denition 4.11
Extended ternary Golay code G
12
:
ternary linear code with generator matrix G =
(I
6
[B), where
B =

0 1 1 1 1 1
1 0 1 2 2 1
1 1 0 1 2 2
1 2 1 0 1 2
1 2 2 1 0 1
1 1 2 2 1 0

.
0-73
Remark 4.12
Any linear code equivalent to above code
extended ternary Golay code.
G
12
: self-dual ternary [12, 6, 6]-code
0-74
Denition 4.13
Ternary Golay code G
11
: code obtained
by puncturing G
12
in last coordinate.
G
11
satises Hamming bound
hence perfect ternary [11, 6, 5]-code.
0-75
4.3 Singleton Bound and MDS Codes
Theorem 4.14 (Singleton bound)
q > 1, n and d: integers s.t. 1 d n.
Then
A
q
(n, d) q
nd+1
.
In particular, when q: prime power,
parameters [n, k, d] of any linear code over
F
q
satisfy
k + d n + 1.
0-76
Denition 4.15
Linear code with parameters [n, k, d] such
that k + d = n + 1 maximum distance
separable (MDS) code.
0-77
Denition 4.16
MDS code C over F
q
is trivial if and only
if C satises one of:
(i) C = F
n
q
;
(ii) C equivalent to code generated by 1 = (1, . . . , 1);
or
(iii) C equivalent to dual of code generated by 1.
Otherwise, C is nontrivial.
0-78
Remark 4.17
q = 2: all MDS codes are trivial ones.
Interesting family of MDS codes: Reed-Solomon
codes.
0-79
4.4 Reed-Solomon Codes
Denition 4.18
k: positive integer,
P
k
(q): set of all polynomials of degree < k
with coecients in F
q
, i.e.,
P
k
(q) = f
0
+ f
1
x + + f
k1
x
k1
:
f
0
, f
1
, . . . , f
k1
F
q
.
0-80
Denition 4.19
F
q
= 0,
1
, . . . ,
q1
.
k q 1: positive integer,
Reed-Solomon code
RS
k
(q)
[[
(f(
1
), f(
2
), . . . , f(
q1
)) :
f(x) P
k
(q).
0-81
Remark 4.20
(i) Order of
1
, . . . ,
q1
is not important.
(ii) Used in applications like: CD players, mobile
communications, digital TV and high-speed
modems, etc.
0-82
Theorem 4.21
RS
k
(q) is linear [q 1, k, q k]-code over
F
q
, so MDS.
0-83
References
[1] R. Hill, A First Course in Coding Theory,
Clarendon Press, Oxford, 1986
[2] W.C. Human & V. Pless, Fundamentals
of Error-Correcting Codes, Cambridge
University Press, 2003
[3] S. Ling & C. Xing, Coding Theory:
A First Course, Cambridge University
Press, 2004
0-84
[4] F.J. MacWilliams & N.J.A. Sloane,
The Theory of Error-Correcting Codes,
North-Holland, 1977
[5] V. Pless, Introduction to the Theory of
Error-Correcting Codes, Third Edition,
John Wiley & Sons, 1998
0-85