EC6651 COMMUNICATION ENGINEERING

UNIT 3 PART B ANSWERS

1. ( i) Describe bandwidth SNR trade off problem of coding.(8)
(ii) Discuss various line coding techniques (May 2013)
Information transferring across channels Channel characteristics and binary symmetric
channel Average mutual information Average mutual information tells us what happens to
information transmitted across channel, or it characterises channel But average mutual
information is a bit too mathematical (too abstract) As an engineer, one would rather
characterises channel by its physical quantities, such as bandwidth, signal power and noise
power or SNR
Also intuitively given source with information rate R, one would like to know if channel is
capable of carrying the amount of information transferred across it
Channel Capacity and Channel Coding Shannons theorem: If information rate R C,
there exists a coding technique such that information can be transmitted over the channel with
arbitrarily small error probability; if R > C, error-free transmission is impossible C is the
maximum possible error-free information transmission rate Even in noisy channel, there is
no obstruction of reliable transmission, but only a limitation of the rate at which transmission
can take place Shannons theorem does not tell how to construct such a capacityapproaching code Most practical channel coding schemes are far from optimal, but
capacityapproaching codes exist, e.g. turbo codes and low-density parity check codes
Bandwidth and SNR Trade off From the definition of channel capacity, we can trade the
channel bandwidth B for the SNR or signal power S P , and vice versa Depending on
whether B or S P is more precious, we can increase one and reduce the other, an d yet
maintain the same channel capacity A noiseless analogue channel ( S P /NP = ) has an
infinite capacity C increases as B increases, but it does not go to infinity as B ; rather C
approaches an upper limit

EXAMPLE FOR BANDWIDTH AND SNR TRADEOFF

A channel has an SNR of 15. If the channel bandwidth is reduced by half, determine the
increase in the signal power required to maintain the same channel capacity

2.(i)Explain the concept of noiseless coding theorem and state its significance. (8)
The source coding theorem shows that (in the limit, as the length of a stream of independent
and identically-distributed random variable (i.i.d.) data tends to infinity) it is impossible to
compress the data such that the code rate (average number of bits per symbol) is less than the
Shannon entropy of the source, without it being virtually certain that information will be lost.
However it is possible to get the code rate arbitrarily close to the Shannon entropy, with
negligible probability of loss.
The source coding theorem for symbol codes places an upper and a lower bound on the
minimal possible expected length of codewords as a function of the entropy of the input word
(which is viewed as a random variable) and of the size of the target alphabet.
For 1 i n let si denote the word length of each possible xi. Define
where C is chosen so that q1 + ... + qn = 1. Then

(ii)Dicuss about error control codes and its applications. (8) (May 2013)
In information
theory and coding
theory with
applications
in computer
science and telecommunication, error detection and correction or error control are
techniques that enable reliable delivery of digital data over unreliable communication
channels. Many communication channels are subject to channel noise, and thus errors may be
introduced during transmission from the source to a receiver. Error detection techniques
allow detecting such errors, while error correction enables reconstruction of the original data
in many cases.

Error detection is the detection of errors caused by noise or other impairments during
transmission from the transmitter to the receiver.
Error correction is the detection of errors and reconstruction of the original, error-free
data.

Error correction may generally be realized in two different ways:

Automatic repeat request (ARQ) (sometimes also referred to as backward error

correction): This is an error control technique whereby an error detection scheme is
combined with requests for retransmission of erroneous data. Every block of data
received is checked using the error detection code used, and if the check fails,
retransmission of the data is requested this may be done repeatedly, until the data can
be verified.
Forward error correction (FEC): The sender encodes the data using an errorcorrecting code (ECC) prior to transmission. The additional information (redundancy)
added by the code is used by the receiver to recover the original data. In general, the
reconstructed data is what is deemed the "most likely" original data.

ARQ and FEC may be combined, such that minor errors are corrected without
retransmission, and major errors are corrected via a request for retransmission: this is
calledhybrid automatic repeat-request (HARQ).
Error detection scheme
Error detection is most commonly realized using a suitable hash
function (or checksum algorithm). A hash function adds a fixed-length tag to a message,

which enables receivers to verify the delivered message by recomputing the tag and
comparing it with the one provided.
There exists a vast variety of different hash function designs. However, some are of
particularly widespread use because of either their simplicity or their suitability for detecting
certain kinds of errors (e.g., the cyclic redundancy check's performance in detecting burst
errors).
A random-error-correcting code based on minimum distance coding can provide a strict
guarantee on the number of detectable errors, but it may not protect against a preimage
attack. A repetition code, described in the section below, is a special case of error-correcting
codes: although rather inefficient, a repetition code is suitable in some applications of error
correction and detection due to its simplicity.
Repetition codes
A repetition code is a coding scheme that repeats the bits across a channel to achieve errorfree communication. Given a stream of data to be transmitted, the data are divided into blocks
of bits. Each block is transmitted some predetermined number of times. For example, to send
the bit pattern "1011", the four-bit block can be repeated three times, thus producing "1011
1011 1011". However, if this twelve-bit pattern was received as "1010 1011 1011" where
the first block is unlike the other two it can be determined that an error has occurred.
A repetition code is very inefficient, and can be susceptible to problems if the error occurs in
exactly the same place for each group (e.g., "1010 1010 1010" in the previous example would
be detected as correct). The advantage of repetition codes is that they are extremely simple,
and are in fact used in some transmissions of numbers stations.
Parity bits
A parity bit is a bit that is added to a group of source bits to ensure that the number of set bits
(i.e., bits with value 1) in the outcome is even or odd. It is a very simple scheme that can be
used to detect single or any other odd number (i.e., three, five, etc.) of errors in the output. An
even number of flipped bits will make the parity bit appear correct even though the data is
erroneous.
Extensions and variations on the parity bit mechanism are horizontal redundancy
checks, vertical redundancy checks, and "double," "dual," or "diagonal" parity (used
in RAID-DP).
Checksums
A checksum of a message is a modular arithmetic sum of message code words of a fixed
word length (e.g., byte values). The sum may be negated by means of a ones'complement operation prior to transmission to detect errors resulting in all-zero messages.
Checksum schemes include parity bits, check digits, and longitudinal redundancy checks.
Some checksum schemes, such as the Damm algorithm, the Luhn algorithm, and theVerhoeff
algorithm, are specifically designed to detect errors commonly introduced by humans in
writing down or remembering identification numbers.
Cyclic redundancy checks (CRCs)
Cyclic redundancy check
A cyclic redundancy check (CRC) is a non-secure hash function designed to detect accidental
changes to digital data in computer networks; as a result, it is not suitable for detecting

maliciously introduced errors. It is characterized by specification of what is called

a generator polynomial, which is used as the divisor in a polynomial long division over
a finite field, taking the input data as the dividend, such that the remainder becomes the
result.
A cyclic code has favorable properties that make it well suited for detecting burst errors.
CRCs are particularly easy to implement in hardware, and are therefore commonly used in
digital networks and storage devices such as hard disk drives.
Even parity is a special case of a cyclic redundancy check, where the single-bit CRC is
generated by the divisor x + 1.
Cryptographic hash functions
Cryptographic hash function
The output of a cryptographic hash function, also known as a message digest, can provide
strong assurances about data integrity, whether changes of the data are accidental (e.g., due to
transmission errors) or maliciously introduced. Any modification to the data will likely be
detected through a mismatching hash value. Furthermore, given some hash value, it is
infeasible to find some input data (other than the one given) that will yield the same hash
value. If an attacker can change not only the message but also the hash value, then a keyed
hash or message authentication code (MAC) can be used for additional security. Without
knowing the key, it is infeasible for the attacker to calculate the correct keyed hash value for
a modified message.
Error-correcting codes
Forward error correction
Any error-correcting code can be used for error detection. A code with minimum Hamming
distance, d, can detect up to d 1 errors in a code word. Using minimum-distance-based
error-correcting codes for error detection can be suitable if a strict limit on the minimum
number of errors to be detected is desired.
Codes with minimum Hamming distance d = 2 are degenerate cases of error-correcting codes,
and can be used to detect single errors. The parity bit is an example of a single-errordetecting code.
Automatic repeat request (ARQ)
Automatic repeat request
Automatic Repeat reQuest (ARQ) is an error control method for data transmission that makes
use of error-detection codes, acknowledgment and/or negative acknowledgment messages,
and timeouts to achieve reliable data transmission. An acknowledgment is a message sent by
the receiver to indicate that it has correctly received a data frame.
Usually, when the transmitter does not receive the acknowledgment before the timeout occurs
(i.e., within a reasonable amount of time after sending the data frame), it retransmits the
frame until it is either correctly received or the error persists beyond a predetermined number
of retransmissions.
Three types of ARQ protocols are Stop-and-wait ARQ, Go-Back-N ARQ, and Selective
Repeat ARQ.
ARQ is appropriate if the communication channel has varying or unknown capacity, such as
is the case on the Internet. However, ARQ requires the availability of a back channel, results

in possibly increased latency due to retransmissions, and requires the maintenance of buffers
and timers for retransmissions, which in the case of network congestion can put a strain on
the server and overall network capacity.
For example, ARQ is used on shortwave radio data links in the form of ARQ-E, or combined
with multiplexing as ARQ-M.
Error-correcting code
An error-correcting code (ECC) or forward error correction (FEC) code is a process of
adding redundant data, or parity data, to a message, such that it can be recovered by a
receiver even when a number of errors (up to the capability of the code being used) were
introduced, either during the process of transmission, or on storage. Since the receiver does
not have to ask the sender for retransmission of the data, a backchannel is not required in
forward error correction, and it is therefore suitable for simplex communicationsuch
as broadcasting. Error-correcting codes are frequently used in lower-layer communication, as
well as for reliable storage in media such as CDs, DVDs, hard disks, and RAM.
Error-correcting codes are usually distinguished between convolutional codes and block
codes:

Convolutional codes are processed on a bit-by-bit basis. They are particularly suitable
for implementation in hardware, and the Viterbi decoder allows optimal decoding.
Block codes are processed on a block-by-block basis. Early examples of block codes
are repetition codes, Hamming codes and multidimensional parity-check codes. They
were followed by a number of efficient codes, ReedSolomon codes being the most
notable due to their current widespread use. Turbo codes and low-density parity-check
codes (LDPC) are relatively new constructions that can provide almost optimal
efficiency.

Shannon's theorem is an important theorem in forward error correction, and describes the
maximum information rate at which reliable communication is possible over a channel that
has a certain error probability or signal-to-noise ratio (SNR). This strict upper limit is
expressed in terms of the channel capacity. More specifically, the theorem says that there
exist codes such that with increasing encoding length the probability of error on a discrete
memoryless channel can be made arbitrarily small, provided that the code rate is smaller than
the channel capacity. The code rate is defined as the fraction k/n of k source symbols
and n encoded symbols.
The actual maximum code rate allowed depends on the error-correcting code used, and may
be lower. This is because Shannon's proof was only of existential nature, and did not show
how to construct codes which are both optimal and have efficient encoding and decoding
algorithms.
Hybrid schemes
Hybrid ARQ
Hybrid ARQ is a combination of ARQ and forward error correction. There are two basic
approaches:

Messages are always transmitted with FEC parity data (and error-detection
redundancy). A receiver decodes a message using the parity information, and requests

retransmission using ARQ only if the parity data was not sufficient for successful
decoding (identified through a failed integrity check).
Messages are transmitted without parity data (only with error-detection information).
If a receiver detects an error, it requests FEC information from the transmitter using
ARQ, and uses it to reconstruct the original message.

The latter approach is particularly attractive on an erasure channel when using a rateless
erasure code.
Applications
Applications that require low latency (such as telephone conversations) cannot
use Automatic Repeat reQuest (ARQ); they must use forward error correction (FEC). By
the time an ARQ system discovers an error and re-transmits it, the re-sent data will arrive
too late to be any good.
Applications where the transmitter immediately forgets the information as soon as it is sent
(such as most television cameras) cannot use ARQ; they must use FEC because when an
error occurs, the original data is no longer available. (This is also why FEC is used in data
storage systems such as RAID and distributed data store).
Applications that use ARQ must have a return channel; applications having no return
channel cannot use ARQ. Applications that require extremely low error rates (such as
digital money transfers) must use ARQ. Reliability and inspection engineering also make
use of the theory of error-correcting codes.

3.(i)Given states S= {S0,S1,S2,S3,S4} and their probabilities P={0.4,0.2,0.2,0.1,0.1}. Find

coding efficiency and entropy for the Huffman coding. .(8) (Nov 2013).
(ii)Give the procedure for Shannon fano coding and use the procedure to obtain the code for
the
source
symbols
.
S0,S1,S2,S3,S4,S5
with
their
respective
probabilities1/2,1/3,1/12,1/12,1/15,1/120,1/120 (8) (Nov 2013).
Entropy is given by H as follows

The entropy is H=2.12

12b)Give the procedure for Shannon fano coding and use the procedure to obtain the
code for the source symbols S0,S1,S2,S3,S4,S5 with their respective probabilities
1/2,1/3,1/2,1/15,1/120,1/120.

For example, assume we have six source symbols A, B, C, D, E, and F, with

respective probabilities 1/2, 1/3, 1/12, 1/15, 1/120, and 1/120. We could encode these using a
simple binary code: let A be 000, B 001, C 010, D 011, E 100, and F 101. However, notice
that A occurs as often as the other codewords combined, and that E and F hardly ever occur.
It would be far more efficient to have a shorter codeword for A, and longer codewords for E
and F.
Shannon-Fano Procedure
The Shannon-Fano method of encoding these symbols in binary proceeds as follows. Write
the source symbols in a column in descending probability order. Choose a point between two
symbols such that the summed probabilities above and below are as close to equal as
possible, and draw a horizontal line. Start a new column to the right of the first,
writing zeros above the line and ones below. The diagram will now look like this:

The source symbols above the line have their codewords assigned, and those below remain
uncoded. Repeat the process: choose a point between two of the remaining symbols which
equally weights probabilities, draw a horizontal line, and write zeros above and ones below:

This process continues until you have only one source symbol below the line. In this case, the
final diagram should look like:

Measuring Efficiency
Now we have a series of codewords for our symbols: A is 0, B is 10, C is 110, D is 1110, E is
11110, and F is 11111. Weighting by probability, we can calculate theaverage length of the
code:
Lave = 1/2 * 1 + 1/3 * 2 + 1/12 * 3 + 1/15 * 4 + 1/120 * 5 + 1/120 * 5
Lave = 1.767 bits
This, compared with the simplistic 3-bit binary code we first tried, uses only 60% of the
output symbols. We can calculate the efficiency of the code by comparing the average length
with the minimum length. Minimum length is defined as the entropy of the source divided
by log to the base 2 of the number base of the output; in this case, with binary output, it is
simply equal to the entropy of the source:
H(X) = - pk log2(pk)
H(X) = - 1/2 log2(1/2) - 1/3 log2(1/3) - 1/12 log2(1/12) - 1/15 log2(1/15) - 1/120 log2(1/120) 1/120 log2(1/120)
H(X) = 1.702 bits
Therefore the minimum possible length for a binary code of these symbols is 1.702 bits on
average. This means that the 3-bit binary code has an efficiency of 1.702/3 = 56.7%, while
the Shannon-Fano code has efficiency of 1.702/1.767 = 96.3%
4.Discuss the concept of coding and decoding of block codes with its mathematical
frameworks and diagram (16) (Nov 2013).
Linear Block Codes: Encoding and Syndrome Decoding
Encoding Linear Block Codes Recall that a linear block code takes k-bit message blocks and
converts each such block into n-bit coded blocks. The rate of the code is k/n. The conversion
in a linear block code involves only linear operations over the message bits to produce
codewords. For concreteness, lets restrict ourselves to codes over F2, so all the linear
operations are additive parity computations. If the code is in systematic form, each codeword
consists of the k message bits D1D2 ... Dk followed by the n k parity bits P1P2 ... Pnk,
where each Pi is some linear combination of the Dis. Because the transformation from
message bits to codewords is linear, one can represent

where D is a k 1 matrix of message bits D1D2 ... Dk, C is the n-bit codeword C1C2 ...Cn,
G is the k n generator matrix that completely characterizes the linear block code, and is
the standard matrix multiplication operation. For a code over F2, each element of the three

matrices in the above equation is 0 or 1, and all additions are modulo 2. If the code is in
systematic form, C has the form D1D2 ... DkP1P2 ... Pnk. Substituting this form , we see
that G is decomposed into a k k identity matrix concatenated horizontally with a k (n
k) matrix of values that defines the code. The encoding procedure for any linear block code is
straightforward: given the generator matrix G, which completely characterizes the code, and a
sequence of k message bits D, use Equation 6.1 to produce the desired n-bit codeword. The
straightforward way of doing this matrix multiplication involves k multiplications and k 1
additions for each codeword bit, but for a code in systematic form, the first k codeword bits
are simply the message bits themselves and can be produced with no work. Hence, we need
O(k) operations for each of n k parity bits in C, giving an overall encoding complexity of
O(nk) operations.

Syndrome Decoding of Linear Block Codes

Syndrome decoding is an efficient way to decode linear block codes. We will study it in the
context of decoding single-bit errors; specifically, providing the following semantics: If the
received word has 0 or 1 errors, then the decoder will return the correct transmitted message.
If the received word has more than 0 or 1 errors, then the decoder may return the correct
message, but it may also not do so (i.e., we make no guarantees). It is not difficult to extend
the method described below to both provide ML decoding (i.e., to return the message
corresponding to the codeword with smallest Hamming distance to the received word), and to
handle block codes that can correct a greater number of errors. The key idea is to take
advantage of the linearity of the code. We first give an example, then specify the method in
general. Consider the (7, 4) Hamming code whose generator matrix G is given by Equation
(6.3). From G, we can write out the parity equations in the

5.
Because the arithmetic is over F2, we can rewrite these equations by moving the Ps to the
same side as the Ds (in modulo-2 arithmetic, there is no difference between a and a +
sign!):

There are n k such equations. One can express these equations, in matrix notation using a
parity check matrix, H, as follows:

H is the horizontal stacking, or concatenation, of two matrices: AT, where A is the submatrix of
the generator matrix of the code from Equation (6.4), and I(nk)(nk), the identity matrix. I.e.,

Hence, for any received word r without errors, H rT = 0. Now suppose a received word r has
some errors in it. r may be written as c + e, where c is some valid codeword and e is an error
vector, represented (like c) as a 1 n matrix. For such an r,

If r has at most one bit error, then e is made up of all zeroes and at most one 1. In this case,
there are n + 1 possible values of H eT; n of these correspond to exactly one bit error, and one
of these is a no-error case (e = 0), for which H eT = 0. These n + 1 possible vectors are
precisely the syndromes introduced in the previous chapter: they signify what happens under
different error patterns. Syndrome decoding pre-computes the syndrome corresponding to each
error. Assume that the code is in systematic form, so each codeword is of the form D1D2 ...
DkP1P2 ... Pnk. If e = 100 ... 0, then the syndrome H eT is the result when the first data bit,
D1 is in error. In general, if element i of e is 1 and the other elements are 0, the resulting
syndrome H eT corresponds to the case when bit i in the codeword is wrong. Under the
assumption that there is at most one bit error, we care about storing the syndromes when one of
the first k elements of e is 1. Given a received word, r, the decoder computes H rT. If it is 0,
then there are no single-bit errors, and the receiver returns the first k bits of the received word as
the decoded message. If not, then it compares that (n k)-bit value with each of the k stored
syndromes. If syndrome j matches, then it means that data bit j in the received word was in error,
and the decoder flips that bit and returns the first k bits of the received word as the most likely
message that was encoded and transmitted. If H rT is not all zeroes, and if it does not match any
stored syndrome, then the decoder concludes that either some parity bit was wrong, or that there
were multiple errors. In this case, it might simply return the first k bits of the received word as
the message. This method produces the ML decoding if a parity bit was wrong, but may not be
the optimal estimate when multiple errors occur. Because we are likely to use single-error
correction in cases when the probability of multiple bit errors is extremely low, we can often
avoid doing anything more sophisticated than just returning the first k bits of the received word
as the decoded message. The preceding two paragraphs provide the essential steps behind
syndrome decoding for single bit errors, producing an ML estimate of the transmitted message in
the case when zero or one bit errors affect the codeword
6.Discuss in detail BSC and BEC . Find the Shannon fano code for the following seven
messages
with
probabilities
indicated.
S={S1,S2,S3,S4,S5,
S6,S7},P(S)=
{0.05,0.15,0.2,0.05,0.15,0.3,0.1} (Nov 2012).
Refer the problem 3 ii)
7.Explain the concept of block codes and coding efficiency. Find the Huffman code for
the following seven messages with probabilities indicated.S={S1,S2,S3,S4,S5,S6,S7},
P(S)= {0.05,0.15,0.2,0.05,0.15,0.3,0.1} (Nov 2012).
Refer the problem 3 i)
7.(i) With suitable examples explain Shannon-fano encoding scheme . (8)
(ii) explain Huffman coding with suitable examples (8) (May 2012)
A ShannonFano tree is built according to a specification designed to define an effective
code table. The actual algorithm is simple:
1. For a given list of symbols, develop a corresponding list of probabilities or frequency
counts so that each symbols relative frequency of occurrence is known.
2. Sort the lists of symbols according to frequency, with the most frequently occurring
symbols at the left and the least common at the right.
3. Divide the list into two parts, with the total frequency counts of the left part being as
close to the total of the right as possible.
4. The left part of the list is assigned the binary digit 0, and the right part is assigned the
digit 1. This means that the codes for the symbols in the first part will all start with 0,
and the codes in the second part will all start with 1.

5. Recursively apply the steps 3 and 4 to each of the two halves, subdividing groups and
adding bits to the codes until each symbol has become a corresponding code leaf on
the tree.
The ShannonFano algorithm doesn't always generate an optimal code. In 1952, David A.
Huffman gave a different algorithm that always produces an optimal tree for any given
symbol weights (probabilities). While the ShannonFano tree is created from the root to the
leaves, the Huffman algorithm works in the opposite direction, from the leaves to the root.
1. Create a leaf node for each symbol and add it to a priority queue, using its frequency
of occurrence as the priority.
2. While there is more than one node in the queue:
1. Remove the two nodes of lowest probability or frequency from the queue
2. Prepend 0 and 1 respectively to any code already assigned to these nodes
3. Create a new internal node with these two nodes as children and with
probability equal to the sum of the two nodes' probabilities.
4. Add the new node to the queue.
3. The remaining node is the root node and the tree is complete.
8.With suitable example explain the various line coding techniques (May 2012)
Signal

Comments

Non return to zero level. This is the standard positive logic signal
format used in digital circuits.
NRZL
1 forces a high level
0 forces a low level
Non return to zero mark
NRZM

1 forces a transition
0 does nothing
Non return to zero space

NRZS

1 does nothing
0 forces a transition
Return to zero

RZ

BiphaseL

1 goes high for half the bit period

0 does nothing
Manchester. Two consecutive bits of the same type force a transition at
the beginning of a bit period.

1 forces a negative transition in the middle of the bit

0 forces a positive transition in the middle of the bit
There is always a transition at the beginning of a bit period.
BiphaseM

1 forces a transition in the middle of the bit

0 does nothing
There is always a transition at the beginning of a bit period.

BiphaseS

1 does nothing
0 forces a transition in the middle of the bit
There is always a transition in the middle of a bit period.

Differential
Manchester

1 does nothing
0 forces a transition at the beginning of the bit
The positive and negative pulses alternate.

Bipolar

1 forces a positive or negative pulse for half the bit period

0 does nothing

9.(i)Write in detail the procedure of Shannon-fano coding scheme (8) (ii) Apply the
above method to a source which generates symbols x1,x2,x3,x4 with probabilities
1/8,1/2,1/4 and 1/8 respectively .determine the coding efficiency .
For example, assume we have six source symbols A, B, C, D, E, and F, with
respective probabilities 1/2, 1/3, 1/12, 1/15, 1/120, and 1/120. We could encode these using a
simple binary code: let A be 000, B 001, C 010, D 011, E 100, and F 101. However, notice
that A occurs as often as the other codewords combined, and that E and F hardly ever occur.
It would be far more efficient to have a shorter codeword for A, and longer codewords for E
and F.
Shannon-Fano Procedure
The Shannon-Fano method of encoding these symbols in binary proceeds as follows. Write
the source symbols in a column in descending probability order. Choose a point between two
symbols such that the summed probabilities above and below are as close to equal as
possible, and draw a horizontal line. Start a new column to the right of the first,
writing zeros above the line and ones below. The diagram will now look like this:

The source symbols above the line have their codewords assigned, and those below remain
uncoded. Repeat the process: choose a point between two of the remaining symbols which
equally weights probabilities, draw a horizontal line, and write zeros above and ones below:

This process continues until you have only one source symbol below the line. In this case, the
final diagram should look like:

Measuring Efficiency
Now we have a series of codewords for our symbols: A is 0, B is 10, C is 110, D is 1110, E is
11110, and F is 11111. Weighting by probability, we can calculate theaverage length of the
code:
Lave = 1/2 * 1 + 1/3 * 2 + 1/12 * 3 + 1/15 * 4 + 1/120 * 5 + 1/120 * 5
Lave = 1.767 bits
This, compared with the simplistic 3-bit binary code we first tried, uses only 60% of the
output symbols. We can calculate the efficiency of the code by comparing the average length
with the minimum length. Minimum length is defined as the entropy of the source divided
by log to the base 2 of the number base of the output; in this case, with binary output, it is
simply equal to the entropy of the source:
H(X) = - pk log2(pk)
H(X) = - 1/2 log2(1/2) - 1/3 log2(1/3) - 1/12 log2(1/12) - 1/15 log2(1/15) - 1/120 log2(1/120) 1/120 log2(1/120)
H(X) = 1.702 bits
Therefore the minimum possible length for a binary code of these symbols is 1.702 bits on
average. This means that the 3-bit binary code has an efficiency of 1.702/3 = 56.7%, while
the Shannon-Fano code has efficiency of 1.702/1.767 = 96.3%
10.( i) Describe bandwidth SNR trade off problem of coding (6) (ii) discuss any one of
the decoding methods of convolutional coding precisely
See Answer for Question 1 i)
ii) Syndrome Decoding of Linear Block Codes Syndrome decoding is an efficient way to
decode linear block codes. We will study it in the context of decoding single-bit errors;
specifically, providing the following semantics: If the received word has 0 or 1 errors, then
the decoder will return the correct transmitted message. If the received word has more than 0
or 1 errors, then the decoder may return the correct message, but it may also not do so (i.e.,
we make no guarantees). It is not difficult to extend the method described below to both
provide ML decoding (i.e., to return the message corresponding to the codeword with
smallest Hamming distance to the received word), and to handle block codes that can correct
a greater number of errors. The key idea is to take advantage of the linearity of the code. We
first give an example, then specify the method in general. Consider the (7, 4) Hamming code
whose generator matrix G is given by Equation (6.3). From G, we can write out the parity
equations in the

Because the arithmetic is over F2, we can rewrite these equations by moving the Ps to the
same side as the Ds (in modulo-2 arithmetic, there is no difference between a and a +
sign!):

There are n k such equations. One can express these equations, in matrix notation using a
parity check matrix, H, as follows:

H is the horizontal stacking, or concatenation, of two matrices: AT, where A is the submatrix
of the generator matrix of the code from Equation (6.4), and I(nk)(nk), the identity
matrix. I.e.,

H has the property that for any valid codeword c (which we represent as a 1n matrix),

Hence, for any received word r without errors, H rT = 0. Now suppose a received word r has
some errors in it. r may be written as c + e, where c is some valid codeword and e is an error
vector, represented (like c) as a 1 n matrix. For such an r,

If r has at most one bit error, then e is made up of all zeroes and at most one 1. In this case,
there are n + 1 possible values of H eT; n of these correspond to exactly one bit error, and
one of these is a no-error case (e = 0), for which H eT = 0. These n + 1 possible vectors are
precisely the syndromes introduced in the previous chapter: they signify what happens under
different error patterns. Syndrome decoding pre-computes the syndrome corresponding to
each error. Assume that the code is in systematic form, so each codeword is of the form
D1D2 ... DkP1P2 ... Pnk. If e = 100 ... 0, then the syndrome H eT is the result when the
first data bit, D1 is in error. In general, if element i of e is 1 and the other elements are 0, the
resulting syndrome H eT corresponds to the case when bit i in the codeword is wrong.
Under the assumption that there is at most one bit error, we care about storing the syndromes
when one of the first k elements of e is 1. Given a received word, r, the decoder computes H
rT. If it is 0, then there are no single-bit errors, and the receiver returns the first k bits of the
received word as the decoded message. If not, then it compares that (n k)-bit value with
each of the k stored syndromes. If syndrome j matches, then it means that data bit j in the
received word was in error, and the decoder flips that bit and returns the first k bits of the
received word as the most likely message that was encoded and transmitted. If H rT is not all
zeroes, and if it does not match any stored syndrome, then the decoder concludes that either
some parity bit was wrong, or that there were multiple errors. In this case, it might simply
return the first k bits of the received word as the message. This method produces the ML
decoding if a parity bit was wrong, but may not be the optimal estimate when multiple errors
occur. Because we are likely to use single-error correction in cases when the probability of
multiple bit errors is extremely low, we can often avoid doing anything more sophisticated
than just returning the first k bits of the received word as the decoded message. The
preceding two paragraphs provide the essential steps behind syndrome decoding for single bit
errors, producing an ML estimate of the transmitted message in the case when zero or one bit
errors affect the codeword.

11.Explain coding and decoding process of block codes

Refer Answer for Question no 4
12.Brief the properties of entropy. (4) (ii) Five symbols of the alphabet of DMS and
their probabilities are given below. S={So,S1,S2,S3,S4}
P(S) = {0.4, 0.2, 0.2, 0.1, 0.1}.Code the symbols using Huffman coding.(12)

In information theory, systems are modeled by a transmitter, channel, and receiver. The
transmitter produces messages that are sent through the channel. The channel modifies the
message in some way. The receiver attempts to infer which message was sent. In this
context, entropy (more specifically, Shannon entropy) is the expected value (average) of
the information contained in each message. 'Messages' can be modeled by any flow of
information.
In a more technical sense, there are reasons (explained below) to define information as the
negative of the logarithm of the probability distribution. The probability distribution of the
events, coupled with the information amount of every event, forms a random variable
whose expected value is the average amount of information, or entropy, generated by this
distribution. Units of entropy are the shannon, nat, or hartley, depending on the base of the
logarithm used to define it, though the shannon is commonly referred to as a bit.
The logarithm of the probability distribution is useful as a measure of entropy because it is
additive for independent sources. For instance, the entropy of a coin toss is 1 shannon,
whereas of m tosses it is m shannons. Generally, you need log2(n) bits to represent a variable
that can take one of n values if n is a power of 2. If these values are equally probable, the
entropy (in shannons) is equal to the number of bits. Equality between number of bits and
shannons holds only while all outcomes are equally probable. If one of the events is more
probable than others, observation of that event is less informative. Conversely, rarer events
provide more information when observed. Since observation of less probable events occurs
more rarely, the net effect is that the entropy (thought of as average information) received
from non-uniformly distributed data is less than log2(n). Entropy is zero when one outcome is
certain. Shannon entropy quantifies all these considerations exactly when a probability
distribution of the source is known. The meaning of the events observed (the meaning
of messages) does not matter in the definition of entropy. Entropy only takes into account the
probability of observing a specific event, so the information it encapsulates is information
about the underlying probability distribution, not the meaning of the events themselves.
Generally, entropy refers to disorder or uncertainty. Shannon entropy was introduced
by Claude E. Shannon in his 1948 paper "A Mathematical Theory of
Communication".[1]Shannon entropy provides an absolute limit on the best possible average
length of lossless encoding or compression of an information source. Rnyi
entropy generalizes Shannon entropy.

Entropy is given by H as follows

The entropy is H=2.12

13.(i)Briefly discuss on various error control codes and explain in detail with one
example for convolution code.(12)
(ii) Draw the polar, unipolar, bipolar and Manchester NRZ line code format for an
information
{ 1 0 1 1 0 0} (4) . (Nov 2010).
Channel is noisy Channel output prone to error we need measure to ensure correctness
of the bit stream transmitted Error control coding aims at developing methods for coding to
check the correctness of the bit stream transmitted. The bit stream representation of a symbol
is called the codeword of that symbol. Different error control mechanisms: Linear Block
Codes Repetition Codes Convolution Codes
Linear Block Codes A code is linear if two codes are added using modulo-2 arithmetic
produces a third codeword in the code. Consider a (n, k) linear block code. Here, 1. n
represents the codeword length 2. k is the number of message bit 3. n k bits are error
control bits or parity check bits generated from message using an appropriate rule. We may
therefore represent the codeword as