You are on page 1of 23

# Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no.

1
Arithmetic coding
Elias coding
Arithmetic coding as finite-precision Elias
coding
Redundancy due to finite precision
Multiplication-free arithmetic coding

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 2
Elias coding
Entropy coding algorithm for sequences of
symbols x with general (conditional) probabilities
Representation of x by a subinterval of the unit
interval [0,1)
Width of the subinterval is approximately equal to
the probability f
X
(x)
Subinterval for x can be determined by recursive
subdivision algorithm
Represent x by shortest binary fraction in the
subinterval
Subinterval of width f
X
(x) is guaranteed to contain
one number that can be represented by L binary
digits, with

( )
f
X
x
0.1111
0.111
0.1101
0.11
0.1011
0.101
0.1001
0.1
0.0111
0.011
0.0101
0.01
0.0011
0.001
0.0001
0
0
1
( )
2
log L f ~
X
x
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 3
Example: Elias coding of memoryless binary source
( ) ( )
1 3
0 1
4 4
X X
f f = =
0
1
1
4
n
c
n n
c a +
0
1
4
0
1
1
16
4
16
1
7
64
16
64
0
37
256
28
256
1
121
1024
148
1024
1 1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 4
Example: Elias coding of memoryless ternary source
Courtesy: T. Wiegand

f
X
a
( )
= f
X
b
( )
=
1
4
f
X
c
( )
=
1
2
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 5
Elias coding: choose binary fraction in subinterval
Uniquely identify interval [c
n
,c
n
+a
n
) by a binary fraction

Length of bit-string

Bit-string

bits
0. [ , )
n
n n n
L
bbbbbb b c c a e +

L
n
= log
2
1
a
n

(
(
(
+1= h
X
0:n1
x
0:n1
( )

(
(
+1
2 2 1
n n
L L
n n n
c c c

(
= + >

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 6
Successive decoding
Elias bit string can be decoded symbol by symbol, starting with the
first symbol
For each symbol n=0,1,2, . . .
Calculate the intervals [c
n
,c
n
+a
n
) corresponding to all possible x
n
Determine the interval for which
Emit the corresponding x
n

)
,
n n n n
c c c a e +

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 7
Elias coding for memoryless source
Elias coding algorithm
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=

+
Interval
width
Interval
lower limit
interval
n-th symbol
to be
encoded
Cumulative distribution
(excluding current symbol)
( ) ( )
1
0
0 1 2 1

where { , , ,..., }
i
X i X j
j
X K
F f o o
o o o o

=
=

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 8
Elias coding for changing probabilities
Elias coding algorithm
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n
n
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=

+
Interval
width
Interval
lower limit
interval
n-th symbol
to be
encoded
Cumulative distribution
(excluding current symbol)
( ) ( )
1
0
0 1 2 1

where { , , ,..., }
n n
n
i
X i X j
j
X K
F f o o
o o o o

=
=

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 9
Elias coding for Markov random processes
Elias coding algorithm for Markov-p sources
{ }
( )
( )
: 1
: 1
0 0
1 | : 1
1 | : 1
Initialize 0 and 1.
For each 0,1,...
Set min ,
Update ,
Update ,
n n p n
n n p n
n n X n n p n
n n n X n n p n
c a
n
p p n
a a f x
c c a F x
'
'
' +
' +
= =
=
'
=

+
X
X
x
x
Interval
width
Interval
lower limit
interval
n-th symbol
to be
encoded
Cumulative conditional
distribution (excluding
current symbol)
Up to p
previous
symbols
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 10
Arithmetic coding
Elias coding not practical for long symbol strings:
required arithmetic precision grows with string length
Finite precision implementations: arithmetic coding
Widely used in modern image and video compression algorithms:
JBIG, JPEG, JPEG-2000, H.263, H.264/AVC

n
x
Arithmetic
coder
Arithmetic
decoder
n
x
( )
PMF
n
X n
f x
( )
PMF
n
X n
f x
bit stream c
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 11
Finite precision for arithmetic coding
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=

+
N-bit precision is
permissible, if
rounding down
P-bit approximation of
probabilities
At most N+P LSBs change,
except for a possible carry
affecting MSBs.
Each bit can be affected by
at most one carry.

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 12

Represent by
state variable C
n

Represent by
state variable r
n

Send
Finite precision for arithmetic coding (cont.)
Output code string

Maximum value of r
n
can be limited by bit stuffing

MSBs that will 1 bits LSBs that
no longer change represent lower that might be
interval boundary for affected by a
further computation carry
. 01111111 1
n
n
r N P
c xxxxxxxxx xxx cccccccc c
+ +
=
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 13
Inefficiency due to interval rounding
Recall: Subinterval of width f
X
(x) is guaranteed to contain one
number that can be represented by L
n
binary digits, with

Hence, rounding one interval value a
n+1
increases bit string by

L
n
~ log
2
f
X
0:n1
x
0:n1
( )

log
2
a
n
f
X
x
n
( )
a
n+1
< log
2
2
N1
+1
2
N1
= log
2
1+ 2
1N
( )
~
1
ln2
2
1N
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 14
( )
( )
( )
( ) ( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
2 2 2
log 1 log log 1
( )
1

ln 2
0
X X
X X
X X
X
X
X X
X X
x x
X X
f x f x
E f x E f x E
f x f x
H X
f x
E
f x
f x f x
E f x f x
f x f x
(
| | (
| | | | A A
(
+ = + ( | ( | |

| |
|
( (
\ . \ .
\ .

=
( A
~
(

( A A
= = A =
(

Limited precision probabilities
Efficiency loss
provided that rounded
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 15
Example: inefficiency for binary source
due to representing smaller probability by 2
-P

log
2
p
( )
r
e
l
a
t
i
v
e

b
i
t
-
r
a
t
e

i
n
c
r
e
a
s
e

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 16
Interval width is normalized to by binary shift after
each symbol
Approximation, with :

Corresponds to approximating of p(L) within a factor of 2:

Factor o can be lumped into external probabilities
Optimum o depends somewhat on data, typically

JBIG
Multiplication-free binary arithmetic coding
2 3
or 0.708 or
3 4
o o o = = =
( )
( )
( ) ( )
( ) (1 ( )) ( )
Ap L p L p L
Ap M A p L A p L A p L
o
o
'
~ =
'
= ~ =
1/ 2 1 A s <
p(L) s p(M)
( ) ( )
p L p L
A
o
~
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 17
MPS-LPS switch
Problem: interval width for symbol M can become negative

Solution: switch the role of M and L, if

Problem: interval for symbol M can be smaller than interval for L if

Solution: Conditional exchange of symbols, when problem occurs
Improves compression efficiency, implemented in JPEG and JBIG
( ) ( )
1
0 if ,1
2
A p L p L A

|
' '
< > e
|

( )
1
2
p L
'
>
( ) ( ) ( ) )
1 1
4 2
1
,
2
p L A p L p L A
' ' '
> > e

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 18
Sufficiency of binary coders

Consider r.v. equivalent to a r.v.

No loss by conditionally encoding bits (in any order)

Supply arithmetic coder with symbol/probability pairs

Total number of (conditional) probabilities to be estimated per symbol x

. . . same as with K-ary encoding of x
{ }
0,1, , 2 1
K
X e
0
1
1 K
B
B
B

| |
|
|
=
|
|
|
\ .
B
MSB
LSB
( ) ( )
( ) ( ) ( )
0 1 0 1 0 1 2
| | , ,
K K
H X H
H B H B B H B B B B

=
= + + +
B
( ) ( ) ( ) ( )
0: 1
0 0 1 1 1 1 | 0: 1
, , , , , , with 0,
k k
K K k B k
b p b p b p p f

=
B
b
1
1 2 2 2 1
K K
+ + + =
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 19
Consider stationary binary Markov-k process with probabilities

Context labeling function, enumerating all possible context vectors

Backward probability estimation separately for each context

} {
n
X
( ) ( )
0: 1 ( ): 1
| 0: 1 | ( ): 1
0, 0,
k k n n k n
X k X n k n
f f

=
X X
x x
Context vector,
2
k
possible
combinations
( ) { } { }
: 0,1 0,1, , 2 1
k
k
e x x
| |
( )
( )
( )
( )
( )
0 ( ): 1
0
0 ( ): 1 1 ( ): 1
,
, ,
n k n
n k n n k n
C n
p n
C n C n

(
+ A

=
( (
+ A + + A

x
x x
Count of zeroes that
have previously
occured in that context
Probability
of symbol
x[n]=0
Bias for low counts,
typically A=1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 20
Tracking changing statistics, compromise between
Rapid adaptation (small past sample counts)
Accurate estimates (large past sample counts)
Scaled count estimation algorithm

{ } { }
| |
0 1
1 1
0 0
0 1 min 0 1 max
0 1
0 1
0
0
0 1
Initialize 0
For 0,1,
If 1
C 1
else
C 1
If min C , or max C ,
C ; C
2 2
Estimate
2
n
C C
n
x
C
C
C C C C
C C
C
p n
C C
= =
=
=
+
+
> >
( (

( (

+ A
=
+ + A
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 21
2-d extension of Markov model

Markov conditional independence property

N

+ n
causal half-plane
| |
x n
Coding
context

N+ n

f
X n

X
N

+n
X n

, X
N

+n
( )
= f
X n

X
N+n
X n

, X
N+n
( )
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 22
Coding context vector can be mapped into context label directly
without loss
Feasible for binary images
Example: JBIG uses 10 binary pixels as context label
For 8-bit images, number of different contexts
Context might have to be clustered
Combine with prediction

256
N

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 23