You are on page 1of 23

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no.

1
Arithmetic coding
Elias coding
Arithmetic coding as finite-precision Elias
coding
Redundancy due to finite precision
Multiplication-free arithmetic coding
Context-adaptive coding


Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 2
Elias coding
Entropy coding algorithm for sequences of
symbols x with general (conditional) probabilities
Representation of x by a subinterval of the unit
interval [0,1)
Width of the subinterval is approximately equal to
the probability f
X
(x)
Subinterval for x can be determined by recursive
subdivision algorithm
Represent x by shortest binary fraction in the
subinterval
Subinterval of width f
X
(x) is guaranteed to contain
one number that can be represented by L binary
digits, with

( )
f
X
x
0.1111
0.111
0.1101
0.11
0.1011
0.101
0.1001
0.1
0.0111
0.011
0.0101
0.01
0.0011
0.001
0.0001
0
0
1
( )
2
log L f ~
X
x
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 3
Example: Elias coding of memoryless binary source
( ) ( )
1 3
0 1
4 4
X X
f f = =
0
1
1
4
n
c
n n
c a +
0
1
4
0
1
1
16
4
16
1
7
64
16
64
0
37
256
28
256
1
121
1024
148
1024
1 1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 4
Example: Elias coding of memoryless ternary source
Courtesy: T. Wiegand

f
X
a
( )
= f
X
b
( )
=
1
4
f
X
c
( )
=
1
2
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 5
Elias coding: choose binary fraction in subinterval
Uniquely identify interval [c
n
,c
n
+a
n
) by a binary fraction


Length of bit-string



Bit-string




bits
0. [ , )
n
n n n
L
bbbbbb b c c a e +

L
n
= log
2
1
a
n

(
(
(
+1= h
X
0:n1
x
0:n1
( )

(
(
+1
2 2 1
n n
L L
n n n
c c c

(
= + >

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 6
Successive decoding
Elias bit string can be decoded symbol by symbol, starting with the
first symbol
For each symbol n=0,1,2, . . .
Calculate the intervals [c
n
,c
n
+a
n
) corresponding to all possible x
n
Determine the interval for which
Emit the corresponding x
n

)
,
n n n n
c c c a e +

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 7
Elias coding for memoryless source
Elias coding algorithm
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=

+
Interval
width
Interval
lower limit
Start with unit
interval
n-th symbol
to be
encoded
Cumulative distribution
(excluding current symbol)
( ) ( )
1
0
0 1 2 1

where { , , ,..., }
i
X i X j
j
X K
F f o o
o o o o

=
=

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 8
Elias coding for changing probabilities
Elias coding algorithm
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n
n
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=

+
Interval
width
Interval
lower limit
Start with unit
interval
n-th symbol
to be
encoded
Cumulative distribution
(excluding current symbol)
( ) ( )
1
0
0 1 2 1

where { , , ,..., }
n n
n
i
X i X j
j
X K
F f o o
o o o o

=
=

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 9
Elias coding for Markov random processes
Elias coding algorithm for Markov-p sources
{ }
( )
( )
: 1
: 1
0 0
1 | : 1
1 | : 1
Initialize 0 and 1.
For each 0,1,...
Set min ,
Update ,
Update ,
n n p n
n n p n
n n X n n p n
n n n X n n p n
c a
n
p p n
a a f x
c c a F x
'
'
' +
' +
= =
=
'
=

+
X
X
x
x
Interval
width
Interval
lower limit
Start with unit
interval
n-th symbol
to be
encoded
Cumulative conditional
distribution (excluding
current symbol)
Up to p
previous
symbols
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 10
Arithmetic coding
Elias coding not practical for long symbol strings:
required arithmetic precision grows with string length
Finite precision implementations: arithmetic coding
Widely used in modern image and video compression algorithms:
JBIG, JPEG, JPEG-2000, H.263, H.264/AVC

n
x
Arithmetic
coder
Arithmetic
decoder
n
x
( )
PMF
n
X n
f x
( )
PMF
n
X n
f x
bit stream c
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 11
Finite precision for arithmetic coding
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=

+
N-bit precision is
permissible, if
rounding down
P-bit approximation of
probabilities
At most N+P LSBs change,
except for a possible carry
affecting MSBs.
Each bit can be affected by
at most one carry.

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 12




Represent by
state variable C
n




Represent by
state variable r
n




Send
Finite precision for arithmetic coding (cont.)
Output code string








Maximum value of r
n
can be limited by bit stuffing

MSBs that will 1 bits LSBs that
no longer change represent lower that might be
interval boundary for affected by a
further computation carry
. 01111111 1
n
n
r N P
c xxxxxxxxx xxx cccccccc c
+ +
=
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 13
Inefficiency due to interval rounding
Recall: Subinterval of width f
X
(x) is guaranteed to contain one
number that can be represented by L
n
binary digits, with


Hence, rounding one interval value a
n+1
increases bit string by






L
n
~ log
2
f
X
0:n1
x
0:n1
( )

log
2
a
n
f
X
x
n
( )
a
n+1
< log
2
2
N1
+1
2
N1
= log
2
1+ 2
1N
( )
~
1
ln2
2
1N
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 14
( )
( )
( )
( ) ( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
2 2 2
log 1 log log 1
( )
1

ln 2
0
X X
X X
X X
X
X
X X
X X
x x
X X
f x f x
E f x E f x E
f x f x
H X
f x
E
f x
f x f x
E f x f x
f x f x
(
| | (
| | | | A A
(
+ = + ( | ( | |

| |
|
( (
\ . \ .
\ .

=
( A
~
(

( A A
= = A =
(


Limited precision probabilities
Efficiency loss
provided that rounded
probabilities still add to 1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 15
Example: inefficiency for binary source
due to representing smaller probability by 2
-P

log
2
p
( )
r
e
l
a
t
i
v
e

b
i
t
-
r
a
t
e

i
n
c
r
e
a
s
e

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 16
Interval width is normalized to by binary shift after
each symbol
Approximation, with :



Corresponds to approximating of p(L) within a factor of 2:


Factor o can be lumped into external probabilities
Optimum o depends somewhat on data, typically





JBIG
Multiplication-free binary arithmetic coding
2 3
or 0.708 or
3 4
o o o = = =
( )
( )
( ) ( )
( ) (1 ( )) ( )
Ap L p L p L
Ap M A p L A p L A p L
o
o
'
~ =
'
= ~ =
1/ 2 1 A s <
p(L) s p(M)
( ) ( )
p L p L
A
o
~
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 17
MPS-LPS switch
Problem: interval width for symbol M can become negative



Solution: switch the role of M and L, if

Problem: interval for symbol M can be smaller than interval for L if


Solution: Conditional exchange of symbols, when problem occurs
Improves compression efficiency, implemented in JPEG and JBIG
( ) ( )
1
0 if ,1
2
A p L p L A

|
' '
< > e
|

( )
1
2
p L
'
>
( ) ( ) ( ) )
1 1
4 2
1
,
2
p L A p L p L A
' ' '
> > e

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 18
Sufficiency of binary coders

Consider r.v. equivalent to a r.v.

No loss by conditionally encoding bits (in any order)



Supply arithmetic coder with symbol/probability pairs


Total number of (conditional) probabilities to be estimated per symbol x


. . . same as with K-ary encoding of x
{ }
0,1, , 2 1
K
X e
0
1
1 K
B
B
B

| |
|
|
=
|
|
|
\ .
B
MSB
LSB
( ) ( )
( ) ( ) ( )
0 1 0 1 0 1 2
| | , ,
K K
H X H
H B H B B H B B B B

=
= + + +
B
( ) ( ) ( ) ( )
0: 1
0 0 1 1 1 1 | 0: 1
, , , , , , with 0,
k k
K K k B k
b p b p b p p f


=
B
b
1
1 2 2 2 1
K K
+ + + =
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 19
Consider stationary binary Markov-k process with probabilities



Context labeling function, enumerating all possible context vectors

Backward probability estimation separately for each context








Adaptive probability estimation
} {
n
X
( ) ( )
0: 1 ( ): 1
| 0: 1 | ( ): 1
0, 0,
k k n n k n
X k X n k n
f f


=
X X
x x
Context vector,
2
k
possible
combinations
( ) { } { }
: 0,1 0,1, , 2 1
k
k
e x x
| |
( )
( )
( )
( )
( )
0 ( ): 1
0
0 ( ): 1 1 ( ): 1
,
, ,
n k n
n k n n k n
C n
p n
C n C n




(
+ A

=
( (
+ A + + A

x
x x
Count of zeroes that
have previously
occured in that context
Probability
of symbol
x[n]=0
Bias for low counts,
typically A=1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 20
Adaptive probability estimation (cont.)
Tracking changing statistics, compromise between
Rapid adaptation (small past sample counts)
Accurate estimates (large past sample counts)
Scaled count estimation algorithm

{ } { }
| |
0 1
1 1
0 0
0 1 min 0 1 max
0 1
0 1
0
0
0 1
Initialize 0
For 0,1,
If 1
C 1
else
C 1
If min C , or max C ,
C ; C
2 2
Estimate
2
n
C C
n
x
C
C
C C C C
C C
C
p n
C C
= =
=
=
+
+
> >
( (

( (

+ A
=
+ + A
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 21
Context adaptive coding
2-d extension of Markov model








Markov conditional independence property

N

+ n
causal half-plane
| |
x n
Coding
context

N+ n

f
X n

X
N

+n
X n

, X
N

+n
( )
= f
X n

X
N+n
X n

, X
N+n
( )
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 22
Context adaptive coding (cont.)
Coding context vector can be mapped into context label directly
without loss
Feasible for binary images
Example: JBIG uses 10 binary pixels as context label
For 8-bit images, number of different contexts
Context might have to be clustered
Combine with prediction


256
N

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 23
Reading
W. B. Pennebaker, J. L. Mitchell, G. G. Langdon, Jr., R. B.
Arps, An overview of the basic principles of the Q-Coder
adaptive binary arithmetic coder, IBM J. Res. Develop., vol.
32, no. 6, November 1988.
Witten, Radford, Neal, Cleary, Arithmetic Coding for Data
Compression Communications of the ACM, vol. 30, no. 6,
pp. 520-540, June 1987.
Moffat, Neal, Witten, Arithmetic Coding Revisited, ACM
Transactions on Information Systems, vol. 16, vo. 3, pp.
256294, July 1998.