Академический Документы
Профессиональный Документы
Культура Документы
Abstract — Compressed video sequences are very spatial redundancy in the time domain or frequency domain to
vulnerable to channel disturbances when they are transmitted estimate the lost macroblock (MB). They interpolate the lost
through the burst-packet-loss networks such as a wireless MBs by averaging available pixels in the neighboring MBs [3].
channel. An effective solution for packet loss is to perform These algorithms are easy to implement, but the reconstructed
concealment at the receiver. Most of the spatial and/or MBs have the blurring artifacts. Also, the SEC methods try to
temporal concealment methods estimate the true motion minimize a measure of spatial variation between the lost MB
vector (MV) associated with a missing block by using the MVs and the neighboring MBs in the frequency domain [4], [5].
of adjacent blocks. However, these methods can not conceal Although these algorithms in [4] and [5] produce the better
effectively the missing blocks, since the neighboring blocks reconstructed image quality than the time domain SEC
can be easily lost by the burst packet losses destroying a large
methods, they introduce heavy computational costs which are
area of coded video. In this paper, we present a robust error
unfavorable properties in the mobile environment. The TEC
concealment method of estimating the true MV of the missing
techniques use the temporal redundancy found in common
block by using the bidirectional motion tracking technique
within neighboring frames of a damaged frame. This motion video sequences to conceal the missing MBs. They estimate
tracking technique uses the linear trajectory feature of the the missing MVs and exploit the pixels from the reference
image block in the forward and backward prediction frames to fill the damaged areas with the motion compensation
directions. Experimental results show that the proposed MBs. Motion compensated temporal prediction has been
algorithm has superior subjective and objective video quality shown to be effective at concealing lost MBs [2]. The most
than conventional error concealment methods in the burst- precise way to find the lost MV is the motion estimation which
packet-loss environments . searches the best MV with the spatial coherence criterion.
Zhang et al. [1] estimates the MV by searching the most
Index Terms — Error concealment, boundary matching
similar neighbors of the lost MB in the previous frame. In [6],
algorithm, motion tracking, motion vector.
a block-matching algorithm is presented to match the upper
and lower adjacent MBs of the lost one to conceal the damage.
I. INTRODUCTION Motion estimation, however, is not proper to mobile terminal
With the generalization of mobile communication services, because of its heavy computational costs. Therefore, the error
recent interest has been focused on the transmission of real concealment method with the low computation complexity is
time video over wireless and other error-prone communication preferable to the mobile environments.
networks. A major drawback in these networks is that they As the well known error concealment method requiring low
introduce a significant number of transmission errors into the computation load, boundary matching algorithm (BMA)
digital bitstream. For compressed video sequences, selects the lost MV among a set of candidates based on a
transmission errors not only corrupt the current decoded frame, spatial smoothness constraint [7]. The lost MV can be selected
but cause the error propagation to succeeding frames which from the predefined MVs such as zero MV, median, and mean
degrades the quality of services. Therefore, it is considerably of MVs. The BMA, however, does not work properly when the
important to develop the error resilience techniques to edges are diagonal or anti-diagonal. To solve this problem, the
minimize the visual degradation caused by transmission error. modified boundary matching algorithm (MBMA) defines
Error resilience can be divided into four parts: error several boundary difference signals for different edge
detection, resynchronization, data recovery, and concealment orientations [8]. This TEC method selecting MV from
[1]. While all of these parts are important, error concealment candidates can effectively conceal the lost MBs when there are
by post-processing is an effective way to combat transmission a few corrupted MBs in the frame. However, in burst error
errors and has been widely investigated [2]. There are two conditions, these algorithms may fail due to the decreased
groups in the error concealment techniques. The one is spatial number of candidates. That is, they can not use neighboring
error concealment (SEC) method and the other is temporal MVs of the lost MB when the burst error occurred in the frame.
error concealment (TEC) method. The SEC algorithms use the In wireless communication channel, the narrow bandwidth
restricts the frame size as QCIF (176 х 144 pixels) so that a
Jae-Young Pyun, Jun-Suk Lee, Jin-Woo Jeong, Jae-Hwan Jeong, and frame is divided into just several packets. Therefore, several
Sung-Jea Ko are with the Department of Electronics Engineering, Korea rows of MBs of one frame are likely to be lost under the burst
University, Seoul, Korea (e-mail: {jypyun, oldy, zski, meeso2, error conditions. By this observation, we propose a robust
sjko}@dali.korea.ac.kr).
Contributed Paper
Manuscript received June 21, 2003 0098 3063/00 $10.00 © 2003 IEEE
1014 IEEE Transactions on Consumer Electronics, Vol. 49, No. 4, NOVEMBER 2003
error concealment technique to overcome the burst error. The the corrupted area do not have any available MVs since all
proposed algorithm estimates the lost MV by using the surrounding blocks are damaged by the burst packet losses. In
forward motion tracking (FMT) and backward motion this case, only zero MV and collocated MV (MVP) are
tracking (BMT). The FMT extrapolates every MV of available which do not help to reach a high quality of temporal
previously decoded frame and estimates the lost MV by using concealment [9].
the overlapped area between the motion extrapolation MBs
and the damaged MB. With the BMT, the proposed algorithm
projects every MV of the next frame into the current frame and Reference Frame
estimates the moving trajectory of the MBs by using the Candidate MVs
MVT[B]
Current Frame
overlapped area between backward motion compensated MBs MVL[R]
MVP
and the damaged MB. Then, two MVs estimated by BMT and Avg(MVT[B], MVL[R])
Median( MVT[B] , MVL[R] )
FMT are averaged and the averaged MV is handled as one of Zero
MVP
the candidate MVs of the damaged MB. MVT Corrupted block
This paper is organized as follows. The conventional error (Collocated MB) MVL
concealment methods are described in Section II. Section III Corrupted area
presents our proposed robust error concealment method using
BMT and FMT. Experimental results and conclusions are MVR
given in Section IV and V, respectively MVB
Fig. 1. MV candidates for the corrupted MBs used in MBA and MBMA
methods.
II. OVERVIEW OF ERROR CONCEALMENTS
In this section, we review a conventional well-known error
concealment technique MBMA with the low computation load, III. PROPOSED ROBUST ERROR CONCEALMENT
and introduce the candidate MVs of the corrupted MB. ALGORITHM USING BIDIRECTIONAL MOTION VECTOR
TRACKING
MVL MVR M
n
∑ [α ⋅ MVi i
n −1
]
MVB MV = E
i =1
M
, (1)
Corrupted area
∑α
i =1
i
Correctly received MB
Overlapped
1 2 3 1 2 3 area with MBe
Corrupted
MV3
3
area
A3 AA3 3 2. MV estimation
MV3
4 5 6 4 5 6 of MBe A3 A3
A5 A5 MBe
MV5 A
MV5 5 Extrapolated
A6 A7 A5
6
A7 MB
MV7 7
7 8 9 Motion 7 A7 8 9
compenstation MV7 1. MV extrapolation
Corrupted
A7
MB
Fn-2 Fn-1 Fn
(a)
Correctly received MB
Corrupted
1 2 3 area 1 2 3 Overlapped
3 MV3 area with
A3 MV3 AA
MBe
A3 A3 33
4 5 6 A5 MV5
MBe 4 5 6
5 A5
A7 A5 1. Motion
A6 MV7 A6
A7
7 tracking MV5
7 8 9 2. MV estimation 7 A7 8 9
of MBe MV7
Corrupted MB
Fn-1 Fn Fn+1
(b)
Fig. 3. MV estimation of missing MB; (a) Motion vector extrapolation (FMT) and (b) Backward motion tracking (BMT).
MVBn , i.e.,
M
n
∑ [α ⋅ MVi i
n +1
]
MVBIn = Average[ MVEn , MVBn ] . (3)
MV = B
i =1
M
, (2)
∑α
i =1
i
If Fn −1 and/or Fn +1 are/is damaged, we use the selection
rule given in the Table I. For example, when Fn −1 is heavily
n +1
where α i is the weighting factor of MVi of MB i in frame
damaged and Fn +1 is correctly received, the BMT method is
n
Fn +1 . After the estimated MV MV is calculated, it becomes
B chosen to estimate the candidate MV of MBe .
the MVC of MBe . In the same way as the FMT method, the
best-match motion compensated MV is selected among the TABLE I
SELECTION OF MV ESTIMATION METHODS
candidate MVs and the selected MV is used for the motion
compensated temporal replacement to conceal the damaged Fn −1 Fn +1 MV estimation method
MB. G G BMVT
G B FMT
D. Bidirectional Motion Vector Tracking (BMVT) B G BMT
Simple temporal replacement or
B B spatial error concealment
The MVEn and MVBn estimated by the FMT and BMT method [9]
methods might be different, since image objects in a sequence G : Correctly received
do not move orderly in the real video stream. Assume that two B : Heavily damaged
neighboring frames Fn −1 and Fn +1 are received correctly. We After estimating the candidate MVs associated with the
n
first perform BMVT by taking the average of MV and E
missing block by using the BMVT, we use the SMD method to
determine the true MV of the missing block. Fig. 4 shows the
J.-Y. Pyun et al.: Robust Error Concealment for Visual Communications in Burst-Packet-Loss Networks 1017
IN
OUT
(a) (b)
Fig. 4. Matching area for the side match distortion; (a) Matching area
for MVT and MVL, (b) Matching area for MVB and MVR
Fig. 5. Results of different concealment methods at frame 4 of the Foreman sequence; (a) Original frame, (b) Corrupted frame, (c) MBMA,
(d) FMT, (e) BMT, (f) BMVT.
TABLE Ⅱ
PSNR AND CPU TIME OF THE ERROR CONCEALMENT METHODS IN THE SEQUENCE FOREMAN
Selected frame number
Methods Avg.
4 5 10 11 23
PSNR 25.14 22.94 26.76 25.09 27.84 25.55
MBMA
CPU time 2.33 2.23 2.5 2.46 2.3 2.38
PSNR 25.71 22.91 28.39 26.33 27.82 26.23
FMT
CPU time 2.54 2.39 2.92 2.69 2.92 2.69
PSNR 26.23 22.61 28.06 26.79 28.60 26.46
BMT
CPU time 2.63 2.69 2.85 2.69 2.53 2.68
PSNR 26.20 23.60 28.42 27.21 28.00 26.69
BMVT
CPU time 2.75 2.56 2.79 2.77 2.58 2.69
TABLE Ⅲ
VIDEO QUALITY AND CPU TIME OF THE ERROR CONCEALMENT METHODS IN THE SEQUENCE TENNIS
Selected frame number
Methods Avg.
2 3 4 16 21
PSNR 21.45 21.33 20.44 22.16 22.08 21.48
MBMA
CPU time 2.69 2.23 2.31 2.46 2.54 2.45
PSNR 22.80 22.11 22.46 24.22 20.08 22.33
FMT
CPU time 2.85 2.09 2.69 2.85 2.62 2.74
PSNR 22.15 21.30 20.63 22.36 24.30 22.14
BMT
CPU time 2.77 2.38 2.89 2.38 2.62 2.60
PSNR 22.44 22.18 21.43 23.38 23.07 22.50
BMVT
CPU time 2.85 2.54 2.23 2.77 2.62 2.60
[7] W. M. Lam, A. R. Reibman, and B. Liu, “Recovery of lost or Jin-Woo Jeong received the B.S. degree in Computer
erroneously received motion vectors,” in Proc. ICASSP’93, pp. V417- Engineering with Department of Computer Science and
V420, Apr. 1993. Engineering from In-ha University, in 2002. He is now a
[8] J. Feng, K. T. Lo, H. Mehrpour, and A. E. Karbowiak, “Cell loss M.S. candidate in electronics engineering with the
concealment method for MPEG video in ATM networks,” in Proc. Department of Electronics Engineering at Korea
IEEE Global Telecommunications Conference, vol. 3, pp. 1925-1929, University. His research interests are Mobile QoS, IP QoS,
Nov. 1995. and Error concealment.
[9] S. Valente, C. Dufour, F. Groliere, and D. Snook, “An efficient error
concealment implementation for MPEG-4 video streams,” IEEE Trans.
Consumer Electron., vol. 47, no. 3, pp. 568-578, Aug. 2001.
[10] A. Vetro, H. Sun, Y. Chen, and S. Kung, “True Motion Vectors for
Robust Video Transmission,” in Proc. of SPIE Visual Communications Jae-Hwan Jeong received the B.S. degrees in electronics
and Image Processing, pp. 230-240, Jan. 1999. engineering with the Department of Electronics
[11] ITU-T/SG-15 Video Coding Experts Group, Video codec test model, Engineering from Korea University, in 2002. He is
TMN8, 1997. currently working toward the M.S. degree in electronics
[12] J. Ribas-Corbera and S. Lei, “Rate control in DCT video coding for low- engineering with the Department of Electronics
delay communications,” IEEE Trans. Circuits Syst. Video Technol., vol. Engineering at Korea University. His research interests are
9, no. 1, pp. 172-185, Feb. 1999. in the areas of Mobile QoS, IP QoS, Handoff, video signal
[13] G. S. Yu, M. M. K. Liu, and M. W. Marcellin, “POCS-based error processing and multimedia communications.
concealment for packet video using multiframe overlap information,”
IEEE Trans. Circuits Syst. Video Technol., vol. 10, no. 4, Jun. 2000.