Adaptive 360-Degree Video Streaming using Layered Video Coding
Afshin Taghavi Nasrabadi ∗ Anahita Mahzari†
The University of Texas at Dallas, Texas, U.S.A.
A BSTRACT
Virtual reality and 360-degree video streaming are growing rapidly;
however, streaming 360-degree video is very challenging due to
high bandwidth requirements. To address this problem, the video
quality is adjusted according to the user viewport prediction. High
quality video is only streamed for the user viewport, reducing the
overall bandwidth consumption. Existing solutions use shallow
buffers limited by the accuracy of viewport prediction. Therefore,
playback is prone to video freezes which are very destructive for the
Quality of Experience(QoE). We propose using layered encoding
for 360-degree video to improve QoE by reducing the probability
of video freezes and the latency of response to the user head move-
ments. Moreover, this scheme reduces the storage requirements
significantly and improves in-network cache performance.
Keywords: Adaptive 360 video streaming, SVC, Video freeze
Index Terms: H.5.1 [ Multimedia Information Systems]: Artifi-
cial, augmented, and virtual realities—Video;
1 I NTRODUCTION
With advances in multimedia technologies, the field is evolving to-
ward providing an immersive experience for users. The demand for
Virtual Reality (VR) Head Mounted Displays (HMD) is growing in
order to enable the immersive video experience. VR HMDs utilize
360-degree videos, also known as spherical video, which involve
a 360-degree view of the scene captured from a single point. The
captured video is mapped to the internal surface of a sphere. An
HMD views a limited portion of the spherical video as seen from
the center of the sphere, refered to as viewport. The area covered
by the viewport is limited to the HMD’s Field of View (FoV) and
its coordinates are based on the orientation of the user’s head.
The growth in popularity of VR has led video service providers,
such as YouTube and Facebook, to enable streaming 360-degree
videos. However, streaming 360-degree video brings up new chal-
lenges. Presenting high video quality in the viewport requires a
complete high-quality 360-degree frame. For example, Oculus rift
viewing resolution is 2160x1200 with a FoV of 110 degree; thus,
the complete 360-degree frame should have the resolution equal
to 8K video with bitrate of 50Mbps to exploit the highest quality
that the device can offer. Therefore, not only does streaming of the
whole video in highest quality require large amount of bandwidth
that most end-users do not have, but transmitting complete video
to the user at highest quality also results in bandwidth waste, since
only a subset, namely the viewport, is viewed by the user.
One solution is adaptive 360 video streaming. The scheme is
similar to Dynamic Adaptive Streaming over HTTP (DASH), but
spatial dimension is also added to the adaptation space. The client
predicts the user’s future viewport and requests the future segments
∗ e-mail: afshin@utdallas.edu
† e-mail:anahita.mahzari@utdallas.edu
‡ e-mail:joseph.beshay@utdallas.edu
§ e-mail:ravip@utdallas.edu
2017 IEEE Virtual Reality (VR)
March 18-22, 2017, Los Angeles, CA, USA
978-1-5090-6647-6/17/$31.00
c 2017 IEEE diction and network condition.
Joseph D. Beshay‡ Ravi Prakash§
with higher quality in the predicted viewport. The server stores a
representation set that provides each time-domain segment at differ-
ent qualities with higher quality at a set of possible viewports. Each
representation of a segment has a higher quality for a specific view-
port that could be of interest to the viewer and lower quality for the
rest of the 360-degree frame. Thus, higher quality can be provided
to the user with less bandwidth usage. The current state-of-the-art
solutions try to predict the user viewport for a short duration in the
future and request the segments that provide this viewport at higher
quality. When there is a mismatch between the user’s viewport and
the predicted one, the lower quality parts are displayed. To mini-
mize the occurrence of such mismatches, the video is not buffered
for durations longer than the duration for which the viewport can be
predicted accurately, thus increasing the probability of video freeze
in the event of network bandwidth fluctuation. Video freeze is most
destructive for user QoE [3]. In this paper, we argue that using
Scalable Video Codec [6] encoding in adaptive 360-degree video
streaming can mitigate this problem. SVC also provides other ad-
ditional benefits: 1-using enhancement layers at the client allows
more flexibility in the adaptation 2-the storage requirement on the
server-side is reduced, and caching performance is improved.
2 R ELATED W ORK
This section provides an overview of 360-degree video streaming.
First mapping and encoding techniques are introduced. Then, the
recent works on adaptive 360-degree video are discussed.
Existing video encoders cannot encode the spherical videos di-
rectly. In this respect, the video should be mapped to a 2D plane
before encoding. There are various map projection methods such
as equirectangular, pyramid and cube projections. Equirectangular
projection maps the sphere into a rectangle, however, it results in
severe stretching at the north and south poles of sphere which in
turn reduces coding efficiency and increases bandwidth consump-
tion. In the pyramid projection, the viewport is mapped to the base
of the pyramid, and the remaining parts are mapped to the sides.
This causes severe quality degradation for areas far from the base.
Moreover, this mapping is GPU-intensive [2]. Cube projection, as
shown in Figure 1-a,b, maps a sphere into six faces of a cube. It pro-
vides smooth quality degradation with lower processing complexity
than pyramid, and lower bitrate overhead than equirectangular.
Adaptive 360-degree video streaming solutions divide a video
into segments with several seconds duration, and each segment is
sliced into tiles in the spatial domain. Each tile is encoded at sev-
eral quality levels. The client fetches a set of tiles for each segment
based on viewport prediction and network condition. Based on this
scheme a streaming system is proposed in [5]. Authors have ana-
lyzed the relation among tile size and bandwidth saving. A similar
method is proposed in [4] with the goal of reducing bandwidth con-
sumption in cellular networks. Different methods are studied for
viewport prediction, and it has been shown that they reduce band-
width consumption by 78%. Unlike [5] and [4] which have used
equirectangular mapping, Corbillon et al. [1] have shown that cube
mapping provides higher quality. Additionally, instead of encod-
ing each tile of a segment independently, different representations
of a segment are available according to a set of possible viewports.
Then, the client requests segments according to the viewport pre-
347
 Figure 1: Mapped sphere into cube, and layered encoding
3 P ROPOSED M ETHOD
Given a video sequence with the duration of D seconds, we pro-
pose segmenting it in the time domain into segments of s seconds
each. St denotes the t th segment, where t ∈ {0, ..., D/s}. Each seg-
ment contains the six faces of the cube-mapped 360-degree video.
Each face of the cube is sliced in the spatial domain into m × m
tiles denoted T f ,i, j,l , where f ∈ {0, .., 5} is the face number and
i, j ∈ {0, .., m − 1} is the latitude and longitude of each tile on a
face, respectively. Tiles are encoded in L different quality levels
according to the layered scheme of SVC [6]. The lowest quality is
the base layer, and each enhancement layer contains additional data
that can be added to the lower layers to improve the quality. Figure
1-c shows a frame of spherical video mapped into a cube, with each
face sliced into four tiles, and encoded in three layered quality lev-
els. Each color represents a face of the cube. l ∈ {0, ..., L − 1} is the
quality level. The client always requests the base layer for all tiles to
ensure that all viewports are available for viewing if the user’s view-
port changes quickly. For tiles currently within the user’s viewport,
the client requests enhancement layers to provide a better quality.
Depending on the availability of bandwidth, enhancement layers
can be requested for the tiles where the user’s viewport may shift.
The range of the surrounding tiles to request depends on the ac-
curacy of viewport prediction. Under this scheme, we provide the
base layer for all tiles in a single file so the client can fetch it in one
request. For all higher quality levels each tile is made available in a
separate file to be requested separately by the client.
The adaptation algorithm in the client works as follows: assum-
ing the maximum duration of the client buffer is B seconds, the
client first fills the buffer with the base layer. Then, it starts predict-
ing the viewport for the segment with earliest playback deadline
based on the previous viewports. The tiles covered by the predicted
viewport are identified, and the enhancement layers for those tiles
are requested. Figure 2 shows an example of viewport with FoV
of 100 degree on the sphere. The area covered by the viewport is
denoted by yellow color. By mapping the viewport into the cube,
the client can determine which tiles are covered. Figure 2 shows the
tiles requested for each layer. The client can request the enhance-
ment layer tiles only for the tiles in the viewport, or fetch the first
enhancement layer for the neighboring tiles of the viewport.
The proposed method maintains the bandwidth reduction
achieved by the current state-of-the-art method while introducing
several new benefits. Using the layered encoded tiles, the client can
buffer B seconds of the video as opposed to the existing methods
where the client cannot buffer segments for durations longer than
the viewport prediction accuracy (usually about 2 seconds [4]). In
case of network bandwidth drop, our proposed method can survive
long-lived drops by continuing playback of base quality, while other
methods will encounter video playback freeze which is known to be
the most destructive factor on the user QoE. Additionally, our pro-
posed method reduces the storage requirement for 360 video sig-
348
Figure 2: Mapping viewport and adaptation
nificantly. For encoding a video into multiple quality levels, SVC
removes the redundancy between different levels. It only encodes
the residual signal for higher layers, thus reducing the total storage
requirements. If 3 quality levels are required, our proposed method
requires 31% less storage space. Furthermore, SVC makes bet-
ter use of existing in-network caching capabilities since all viewers
need the base layer of a video regardless of their viewport. The base
layer can be cached at intermediate devices to reduce the load on
the server. Also, using SVC in adaptive 360-degree video stream-
ing reduces the cost of updating the quality of already downloaded
segments when the viewport changes suddenly or when the predic-
tion is wrong. Rather than discarding whole tiles or even segments
(as in [1]), SVC only requests the enhancement layers for the tiles
corresponding the new viewport.
4 C ONCLUSION
Using SVC layered video encoding to deliver streaming 360-video
is a promising approach to address a number of the issues with
the existing solutions. We presented an overview of a novel pre-
sentation and adaptation technique. To the end user, it provides a
higher QoE by reducing video freezes even under challenging net-
work conditions. To the network and content providers, it reduces
the storage requirements and avoids unnecessary bandwidth waste.
We are currently working on implementing and evaluating the
performance of our proposed approach.
R EFERENCES
[1] X. Corbillon, A. Devlic, G. Simon, and J. Chakareski. Viewport-
adaptive navigable 360-degree video delivery. arXiv preprint
arXiv:1609.08042, 2016.
[2] E. Kuzyakov and D. Pio. Next-generation video en-
coding techniques for 360 video and VR. 2016.
https://code.facebook.com/posts/1126354007399553/next-generatio-
video-encoding-techniques-for-360-video-and-vr/.
[3] R. K. Mok, E. W. Chan, and R. K. Chang. Measuring the quality of
experience of http video streaming. In 12th IFIP/IEEE International
Symposium on Integrated Network Management (IM 2011) and Work-
shops, pages 485–492. IEEE, 2011.
[4] F. Qian, L. Ji, B. Han, and V. Gopalakrishnan. Optimizing 360 video
delivery over cellular networks. In Proceedings of the 5th Workshop on
All Things Cellular: Operations, Applications and Challenges, pages
1–6. ACM, 2016.
[5] P. Rondao Alface, J.-F. Macq, and N. Verzijp. Interactive omnidirec-
tional video delivery: A bandwidth-effective approach. Bell Labs Tech-
nical Journal, 16(4):135–147, 2012.
[6] H. Schwarz, D. Marpe, and T. Wiegand. Overview of the scalable video
coding extension of the H. 264/AVC standard. IEEE Transactions on
circuits and systems for video technology, 17(9):1103–1120, 2007.