Evaluation and optimization of a peer-to-peer

video-on-demand system
q
Bin Cheng
a
, Xiuzheng Liu
b
, Zheng Zhang
b
, Hai Jin
a,
*
, Lex Stein
b
, Xiaofei Liao
a
a
Huazhong University of Science and Technology, Wuhan, China
b
Microsoft Research Asia, Beijing, China
Received 29 June 2007; received in revised form 16 November 2007; accepted 19 November 2007
Available online 4 December 2007
Abstract
Video-on-demand (VoD) is increasingly popular with internet users. However, VoD is costly due to the load placed on video servers.
Peer-to-peer (P2P) techniques are an approach to alleviating server load through peer-assisted sharing. Existing studies on P2P VoD are
mostly based on simulation and focus on areas such as overlay topology, but little is known about the eectiveness of P2P in a real VoD
system.
In this paper we present a comprehensive measurement study of GridCast, a deployed experimental P2P VoD system. Using a 2-
month log of GridCast, we evaluate its scalability and end user experience. Motivated by the observations on user behavior and unused
peer resource, we further optimize its performance. Our key ndings are: (1) a moderate number of concurrent users can derive satisfac-
tory user experience. However, good network bandwidth at peers and adequate server provisioning are still critical to good user expe-
rience; (2) a simple prefetching algorithm can be eective to improve random seeks; (3) a simple caching across multiple videos has great
potential to further improve system scalability. Overall, we believe that it is feasible to provide a cost-eective P2P VoD service with
acceptable user experience, and there is a fundamental tradeo between good user experience and system scalability.
2007 Elsevier B.V. All rights reserved.
Keywords: Peer-to-peer; Video-on-demand; Caching; Prefetching
1. Introduction
Video-on-demand (VoD) has been the subject of intense
interest both in the research and commercial sectors for a
number of years. The reason for this popularity is twofold.
On one hand, VoD is an attractive Internet service. It pro-
vides video to users with great control when they want it.
On the other hand, VoD is challenging due to high band-
width and real-time playback requirement. These chal-
lenges become particularly acute as the number of
subscribers increases.
Recently, peer-to-peer (P2P) systems have been immen-
sely successful in large scale content distribution, particu-
larly in the elds of le sharing [9,11] and live streaming
[8,13]. However, it has been an open question whether sim-
ilar P2P technologies could be used to provide large scale
VoD services. Unlike le downloading, the globally opti-
mal strategies, such as rarest rst employed in BitTor-
rent, are less applicable in VoD. Additionally, random
seeks also make VoD harder to be optimized than live
streaming. Existing studies are mostly based on simulation
and focus on areas such as overlay topology, but the eec-
tiveness of P2P in a real VoD systems is still poorly
understood.
In this paper we report an initial empirical study of
GridCast. GridCast is the rst real, working P2P VoD
1383-7621/$ - see front matter 2007 Elsevier B.V. All rights reserved.
doi:10.1016/j.sysarc.2007.11.003
q
Our work is partially supported by National Natural Science Foun-
dation of China under Grants 60433040 and 60703050, Hongkong Science
Foundation under Grant 60731160630. This paper has been partially
published in the 6th International Workshop on Peer-to-Peer Systems
(IPTPS07), Bellevue, WA, USA, February 2627, 2007.
*
Corresponding author. Tel.: +86 27 87557047; fax: +86 27 87543529.
E-mail address: hjin@hust.edu.cn (H. Jin).
www.elsevier.com/locate/sysarc
Available online at www.sciencedirect.com
Journal of Systems Architecture 54 (2008) 651663
system with a full set of VCR operations. By using peer-
assisted sharing, it reduces server load and improves user
experience. Since May of 2006, GridCast has been live on
Chinas national CERNET.
1
In peak months, it has served
videos to approximately 20,000 users.
Based on the 2-month log collected from GridCast, we
evaluate its performance in terms of scalability and user
experience. Motivated by the observations on seek behav-
ior and peer resource, we explore how to further optimize
its performance by more aggressive caching and prefetch-
ing. Along this way, we gain a few insights. For instance,
even with limited concurrent users that are spread in vari-
ous playback positions, the performance already
approaches that of an ideal system. However, the benet
only extends to peers that enjoy good bandwidth and when
server stress is less an issue. Peers with low bandwidth not
only have poor user experience, but can also cause prob-
lems to other well-connected peers. We nd that forward
seek dominates backward seek with a 7:3 split, and that
around 80% of seeks are within short distance. We also nd
that a simple prefetching algorithm is eective to improve
random seeks, but further optimizations are necessary.
Through trace-driven simulation, we demonstrate that
caching across multiple recently watched videos can further
improve system scalability.
The remainder of this paper is structured as follows. We
give a brief description of GridCast in Section 2. Section 3
summarizes the deployment of GridCast system and the
collected log data. Section 4 presents an in-depth analysis
of the overall system performance and user experience. In
Section 5, we explore how to further optimize the user
experience and the overall scalability. We discuss related
work in Section 6 and conclude in Section 7.
2. System overview
Fig. 1 presents a high-level architecture overview of
GridCast. The whole system includes a tracker server, the
GridCast web portal [1], a set of source servers and many
peers. As the rendezvous point, the tracker server helps
newly joining peer nodes bootstrap. It also records the
log information from all joined peers. The log enables us
to perform an in-depth analysis of the system performance.
The web portal provides the latest channel list that shows
the number of online peers and the rank of all videos.
The source servers hold all media les and provide media
data for a peer when it cannot nd data from other peers.
In GridCast, all media data are partitioned into chunks of
uniform time to support VCR operations, such as FOR-
WARD, BACKWARD and REWIND. Each chunk has
1 s play time. Peer nodes are the most complicated compo-
nent. Each peer caches all recently watched media data of
the current video in local memory or disk. When the data
in memory exceeds 300 s (about 18 MB for 60 KBps stream
rate), a LRU algorithm is employed to swap part of data
from memory to local disk. When a peer switches to
another video or leaves, all of cached chunks will be
evicted.
Just like other P2P content distribution systems, Grid-
Cast uses a set of source servers to release media les to
participating peers, who asynchronously play the les while
exchanging data among themselves. However, unlike le
downloading and live streaming, a peer is more selsh
in VoD. It only cares about contents after its current play-
ing position and can only help those that are behind. Addi-
tionally, a peer can change its playing position at any time.
These characteristics make a VoD system harder to opti-
mize, rendering globally optimal strategies such as rarest
rst as employed in BitTorrent [5] inapplicable.
To cope with the above problem, a GridCast peer main-
tains a routing table consisting of some peers that are
placed in a set of concentric rings with power law radii, dis-
tanced using relative playback positions, and uses gossips
to keep the routing table up-to-date. This architecture
allows a peer to nd a new group of position-close neigh-
bors in logarithmic steps after it seeks to a new playing
position. The tracker can be considered as a stationary peer
whose playback position stays at the xed time zero. The
peer caches played content onto its local disk. These data
are served to other peers, or to itself in case of backward
seeks. A peer exchanges chunks with its (up to 25) neigh-
bors, starting from those in the innermost ring and then
outwards, or from the source server otherwise. The peer
fetches the next 10 s rst; the playback stalls if these data
are not fetched in time. Next, it tries to fetch the next
200 s. More details can be found in RINDY [2].
3. Deployment
3.1. Deployment
Since May of 2006, GridCast has been live on CER-
NET. CERNET is a general-purpose internet network
and the sole xed network access for students on Chinese
university and college campuses. In China, student housing
is subsidized and substantially cheaper than equivalent o-
campus housing. For this reason and convenience, the vast
majority of students live in a dormitory while attending
college and grad school. Students use CERNET for many
applications; web, email, chat, and news.
In peak months GridCast has attracted approximately
20,000 users and supported up to hundreds of concurrent
users. A majority of users are in dormitories with direct
connections and generally have good network bandwidth
(>100 KB/s), but a small fraction of sessions orignate from
outside the CERNET. All the major national ISPs peer
with CERNET, but these peering connections are con-
strained, at around 45 KB/s. Approximately 2% of the
GridCast user sessions originate from these ISPs. Their
connections are generally inadequate for video and they
1
CERNET stands for China Educational and Research Network. As of
January 2006, CERNET connects about 1500 institutions and about 20
million end users.
652 B. Cheng et al. / Journal of Systems Architecture 54 (2008) 651663
do not have a large enough community to form indepen-
dent sharing pools. There is only so much a system can
do for users that lack adequate connectivity for the content
bitrate.
Our GridCast deployment has two video source servers,
located on dierent physical servers. Source A is more
powerful and hosts more video les than source B. The
tracker and web portal share a third machine. These three
machines are the shared, centralized base. Table 1 summa-
rizes the properties of these servers. All servers are located
on the same lab subnet at the campus of Huazhong Univer-
sity of Science and Technology (HUST), a university of
approximately 56,000 resident students located in Wuhan
city, Hubei province.
As of October 2006, GridCast hosts about 1200 video
les. The videos are movies and TV shows classied into
nine categories, ranging from action movies to TV variety
shows. The video programs typically have bitrate from
400 Kbps to 600 Kbps (with a few movies exceeding
800 Kbps), and are either Window Media Video (WMV)
or Real Video (RMVB) le format. The time length of
the video les ranges from 5 min to just over 2 h, with an
average length of 48 min.
3.2. Data collected
A GridCast client is instrumented to collect logs that are
turned on during playing time, and the logs are uploaded
to the tracker periodically. To evaluate GridCast, we
traced for 2 months, from August 28, 2006 to October 1,
2006. We traced the operation of the system at a ne level
of detail, generating a substantial amount of information.
The trace events include all user joins and leaves, all
VCR operations, all chunk requests and transfers, and
measurements of all jitters, seek latencies, and startup
latencies. Each peer records trace events in a local log.
Every 30 s a peer clears out its local log, packages all the
events in a message and sends it to the tracker. Our detailed
instrumentation at the clients enables us to collect logs
unavailable in previous studies, such as seek, chunk
requests and transfers, which enable us to perform an in-
depth analysis of a running P2P VoD system and help us
answer more interesting questions.
Table 2 presents several interesting trace statistics. In
these two months, about 20,000 users visited the GridCast
web portal and installed the GridCast client software.
About 22.8% users are behind NATs. The total number
Table 1
Properties of the shared server base
Server OS CPU RAM Disk Network
Source server A RedHat Linux AS 4.0 Pentium 4 3.0 GHz 2 GB 800 GB 100 Mbps (shared)
Source server B Windows Server 2003 Celeron 2.0 GHz 512 MB 200 GB 100 Mbps (shared)
Tracker/portal Windows Server 2003 Opteron 1.6 GHz 2 GB 40 GB 100 Mbps (shared)
Table 2
Statistics of a 2-month log, from August 28, 2006 to October 1, 2006
Item Value
Log durations 2 months
Total number of visited users 20,000
Percent of CERNET uers 98%
Percent of NAT users 22.8%
Maximal online users >360
Total number of sessions 250,000
Total number of videos 12,000
Average code rate 500600 kbps
Average video length about 48 min
Total content served from the source server 11,420 GB
Total content played by clients 15,083 GB
Fig. 1. Architecture overview.
B. Cheng et al. / Journal of Systems Architecture 54 (2008) 651663 653
of view sessions is about 250,000. The total video content
played by all of view sessions is about 15,000 GB. About
73% of them were served by source servers. Peer-assisted
sharing help ooad the remaining 27%. This result will
be further explained in Section 4. In peak time GridCast
supported about 360 concurrent users with a shared 100
Mb uplink bandwidth.
4. Evaluation
In this section we examine the performance of GridCast
based on the collected log. First, we evaluate the overall
scalability of GridCast and explore the potential of peer-
assisted sharing in a VoD system. Then we measure its
end user experience in terms of latency and jitter.
4.1. Scalability
The ability of peer-assistance to decrease load on the
source server depends on how much potential sharing
exists between peers. Peer-assistance substitutes underuti-
lized peer upload for constrained server upload. With the
current caching, this substitution can only take place if
there are more than one users watching the same video.
In order to explore how well peer-assisted sharing works
in GridCast, we rst dene two metrics; concurrency and
chunk cost. Concurrency is the number of users in the sys-
tem watching the same video at the same time. If a videos
concurrency is high, then peer-assistance can substitute cli-
ent upload bandwidth for server upload bandwidth and
thereby reduce load on the centralized components. Chunk
cost is dened as the total number of chunks fetched from
source servers for each played chunk. The units are source
server fetches per played chunk and the value represents
the average load induced on the source servers for each
played chunk. Chunk cost is the inverse of scalability.
Lower chunk cost represents better scalability.
Fig. 2 shows how well the GridCast system takes advan-
tage of existing concurrency. In this gure, we compare
GridCast with the ideal and client server models. In the
ideal model, each peer has sucient upload bandwidth
for sharing and never seeks. Therefore, only the rstly
joined peer fetches data from the source server at any time.
The chunk cost of the ideal model is 1/concurrency. In the
client server model, we assume that every peer only fetches
chunks from servers. Therefore, its chunk cost is 1.0. In the
current deployment of GridCast, each peer only caches the
recently watched chunks for the current video. This cach-
ing policy is called single video caching. The curve labeled
as GridCast with single video caching represents the result
from our deployment.
In contrast to the client server model, both GridCast
and the ideal model obtain great improvement when the
concurrency exceeds 1. When the concurrency reaches 4,
the chunk cost of GridCast is about 0.24. That means there
are only 24% chunks fetched from source servers. This
result shows that GridCast with peer-assisted sharing can
signicantly ooad source servers as the concurrency
increases.
Compared to the ideal model, GridCast has more con-
straints, such as insucient bandwidth, content miss
caused by seek, connection problems caused by NAT bar-
rier. These constraints can increase the chunk cost of Grid-
Cast. But surprisingly, the GridCast curve is very close to
the ideal model when the concurrency is over 4. In order
to give a deep understanding, Fig. 3 presents a detailed
breakdown for chunk fetches over dierent concurrencies.
By rerunning the log events then investigating global
peer state at the time of the chunk fetch, we divide all
chunk fetches into dierent categories. For each chunk
fetch, according to the role of its supplier, we determine
it is a source fetch served by the source server or a peer
fetch served by some peer. For a peer fetch, we further
trace whether its supplier exists or not when the chunk is
played. For a source fetch, we analyze why the chunk can-
not be served by any peers. We identify several categories
of causes.
Insucient bandwidth to peers. One or more active peers
in the neighborhood cache the chunk, but there is no
available bandwidth to any of them.
Fig. 2. Chunk cost vs. concurrency. Fig. 3. Breakdown of fetches across concurrency.
654 B. Cheng et al. / Journal of Systems Architecture 54 (2008) 651663
Cannot connect to peers. One or more active peers in the
global system cache the chunk, but there are no avail-
able connections to any of them. There are two reasons
why a peer cannot connect to another. First, the other
peer is behind a NAT. Second, the other peer has
reached its neighbor connection limit.
Global miss. No active peers cache the chunk. There are
two reasons why a chunk does not exist in the global
system. First, the chunk has never been fetched by any
peer in the past. It is completely new. Second, the peer
that ever watched the content leaves or switches to
another video.
In Fig. 3, we observe that about 50% of peer fetches are
from those peers that have left or switched to another video
when the chunks are played. This is because each peer
keeps prefetching chunks from its neighbors and the pre-
fetching rate exceeds the bitrate of current video. These
prefetched chunks have two eects. First, they can satisfy
the local playing. Second, they can be sent to other peers,
even when the original supplier leaves or switches to
another video. Interestingly, GridCast can outperform
the ideal model at two cases. First, a peer has prefetched
so many chunks that it can meet the data requirements
of all of other peers. Second, a peer stays at the system
and has enough chunks to sever all of other peers when
it pauses or nishes the current playing, just like a tempo-
rary server. In either case, there are no peers fetching from
source servers. That is why the chunk cost of GridCast can
be a little lower than that of the ideal model at some con-
currency in Fig. 2.
Fig. 3 also shows that the source fetches caused by the
insucient bandwidth and connection issue become pro-
gressively less severe as the concurrency increases. We
observe that most of source fetches belong to global miss.
These results suggest that decreasing global misses is a
major direction for the further optimization of system sca-
lability. By caching more content of recently watched vid-
eos, we can avoid a part of global misses caused by users
switching videos.
We evaluate the overall scalability of GridCast over
hours. First, we calculate the average chunk cost over hour
of day for all chunk fetches in the log. Then we compute
how many active users can be supported by the client server
model with the same source server load as the deployed
GridCast system. Fig. 4 presents the result. The overall
scale uctuates in a wide range from 0% to 55% as the hour
of day varies. The advantages of peer-assisted sharing are
more obvious when the system scale increases. We dene
two representatives time ranges; hot time and cold time.
Hot time is from 11:00 a.m. to 12:00 p.m. Cold time is from
1:00 a.m. to 10:00 a.m. The peak load of the source servers
always happens in hot time. In contrast to the centralized
server, GridCast improves scalability by an average of
26% in hot time. We nd that the overall scalability is
not as high as we expect. A further observation shows that
the heavy-tailed distribution of content inherently existing
in VoD (seen in Fig. 5) decreases the opportunities of shar-
ing between peers. We also observe that 80% of view ses-
sions happen at the concurrency of less than 2. These
results conrm again that we can increase the sharing
between peers and thereby improve the overall scalability
by caching more content on peers.
4.2. User experience
We measure user experience in three ways; startup
latency, seek latency, and jitter.
4.2.1. Startup and seek latency
After a peer joins a video, it starts to play only after
receiving 10 s worth of content. The same goes for seeks.
Startup latency is the time from when the user presses start
to when the video begins playing. With a seek, playback
only begins once 10 s of video has been buered at the
new target oset. Seek latency is the time from when the
user initiates a seek to when the video begins playing at
the new oset. These two latencies are qualitatively the
same since the user jumps to certain playing point.
Fig. 6 provides the cumulative distribution functions
(CDF) of startup latency and seek latency of all sessions
across all videos. Although the startup latency has a wide
Fig. 4. Scalability of GridCast and clientserver. This gure compares the
scalability of GridCast and the centralized limit over an average of all
days.
Fig. 5. Distribution of video popularity.
B. Cheng et al. / Journal of Systems Architecture 54 (2008) 651663 655
distribution up to 60 s, more than 70% and 90% of sessions
have lower than 5 and 10 s, respectively. Seek latency is
smaller, more than 70% and 90% of the sessions have lower
than 3.5 and 8 s, respectively. There are two reasons why
seek latencies are better. First, startup latency encounters
a 2-s connection delay as the user establishes the initial
neighbors if there are any (and that is the reason the two
are equal when there are no neighbors). Second, as we will
explain in Section 5, many seeks are forward and short
seeks. These seeks would nd data readily prefeched from
neighbors.
These results are not surprising given that a majority of
the users are within CERNET, up to 98% (seen in Section
3). In Fig. 7 we break down the startup and seek latency
according to whether the user is in the same campus, not
in the same campus but still on CERNET, and those that
are from elsewhere. In contrast to other CERNET users,
the campus users enjoy shorter latency due to their better
network locality. However, the non-CERNET users
encounter poor latency because they have lower network
capacity (45 KB/s). These results suggest that if the non-
CERNET users increase in the future, optimizing their
latency is critical since those users are seeing a delay close
to 1 min.
Fig. 8 shows the change of latency with the increasing
size of neighborhood. More neighbors will denitely help,
as the needed contents can be downloaded in parallel. Both
curves drop quickly with more neighbors, with startup
latency elevated by the 2-s initial connection delay. Ideally,
they should drop in reverse proportion to number of peers.
However, this is not true especially for startup latency, and
the dierence lies in the heterogeneity in peers networking
capability. This can be seen more clearly in Fig. 9, which is
a scatter plot of startup latencies of the on-campus users.
While the startup latency generally drops with more neigh-
bors, there are always outliers that suer from longer
startup latencies. A close look at these data points reveals
that, inevitably, there are non-CERNET users in the initial
neighbors. Our current algorithm is such that the initial
content is evenly partitioned and downloaded from those
neighbors, and the slowest one determines the startup
latency. A future improvement will be to be more net-
work-aware when selecting the initial neighbor.
4.2.2. Jitter
A peer will encounter a jitter if the required chunk can-
not come back in time. The jitter of a view session is repre-
sented by the average delay time per second of video play,
calculated by the total delay time divided by the total sec-
onds of the played content during a session. Similar to the
Fig. 6. CDF of latency for all sessions across all videos.
Fig. 7. Latency for dierent kinds of users.
Fig. 8. Latency vs. size of neighborhood.
Fig. 9. Startup latency distribution of campus users.
656 B. Cheng et al. / Journal of Systems Architecture 54 (2008) 651663
latency of seek and startup, jitter is related to the network
capacity of peers. The non-CERNET users always encoun-
ter serious jitters because their available bandwidths with
the source server are too limited (45 KB/s) and there are
not enough non-CERNET neighbors to help them. CER-
NET users do far better. From the observations of CER-
NET users, there are 72.3% and 40.6% sessions without
any jitters for 5 min and 40 min durations, respectively.
Again, GridCast works fairly well for users with good net-
work connectivity. Fig. 10 shows that jitter decreases
quickly with an increasing number of neighbors due to
the buer prefetching algorithm. If the peer has no neigh-
bors, it only prefetches up to 10 s from the sources. If the
peer has neighbors, it prefetches up to 200 s.
4.2.3. Analysis
Our observations show that many factors aect the user
experience of a view session. Primary factors include the
peers network capacity and the number of concurrent
users. Additionally, we also observe that there is a strong
correlation between the user experience and the stress at
the source server. Fig. 11 presents the changes of the over-
all user experience and the source load over hour of day.
Here we measure user experience in terms of the number
of unacceptable seeks and jitters over hour of day. An
unacceptable seek is a seek with more than 10 s latency.
An unacceptable jitter represents a view session with more
than 100 s delay. We rst calculate the average server load,
the total number of unacceptable seeks and the total num-
ber of unacceptable jitters across each hour, then normalize
them with their peak value respectively. It appears that the
uctuations of unacceptable seeks and unacceptable jitters
have a strong correlation with the stress at the source
server.
Server stress peeks when there are more users in the sys-
tem. But the amount of users is not the reason for server
stress increase. During the hot time, the number of active
channels is about three times of that in cold time, and this
causes high server stress that results in poor user experi-
ence. In Fig. 12, we see that in cold time both hot channels
and cold channels deliver good user experience, with only
3.5% unacceptable jitter and 2.5% unacceptable seek,
Fig. 10. Jitter vs. average size of neighborhood.
Fig. 11. User experience vs. server load. This gure shows the correlation
between the user experience and the server stress over hour of day. The
user experience is represented by two metrics; unacceptable seek and
unacceptable jitter, each normalized by their peak value.
Fig. 12. User experience over popularity. In this gure the user experience
is measured in terms of seek and jitter. For seek, the y-axis is the
percentage of unacceptable seeks, calculated by the number of unaccept-
able seeks divided by the total number of seeks in hot time or cold time
and the popularity is the size of neighborhood when a seek happens. For
jitter, the y-axis is the percentage of unacceptable jitters, evaluated by the
number of sessions with unacceptable jitter divided by the total number of
sessions in hot time or cold time and the popularity is the average size of
neighborhood during a session.
Fig. 13. Distribution of seek distance.
B. Cheng et al. / Journal of Systems Architecture 54 (2008) 651663 657
respectively. This is in sharp contrast to the hot time. How-
ever, user experience is heavily inuenced by concurrency.
With more concurrent users, each peer relies less on the
source server but more upon each other, thus their experi-
ence steadily improves.
One solution to reduce server load is to let peers to reg-
ister as source servers in the system. However, it is not nec-
essarily the only long term solution. We believe that to deal
with unexpected demand uctuations, a hybrid VoD sys-
tem such as GridCast must have proper QoS policies
implemented on the server side. This can range from delist-
ing to giving lower priority to the less popular contents
during hot time.
5. Optimization
In this section we investigate how to further optimize the
current system based on our initial observations. Moti-
vated by the observations of seek behavior in VoD, we
examine and evaluate a simple prefetching algorithm,
namely anchor prefetching. Anchor prefetching is to
decrease seek latency and thereby improve user experience.
Acting on the observations of unused peer resource in the
deployed system, we explore how to further increase the
overall scalability by using more peer resource through
trace-driven simulation.
5.1. Anchor prefetching
In a VoD system, peers may start playing at any time,
resulting poor content locality. This is not a problem espe-
cially for popular videos. Another critical challenge is var-
Table 3
Statistics of seek operations
Forward 72% Short distance (6300 s) 81%
Long distance (>300 s) 19%
Backward 28% Short distance (6300 s) 76%
Long distance (>300 s) 24%
Fig. 14. Correlation between seek and popularity.
anchor
Forward Seek
Backward Seek
current
t (sec.)
10 10 10
anchor anchor
300 300
10
anchor
300
adjust
Fig. 15. Anchor distribution.
Fig. 16. Seek latency comparison between with anchor and without anchor.
658 B. Cheng et al. / Journal of Systems Architecture 54 (2008) 651663
ious VCR operations that a user may initiate at anytime.
These operations induce change to the neighbor structures
and, to the end users, latency to fetch new data. We inves-
tigate this problem by analyzing seek behaviors that we
have observed. Fig. 13 presents the cumulative distribution
function (CDF) of the normalized distances of seeks. For
each seek, its normalized distance is the target position
minus the original position divided by the length of the cur-
rent video. In this way, backward seeks have a negative dis-
tance while forward seeks have a positive distance. As
Table 3 summarizes, there is a 7:3 split of forward versus
backward seek. Furthermore, short distance seeks domi-
nate: about 80% of seeks in either direction are within
300 s. Additionally, the seek behavior also has a strong cor-
relation with the popularity. Fig. 14 shows that there are
fewer seeks in more popular contents, which generally also
have longer sessions.
To reduce seek latency and improve user experience, we
have implemented an anchor prefetching mechanism as a
way to deal with random seeks. As Fig. 15 shows, anchors
are segments each consisting of 10 continuous seconds, and
are distributed throughout the media le with xed interval
(e.g. 300 s). When a seek occurs, we adjust the playback
position to the beginning of the closest anchor if the anchor
has been already downloaded. The idea is that when the
user jumps to a new playing point, the seeking is satised
instantly if the target anchor has already been downloaded.
If bandwidth allows, the peer uses a local rarest rst algo-
rithm to fetch these anchor points from the source server or
its neighbors, just like Bittorrent.
Fig. 16 compares the distribution of seek latency
with and without anchors, collected from two dierent
Fig. 17. Anchor utilization vs. number of seek and session duration.
Fig. 18. Available resources in deployed system (single video caching).
B. Cheng et al. / Journal of Systems Architecture 54 (2008) 651663 659
experiments. We observe that the anchor prefetching is
eective. The fact that backward seeks see less benet
is not surprising because there are less backward seeks
to begin with, resulting in less holes behind the current
playing position. In the forward directions, the number
of seeks that nish within one second jumps from 33%
to 63%. Consider that about 80% of seeks are within
300 s, there is obviously more room to improve. Prefe-
ching the next anchor is statistically optimal from the
individual users point of view. On the other hand, rar-
est rst as currently employed is globally optimal in
reducing the source servers load as it releases the rarest
content from the source into the peers. Thus, the best
strategy needs to consider both to reach the optimal
tradeo. Furthermore, past sessions can provide guid-
ance: the parts that were played more are obvious candi-
dates for prefetching, as proposed by Zheng et al. [12].
These options will be evaluated for the next release of
GridCast.
The prefetching mechanism must be cost-eective,
meaning that in addition to the reduction of seek latency,
we need to understand their utilization. The metric anchor
utilization is the ratio of the amount of played versus
fetched anchors, shown in Fig. 17. Utilization rises with
session duration as well as more seek operations. For
all sessions, the utilization averages to 70%, we believe
further optimization is still possible to improve the
eciency.
5.2. Multiple video caching
The current deployment of GridCast only uses local
disks and upload bandwidth to cache and share the current
video. This policy is called single video caching. If each peer
can save several recently watched videos on the local disk,
we can further increase the sharing opportunities between
peers. We call this kind of caching multiple video caching.
With multiple video caching, peers cache all their recently
viewed chunks, not only the chunks from their current
video. Cache eviction is by LRU when the cache reaches
a size bound, not when the user switches videos. When a
peer is online, all cached chunks are available for down-
load, constrained by practical factors such as the peers
maximum upload bandwidth. Multiple video caching
increases local peer cache contents to increase sharing. This
subsection investigates how multiple video caching can fur-
ther decrease load on the source servers to improve
scalability.
Caching uses disk to store chunks and upload to share
chunks. Here we rst investigates if adequate disk and
upload exist for the optimization of multiple video caching.
We lack information on peers network hardware, so we
measure the peak realized upload and download across
10 s periods of the trace and take those as a peers hard
limit. This is conservative because many peers are short-
lived and may not have the opportunity to realize their
peaks.
Fig. 18a shows the distribution of observed peak
bandwidths across users. Across all the active peers, the
average peak download is 2.65 Mbps. This is because
most users are on the CERNET with good connectivity.
The average peak upload is less than the average peak
Fig. 19. Evaluation of multile caching across dierent resource con-
straints. Unless otherwise shown each peer has a 2 GB cache, 0.5 upload
constraint, and no more than 25 neighbors. Here deployed represents the
case of the current GridCast deployment. In (c), 15P means no more than
15 neighbors and 25P means no more than 25 neighbors.
660 B. Cheng et al. / Journal of Systems Architecture 54 (2008) 651663
download, at 2.25 Mbps. This is unsurprising because
peers upload less than they download and more samples
of a distribution can give a higher peak. Fig. 18b shows
the distribution of the unused bandwidth capacity. A
peers bandwidth capacity is its session length multiplied
by its peak bandwidth. This assumes that the peaks are
sustainable. In this gure, about 80% of users have more
than 90% unused upload capacity. This suggests that
there is excess upload available for use by caching.
Unlike with bandwidth, we lack samples to estimate
peers disk capacities. Instead, we look at how reasonably
sized caches can capture the total requested data.
Fig. 18c shows this result. For example, a 2 GB cache
is able to capture more than 70% of the requests over
the month. To summarize, reasonably sized peer caches
are able to cover a good fraction of the requests. If the
cache is 500 MB it can cover about 45% of the total
data. If the cache is 5 GB, it can cover about 90%. These
numbers suggest that we should be able to get a reason-
able hit rate for a cache size that is a small fraction of
the size of an average consumer disk. Taken together,
these results show that there are enough unused peer
resources for multiple video caching.
Using trace-driven simulation, we evaluate the potential
improvement of multiple video caching. The simulator is
driven by the rst month trace described in Section 3.
When a peer joins, it sends the tracker a list of all the vid-
eos for which it caches one or more chunks. When a peer
starts a video, it uses the same protocol as single video
caching. However, a peer may now serve chunks for videos
other than its current video. A cache eviction policy is nec-
essary if the cache is nite. For cache eviction, we use least-
recently used (LRU) on les. It is certainly possible that
this is not the best eviction policy, but it has the virtue of
simplicity.
The caching algorithm has several parameters; maxi-
mum cache size per peer, maximum upload bandwidth uti-
lization, the cache eviction policy, and the maximum
number of neighbors per peer. Upload and cache size are
constrained to provide room for other applications. Grid-
Cast should not monopolize a peers local resources. The
cache size limit bounds the use of disk and the maximum
upload utilization limit bounds the use of upload band-
width. If the system runs with a maximum upload utiliza-
tion of 0.5, then upload can never exceed 50% of peak
across a 10 s scheduling interval.
Fig. 19 shows the chunk cost results for multiple video
caching across the three resource constraints; upload
bandwidth, cache size, and the number of connections.
These graphs include some data from the more thorough
presentation of Table 4. However, that table does not
include the deployed values or the 15 neighbor connec-
tions. Fig. 19 shows that caching improves chunk cost
more in the hot time than the cold time. This is because
there are many more peers online in the hot time. The
cache is larger and more diverse and able to absorb more
requests. Note that the overall cost is an average across
hours, not chunks. If it were across chunks, it would be
much closer to the hot time cost because there are many
more chunk fetches in the hot time than the cold time.
Fig. 19ac demonstrates the quick convergence of chunk
cost with increasing resources. For example, there is little
additional decrease in chunk cost from 0.5 upload or
2 GB cache size limit, even to the unconstrained innite
limits. Innite upload does not help much because the
peers still have real download limits. The ideal case of
Fig. 19c improves chunk cost little over 15 or 25 neighbor
sets. This demonstrates that global connections and global
knowledge do not really help much more than a small set
of well-chosen neighbors.
6. Related work
Whereas VoD has been considered as another potential
application of P2P technology, there are few studies that
present measurement results to examine the eect of P2P
VoD systems in real world. Most of existing work about
P2P VoD [3,4,7] systems was concentrated on the protocol
design of topology and the analysis of simulation results,
including our previous work [2]. Dierent from them, our
study provides real world results that can be used to vali-
date simulation scenarios. Recently, Yu et al. [10] presented
an in-depth understanding of access patterns and user
behaviors in a centralized VoD system. Zheng et al. [12]
proposed a distributed prefetching scheme for random seek
support in P2P streaming application through the analysis
of user behaviors log obtained from a traditional VoD
system. Based on the trace collected from MSN video, a
Table 4
Chunk cost with multile caching
Cache Global knowledge, global connections RINDY, max 25 neighbors
Upload bandwidth constraint Upload bandwidth constraint
Innite 1.0 0.5 0.25 Innite 1.0 0.5 0.25
Inf. 0.420 (0.342) 0.427 (0.342) 0.452 (0.338) 0.542 (0.312) 0.447 (0.351) 0.453 (0.351) 0.478 (0.342) 0.569 (0.313)
5GB 0.427 (0.342) 0.434 (0.341) 0.459 (0.337) 0.547 (0.311) 0.453 (0.350) 0.459 (0.349) 0.484 (0.343) 0.574 (0.311)
2GB 0.467 (0.328) 0.474 (0.327) 0.500 (0.321) 0.586 (0.295) 0.494 (0.335) 0.501 (0.334) 0.526 (0.327) 0.612 (0.294)
1GB 0.532 (0.306) 0.540 (0.305) 0.566 (0.298) 0.650 (0.268) 0.563 (0.313) 0.570 (0.311) 0.595 (0.302) 0.675 (0.268)
This table shows averages across hours with standard deviation bracketed across cache size, upload constraints, and global or localized sharing.
B. Cheng et al. / Journal of Systems Architecture 54 (2008) 651663 661
clientserver based system with lots of popular small vid-
eos, Huang et al. [6] analyzed the potential of P2P in terms
of saving server bandwidths and reducing cross-ISP net-
work trac. Compared with them, our study provides
insights for user experience and overall performance sys-
tems through the analysis of the trace obtained from a real
P2P VoD system.
7. Conclusions
In this paper we presented a measurement study of a
P2P VoD system deployed over CERNET, called Grid-
Cast. Our study demonstrates that peer-to-peer is capable
of providing a cost-eective VoD service with acceptable
user experience, even with moderate number of cooperative
peers. We also found that simple prefetching algorithm can
greatly reduce seek latency. We also identied a number of
problems. For instance, more concurrent users can drive up
number of active channels, leading to server stress growth
and degrading user experience for peers with fewer concur-
rent users. Also, peers with poor network connectivity are
not well supported. The caching policy across recently
watched videos can further increase the sharing opportuni-
ties between peers and thereby improve the overall scalabil-
ity. These insights are helpful to improve future design of
P2P VoD systems.
References
[1] http://www.gridcast.cn.
[2] B. Cheng, H. Jin, X.F. Liao, RINDY: a ring based overlay network
for peer-to-peer on-demand streaming, in: Proc. of Ubiquitous
Intelligence and Computing (UIC06), Wuhan, 2006.
[3] T. Do, K.A. Hua, M. Tantaoui, P2VoD: providing fault tolerant
video-on-demand streaming in peer-to-peer environment, in: Proc. of
ICC04, June 2004.
[4] Y. Guo, K. Suh, J. Kurose, D. Towsley, A peer-to-peer on-demand
streaming service and its performance evaluation, in: Proc. of
ICME03, July 2003.
[5] L. Guo, S. Chen, Z. Xiao, E. Tan, X. Ding, X. Zhang, Measurements,
analysis, and modeling of bittorrent-like systems, in: Proc. of ACM
IMC2005, Berkeley, CA, USA, October 2005.
[6] C. Huang, J. Li, K.W. Ross, Can internet video-on-demand be
protable? in: Proc. of SIGCOMM07, Kyoto Japan, August 2007.
[7] C.S. Liao, W.H. Sun, C.T. King, H.C. Hsiao, OBN: peering for
nding suppliers in P2P on-demand streaming systems, in: Proc. of
the Twelfth International Conference on Parallel and Distributed
Systems (ICPADS06), July 2006.
[8] X.F. Liao, H. Jin, Y.H. Liu, L.M. Ni, D.F. Deng, AnySee: peer-to-
peer live streaming, in: Proc. of IEEE INFOCOM06, April 2006.
[9] N. Magharei, R. Rejaie, Understanding mesh-based peer-to-peer
streaming, in: Proc. of NOSSDAV06, Rhode Island, May 2006.
[10] H.L. Yu, D.D. Zheng, B.Y. Zhao, W.M. Zheng, Understanding user
behavior in large-scale video-on-demand systems, in: Proc. of
Eurosys06, Belgium, 2006.
[11] M. Yang, Z. Zhang, X.M. Li, Y.F. Dai, An empirical study of free-
riding behavior in the maze P2P le-sharing system, in: Proc. of
IPTPS05, New York, USA, February 2005.
[12] C.X. Zheng, G.B. Shen, S.P. Li, Distributed prefetching scheme for
random seek support in peer-to-peer streaming applications, in: Proc.
of the ACM workshop on Advances in peer-to-peer multimedia
streaming, Hilton, Singapore, 2005.
[13] J. Liu, B. Li, T.S.P. Yum, CoolStreaming/DONet: a data-driven
overlay network for peer-to-peer live media streaming, in: Proc. of
IEEE INFOCOM05, March 2005.
Bin Cheng is a Ph.D. student in the School of
Computer Science and Technology, at Huazhong
University of Science and Technology (HUST),
Wuhan, China. He received his Master degree in
Computer Science from HUST at 2005, then
began to pursue his Ph.D. degree at the lab of
Cluster and Grid Computing, at HUST. Since
July of 2006, he has worked at the System
Research Group in Microsoft Research Asia, as
an intern. His research focus is peer-to-peer
streaming technology.
Xuezheng Liu is a Researcher of the System
Research Group in Microsoft Research Asia. He
received a Ph.D. in Computer Science from
Tsinghua University in 2005. His research inter-
ests include statistical learning, image processing
and classication, distributed computing and
large scale network applications.
Zheng Zhang is Senior Researcher and Research
Manager of the System Research Group in
Microsoft Research Asia (MSRA). He entered
Fudan EE undergraduate program at 1984 and
went on to the graduate school at 1987, one year
ahead of schedule. He joined HP-Labs in the fall
of 1996 with a Ph.D. in Electrical and Computer
Engineer. In HP-Labs, he focused on researching
large scale multiprocessors. Since 2001, he led a
team focusing on fundamental properties of large
scale peer-to-peer systems for planetary comput-
ing, and yielded novel solutions for routing, namespace and object
placement and a number of other protocols in MSRA. His current
research interest is ultra-large scale distributed system, utilizing and
advancing state-of-the-art of peer-to-peer technologies.
Hai Jin is a professor in the School of Computer
Science and Technology, at Huazhong University
of Science and Technology (HUST), Wuhan,
China. He received his B.S., M.A. and Ph.D.
degrees in Computer Science from HUST in 1988,
1991 and 1994, respectively. Now he is also the
Dean of the School of Computer Science and
Technology at HUST. In 1996, he was awarded
German Academic Exchange Service (DAAD)
fellowship for visiting the Technical University of
Chemnitz in Germany. He worked for the Uni-
versity of Hong Kong between 1998 and 2000 and participated in the
HKU Cluster project. As a visiting scholar, he worked at the University of
Southern California between 1999 and 2000. He is the chief scientist of the
largest grid computing project, ChinaGrid, in China. He is senior member
of IEEE, member of ACM, and member of Grid Forum Steering Group
(GFSG). His research interests include cluster computing, grid computing,
peer-to-peer computing, network storage, network security, and visuali-
zation computing.
662 B. Cheng et al. / Journal of Systems Architecture 54 (2008) 651663
Lex Stein is a Researcher in the System Research
Group of Microsoft Research Asia (MSRA). In
June 2007, he received a Ph.D. in Computer Sci-
ence from Harvard under the guidance of Margo
Seltzer. Before Harvard, he received an M.Eng. in
CS from Cornell, worked in industry program-
ming network routers, and spent a year at Cal-
Tech researching Linux cluster architectures for
parallel data processing. He received Bachelors
degrees in Commerce (Finance and Economics,
First Class) and Computer Science from Queens
University, Canada. Currently, his interests include le system and dis-
tributed system.
Xiaofei Liao is a Associate Professor in the School
of Computer Science and Technology, at Huaz-
hong University of Science and Technology
(HUST), Wuhan, China. He received his Ph.D.
with honors from HUST in 2005. His research
interests cover multimedia services, peer-to-peer
computing, and visualization computing.
B. Cheng et al. / Journal of Systems Architecture 54 (2008) 651663 663