You are on page 1of 2

Isilon IQ Case Study

DailyMotion
A Breakthrough in Web Based Video Content Sharing

Applications Scale-out NAS to support one of Europes largest video and content sharing web sites Challenge Dailymotion was experiencing rapid growth of content and needed a new type of architecture to help serve large video files to its 37 million unique visitors per month. With the aim of never deleting user content, data integrity was an essential requirement, as well as a solution that could scale into the Petabyte range with minimum distraction costs. Isilon IQ Benefits Dailymotion looked at a number of alternatives included open source and SAN based solutions but selected Isilon IQ following strong testimonials from existing users and an evaluation period using real production systems. The Isilon solution provides Dailymotion with new levels of resilience and a cost-effective and scalable expansion path to meet the challenges of new data formats such as HD, as well as increased user participation.

When Dailymotion launched in 2005, the idea of a web based video content sharing site was regarded by many as a fad. Its humble beginning in a Paris, France consisted of home made Linux servers and limited connectivity via a classical internet connection to serve only a few thousands of the Internets most progressive users. Two years later and Dailymotion is Europes largest homegrown video sharing website with over 37 million unique visitors per month. The success of the site and its desire to maintain all user generated content almost indefinitely provides an enormous challenge for Matthieu Blumberg dit Fleurmont, System Architect for Dailymotion, who has helped evolve DailyMotion IT infrastructure since its early pioneering days. As a video content delivery website, our requirements focus on 24/7 availability, Matthieu explains. Our old solution relied on RAID which provided limited safety for our data and was able to deal with disk failure but not with the complete failure of one of our servers. With content growing at roughly 15TB a month, in mid 2006, Matthieu and his team started looking at an alternative storage infrastructure that would allow the site to keep up with burgeoning capacity and performance requirements. The existing system, based on cheap, homemade servers running on Linux and using SATA RAID disks was starting to become a management challenge as data started to creep into the 100-terabyte range. We intend to be a long tail platform and never have to intentionally delete or accidentally lose any user provided content. We simply needed a more reliable solution, Matthieu explains. As part of the evaluation process, Dailymotion looked at a number of possibilities, As all team members have been working with Free and Open-Source Software for several years, we first looked at various open source projects. But none of them could provide us with the stability and the scalability we required, so we started to contact big storage vendors, but for the amount of data we have, SAN architecture with global and unified file systems are too expensive and time consuming in terms of system administration.

We now have a better availability of data since we can deal with failures at different levels. The system can automatically cope with disk and node failures and the implementation can also handle partial network failure, which is considerably better than our old system. Matthieu Blumberg dit Fleurmont, System Architect for DailyMotion Corp.

DailyMotion
There was also a feeling that many of the vendor solutions providing hotline file system growth were very limited, especially as Dailymotion predicts that it will require around 1 Petabytes by early 2008. The team then started to look at infrastructure deployments of similar websites to scope a potential solution, We knew Isilon as Myspaces provider, and so we looked deeply at this technology as Myspace has similar needs, comments Matthieu. We also have personal relationships with some other big websites teams, which were using Isilon, and they gave us a good feedback on the OneFS operating system. After contacting Isilons local sales team in Paris, Dailymotion was provided with a 10 node IQ 6000 cluster within a few weeks. Matthieu adds, This try and buy option was a good opportunity for us to really benchmark the solution in a real production scenario. One of the key benefits of the Isilon solution was the data security level. Dailymotion chose a FlexProtect +3 to increase their own data protection from the previous Linux solution. The migration from the previous Linux architecture was easy with the NFS access of the Isilon solution. The cluster is now accessed by the HTTP protocol. Another benefit is the usage of the Isilon SmartConnect feature Isilons integrated load balancer to load all storage nodes from all servers of their platform. With SmartConnect they can guarantee 100 percent availability on their storage architecture.

Case Study

Following the success of the pilot project, DailyMotion deployed 24 IQ 6000 nodes each with an additional EX expansion node cluster to deliver a 285 TB storage pool. This cluster is fully integrated into the backend of DailyMotions Content Delivery Network and serves several video formats. We now have a better availability of data since we can deal with failures at different levels, explains Matthieu. The system can automatically cope with disk and node failures and the implementation can also handle partial network failure which is considerable better than our old system. DailyMotion have also increased performance with data accessible through any of the 24 members of the cluster instead of having local file systems for each server. As a side effect, the single file system reduces the management overhead for the administrators. Since deploying Isilon, Dailymotion has delivered record uptime and availability, providing 100 percent uptime for 1,3 billion pages on a monthly basis.

Web Portal Servers MySQL 5.x Servers External CDN Service to unload the internal Dailymotion CDN Limelight 20Gbits/day

External CDN

Encoding Servers

http

ftp
Isilon IQ & EX Nodes 1 Cluster - 285 TB

Proxy Cache

http

50Gbits/day 25M pages/day 2M users/day 1TB/day added

Users

Preview Servers

Stats & Ad Servers

Isilon Systems, Inc. | www.isilon.com 3101 Western Ave, Seattle, WA 98121

Toll-Free: 877-2-ISILON Phone: +1-206-315-7602 Fax: +1-206-315-7501 Email: sales@isilon.com

2001-2009 Isilon Systems, Inc. All rights reserved. Isilon, Isilon Systems and OneFS are registered trademarks, and TrueScale, SyncIQ, SnapshotIQ, SmartConnect, and SmartQuotas are trademarks of Isilon Systems, Inc. Isi-CS16-0609