YouTube Architecture
Posted on August 1, 2007 Comments (2)
This very interesting post by Todd Hoff gives an overview of the YouTube Architecture and thus some interesting ideas on large scale web application architecture.
Each video hosted by a mini-cluster. Each video is served by more than one machine.
* Using a a cluster means:
– More disks serving content which means more speed.
– Headroom. If a machine goes down others can take over.
– There are online backups.
* Servers use the lighttpd web server for video:
– Apache had too much overhead.
– Uses epoll to wait on multiple fds.
– Switched from single process to multiple process configuration to handle more connections.
* Most popular content is moved to a CDN (content delivery network):
– CDNs replicate content in multiple places. There’s a better chance of content being closer to the user, with fewer hops, and content will run over a more friendly network.
– CDN machines mostly serve out of memory because the content is so popular there’s little thrashing of content into and out of memory.
* Less popular content (1-20 views per day) uses YouTube servers in various colo sites.
– There’s a long tail effect. A video may have a few plays, but lots of videos are being played.
* Using a a cluster means:
– More disks serving content which means more speed.
– Headroom. If a machine goes down others can take over.
– There are online backups.
* Servers use the lighttpd web server for video:
– Apache had too much overhead.
– Uses epoll to wait on multiple fds.
– Switched from single process to multiple process configuration to handle more connections.
* Most popular content is moved to a CDN (content delivery network):
– CDNs replicate content in multiple places. There’s a better chance of content being closer to the user, with fewer hops, and content will run over a more friendly network.
– CDN machines mostly serve out of memory because the content is so popular there’s little thrashing of content into and out of memory.
* Less popular content (1-20 views per day) uses YouTube servers in various colo sites.
– There’s a long tail effect. A video may have a few plays, but lots of videos are being played.
I have noticed a large increase in significant delays (taking 10-20 seconds to start playing) with YouTube in the last few months.
2 Responses to “YouTube Architecture”
Leave a Reply
RSS Feed
August 2nd, 2007 @ 6:22 pm
Mmh..lighttpd seems to be all the hype.. a lot of new web2.0 projects I know are using it..
Never tried it.. I guess I’ll have to take a look at it.
April 2nd, 2010 @ 12:07 pm
First of all, I just came across your blog and found it so informative that I could not resist myself from commenting on it! You’re really doing a great job man, thank you and keep posting. I’ve subscribed to your RSS-feed, and will be doing so for a long, long time.
Secondly: Today, YouTube is a lot faster. I remember those old days where you had to wait for ten seconds before the video started playing…
Today, that’s not the case. Today, the video starts playing immediately!