I had a surreal experience yesterday driving here in Sonoma County. My Bluetooth speakerphone rang and it was YouTube calling. How many people get calls from YouTube? The message was simple: I’d written a column critical of YouTube’s live Olympic coverage and they wanted me to try again.
“We’ve made a few changes,” said the voice on the other end of the line.
“What changes?” I asked.
“Just try it,” the voice said.
And so I did. And YouTube was correct.
The service is better for live streaming the Olympics. I can now reliably connect at 720p on a computer that couldn’t do better than 360p before. It’s still not perfect with occasional stutters and 1080p is still beyond me, but the service that was not usable at all a few days ago is now acceptable.
I wonder what they changed to improve the service?
I have some theories, one of which is, having stubbed their toes a bit, YouTube simply has fewer Olympic viewers than it had that first day of competition.
Taking a more positive view, to make their live service better YouTube could have increased available bandwidth, thrown more hardware at the problem, found and fixed bugs that were inhibiting performance, or made some changes in how they were encoding the video.
They probably made more than one change, but I’m fairly sure they did the last one — changing the way they encode video.
I was surprised when I visited YouTube last week (another surreal experience) and they told me the live video streams would be recorded and then reused as-is for repeat plays. This didn’t make sense to me so I asked about it twice just to be sure.
Encoding and recording a live performance means using single-pass encoding, which is less efficient than two-pass encoding where the computer takes one pass through the video to locate all the complex segments that will be difficult to encode, comes up with a plan to deal with them, then goes back for a second pass where the actual encoding takes place. By knowing what’s coming next, which isn’t possible with a truly live stream, the two-pass encoder can more fully compress the video, using fewer bits to transmit the same picture.
You have to use single-pass encoding for a live stream, but my expectation going into my meeting there was that YouTube would do two-pass encoding for the replays. When they said “no” I was surprised.
In my earlier Olympics column I could immediately see the problem, because the commercials, which are all two-pass encoded, played fine. It was just the one-pass encoded competitive action that wasn’t playing right for me.
I’m guessing someone at YouTube came to the same conclusion.
Of course live coverage still has to be done using single-pass encoding. But remember all encoding is done and video streams are served from the Google Cloud, meaning they share infrastructure. If two-pass encoding can lower the playback bit load by 50 percent, which it often does, then in the mix of millions of live and recorded streams being played at any time the recorded streams suddenly need half as much bandwidth, giving that up for use on the live side.
Or maybe my guess is wrong. I hope someone at YouTube will explain, perhaps in the comments. But that’s the best idea I could come up with about how YouTube could so quickly improve its Olympic video performance.
I’m glad they called.
“hello is mr cringely there? mr cringely, this is ellen from youtube and i have mr tube on the line…” however it went, sounds like an interesting call.
Is your number publicly listed? If not that’s scary..
Sports events are usually complex images that take a lot of resources to decode (and encode). Lots of water splashing around, complex backgrounds, etc, while ads tend to be very simple. What sort of footage was slow and stuttery?
Magnum, you can find Bob’s number on the “About Bob” tab of this very page.
I hope espn3 figures it out too. Ads are perfect. The live event is mediocre. My aging MacBookPro’s fans crank up during the live event and spin down during the ads–exactly the opposite of what I want.
Maybe when Apple buys Disney, espn3 can use the Maiden datacenter for massive compute power.
Seems like the memory leak (presumably) in Flash has been fixed. I watched the entire USA women field hockey match against Australia without a glitch.
Scratch that. It still leaks, albeit more slowly.
I think the performance also was helped by the fact that
1) there are fewer channels active during the day now than there were initially when lots of tournament and preliminary things were happening in parallel.
2) during the weekday, fewer are able to access it than might over a weekend (so we’ll see tomorrow if the performance goes back down)
3) they now do the archive processing after the stream was closed for the day instead of archiving in real-time for immediate access. it maybe that now they do the archive processing from the original raw mixed feed as a 2-pass which improves the size and performance of the archives and the excerpt feeds NBC is putting out there.
I am still having problems with Adobe Flash crashing. The adds always play fine and the events play fine once they actually start, but there are a few videos that crash flash every time I try to watch them. Is there a way to force my browser to use HTML5 instead?
Sure, just disable Flash. In Chrome, browse to chrome://plugins and in Firefox go to Tools->Add-ons – find “Shockwave Flash” and click disable.
So, not being the extreme technical expert, I’ll throw a bone at how they could do 2 pass encoding on a live event, although I doubt they were able to do this in a few days to improve performance.
Two Pass encoding on a live event can be done by creating a “tape delay”. The delay of 10-30 seconds would allow the first server in the encoding process to do the first pass on the section of video (data) and then pass the encoding optimization patterns to the second server for the actual second pass encoding step.
I’m sure Youtube has their render farms optimized for Live Video encoding. They just had to make some adjustments to their encoding rules.
I had a similar thought. Bob tried to make a distinction between “live” and “delayed” but actually nothing is ever live, even if you are right there, due to the finite speed of light. Thus the more you delay the live stream, the more hardware you can pass it through to process it in “real time”.
IBM had been the technology provider to the Olympics for 40 years. Things started turning sour during the 1996 games in Atlanta. There were a number of problems that year. Later it was learned all parties were partially at fault, but it was IBM who took the biggest PR hit. In 1998 IBM redoubled its efforts and the technology for the Nagano games worked flawlessly. By this point the Internet was becoming big and IBM was especially proud of the web hosting services they provided during the Olympics, and rightfully so.
In 1998 IBM and the IOC were unable to come to an agreement and they parted ways. IBM was SURE no one could do what they did for the 2000 games. They were expecting problems and was hoping the IOC would need them again.
Well it didn’t happen. The new technology providers in the 2000 Olympics got the job done. The Olympic web site stayed up. The attention of the world was on the games and not on technology problems.
One of the differences is the new technology providers used distributed web hosting technology. Content was provided from servers all over the Internet and all over the world. This approach was much more scaleable. Than the methods of the past. Today Cloud computing as done by Google and a few others improves on this approach by several orders of magnitude.
In an event like the Olympics it is difficult to anticipate the demand and provide the right amount of capacity. However with Cloud technology it is much easier to make adjustments and increase capacity.
My compliments to YouTube for pulling it together and using great technology well.
It is important to note…the term “cloud” can be misleading. What Google does is very different than what firms like IBM do. Google infrastructure is enormous and their processing capacity is unbelievable. If someone sells you “cloud” technology, don’t assume it is as good or is even the same as what firms like Google do.
Google is just playing the stuff that my cable co. (cox) offers to computers but with much less content offered. I watch USA win a gold in skeet shooting a couple of days ago but google’s menu offers no shooting.
We need a tv guide type menu that plays on deillkmand.
b
Gee….and I thought this was going to be another of your “Blufferbloat” stories”!
Again FUCK the olympics and NBC too, it’s crooked that I need to PAY (twice by watching commercials too) just to watch the olympics on youtube where all the other fucking content is free too!
I am an over the air TV person because I hate paying for content I can get for free. 99% of the popular shows are broadcast anyway. The only exception is my iTunes subscription to MythBusters.
OTA is over rated. Sure it’s technically superior and free to boot, but you don’t get what you don’t pay for. For a basic 60 channel cable subscription, I get to see new content year round. The major OTA networks only have new content for half the year. The cable channels fill that void nicely.
Cable shows for the most part don’t interest me. If I really like them I’ll find a way to get them online without paying $100 or more PER MONTH to Comcast. iTunes, Netflix, Hulu, Youtube are all alternatives to cable.
I agree with one of Bob’s older posts about the phone companies not having the incentive to build fiber to the home and that they’re handing broadband services over to the cable companies. If that were to happen we’d have a real competitor to Comcast and weak satellite and prices would/should go down. If not then worst case we end up with a faster bit schlepping scheme into our homes which is still a win for online providers.
“If that were to happen we’d have a real competitor to Comcast…” Exactly, the operative word being IF. It simply costs money to provide good content and good technology. And in both cases people have different opinions about what is good enough.
“I wonder what they changed to improve the service?” Me too. After reading this column I decided to give it another try. It wasn’t any better for me. Perhaps it’s my expensive T1 connection that is limited to 1.5 mbps, or maybe the router that my T1 provider sold me a long time ago for $700. Or maybe it’s just Internet bufferbloat, or overloaded Youtube servers. Or maybe I need a faster processor or html5. Yet it’s strange that most other Youtube stuff plays OK. (It’s a good thing my only interest in the Olympics is as a technical indicator of the state of IP TV.)
Also recently tried it again a few days ago. Very poor experience, so I paid for a VPN with location in London. iPlayer access to both live and “catch-up” feeds is a remarkable experience. There are so many elements: no ads, great tagging in catch-up videos, excellent commentary (vs. empty feed for YT minor sports). Will often watch the NBC prime-time show at my dad’s — but can hardly bear it. The ads, the constant attempts to emotionally pitch the athletes. A few nights ago we started watching at 8pm (waiting for all the penultimate day’s swimming), and the FIRST 25 minutes were a gymnastics retrospective about Kerri Strug, Shannon Miller, and the Kayrolis (sp?). I think by about the 45 minute mark we had watched 5 minutes or less of woman’s team volleyball. No other sports, just the retrospective thing plus some other feel-good pieces. Guess what I am doing is quasi-legal, but then so is the farce that U.S. Live Streaming and PrimeTime coverage have been.
Absolutely agree with you about NBC’s coverage, and US sports coverage in general. The fact is their Neilsen data show that the demographic for Olympic viewing skews female, so they Hallmark the crap out of their coverage with saccharine filler. It’s unbearable.
Coverage of the Olypics has gotten progressively worse since 1984 (not coincidentally, the first year it was profitable). Every decision is based on maxing out ratings and ad revenue. Remember in Atlanta when IOC cops were serving cease-and-desist orders to restaurants that dared to write “Olympic special” on their lunch boards? The current revenue model can’t die fast enough.
The YouTube channel is awesome because it lets you cut right to the sports and skip the Hollywood back story. No ads either (for me anyway, watching in Asia).
If this is the future of sports TV, I will happily pay money for it. Kind of like the NFL Pass we can buy (outside the USA). It’s not cheap but you get a brilliant, thorough, and thoughtful product. Ads optional, total control, full archives.
Glad you like the YT version. I was frustrated with it. Might be better now, but since I’ve paid for the London VPN connection for the month, I will never know. BBC iPlayer coverage could not be better. Really fantastic.
On a hunch, after reading this post, I went back to the NBCOlympics site to see what the paywall situation was. I’d been shut out as a non-tv-subscriber.
Well, now it appears that recorded events older than a couple of days are no longer behind the paywall, and I can see all but the most recent events. Maybe this was the policy all along, but I really wasn’t able to find it spelled out anywhere on the site.
It makes sense in terms of the bandwidth issues and optimizations you speculated about–reserve the costly non-optimized streams for paying customers, make the reprocessed lower-bandwidth streams available to the groundlings.
So I guess I have to eat some of my words. Maybe next time they’ll see the way clear to some kind of pay-per-view system for non-subscribers for live feeds.
Thanks for the update!
Prior to your original column about YouTube processing, my uploaded videos would be processed for resolutions 144p-720p flv/mp4 immediately, but ..
if the video is under 20 minutes, then the 480p & 720p webm versions would occur at least 8 hours later. If the video was over 20 minutes, those resolutions would never be done.
Now for short videos (under 10 minutes) all resolutions and container formats appear “immediately.”
For videos under 20 minutes, the 480p-720p webm are done within about 15-30 minutes. And glory be! the 30 min videos are now processed too within 1-2 hours.
Looks to me like Google/YouTube have decided to take YouTube delivery more seriously.
30 mionute videos are now being routinely pro
Maybe I’m dumb but how do you watch the Olympics live on YouTube? I know this is the channel: http://www.youtube.com/user/olympic
But I’m unable to click on the icons.
I’m in the US and have a cable subscription; I go to http://www.nbcolympics.com and click on the word “live”. I tried your youtube link but the link there tries to take me to a doubleclick tracking ad, which I’ve blocked via my hosts file.
Maybe there really is an opportunity for an Apple I TV.
Imagine turn on your tv and watch what you want.
I think the reason the commercials play better than the live streams is because they use different distribution methods. Commercials can be pushed to the thousands of on-demand servers close to the edge because they are produced in advance. Live streams can only hit a relatively small number of “live” servers. They fact that all those live streams worked as well as they did kind of astonished me. Scaling live has always been a major challenge. NBC deserves high marks for getting all those streams out there at all. But they get an overall F for other reasons (the flash wrapper that crashes constantly, the terrible site navigation, the crummy apps, the over-dramatic primetime coverage, the lack of live closing ceremony coverage, cutting off the primetime closing ceremony feed and botching the commentary, forcing ads in front of each stream even if you’re just trying to “stream surf” because the app and site can’t tell you what specific match is actually on the stream before you play it.)
If you read Bob’s description of the issue carefully in his first post, you can see that it cannot be a distribution problem. The ads AND live streams play just fine on his fast machine. On the “slow” machine (still fast by most standards), the ads are OK but the live streams are glitchy. This means it can’t be a bad feed from YouTube or your ISP. Otherwise the fast machine would be just as glitchy.
The best explanation is that they changed the encoding and/or fixed some bugs in the player so that the content can be played with less CPU power.
Bob, it would be nice to know what your CPU usage was before and after the “fix”. Or others here who had problems, what was the CPU usage?
What i don’t understood is in truth how you’re no longer really a lot more neatly-favored than you might be right now.
You are so intelligent. You recognize therefore significantly in relation to this subject, produced me individually imagine
it from so many numerous angles. Its like women and
men are not interested except it is one thing to accomplish with Lady gaga!
Your own stuffs great. Always deal with it up!
Because you would undoubtedly keep secret coming from an opponent, determine it via the a friend.
UGG Pas Cher http://www.botteuggsoldes.com/
Bliss is the cologne you cannot rain cats and dogs within many more whilst not obtaining number reduces within on your own.
nike free 3.0 http://www.nikefreerunherrenfrauen.com/