In video streaming, it’s good to be a victim of your own success. At least, it’s better than having no viewership at all.
Perhaps you’re using a world-class CDN and, yet, some of your users are still not enjoying a good streaming experience, especially during large live events when the CDN’s servers and peering points become saturated with traffic. Every CDN has a finite amount of capacity, so every CDN has its limits.
To make matters worse, every CDN is a black box that offers little to no insight into how much capacity is available at any given moment. As a result, it’s impossible to predict how the CDN will actually perform during a large event, especially if that event is going to be streamed by a few different content providers. You’ll find yourself actually competing with other content providers for the scarce resources of your shared CDN.
The mother of all events is coming - the FIFA World Cup starts in a few days and it looks to be the largest streaming event ever. We spoke with a few of our broadcaster customers who will be streaming the games to ask them how they are preparing for this massive event.
Based on these conversations, we’ve compiled a list of 10 precautions you can proactively take to ensure your users have the best experience possible when streaming the World Cup. They’ve been waiting for this event for 4 years - you don’t want to disappoint them:
# 1 Data is crucial - Get visibility into how your CDN is performing
Use services like Mux, Youbora or Conviva to understand exactly how your CDN performs. Again, every CDN is a black box, but you should at least test it from the outside and get meaningful data about your delivery.
Taken from Peer5 analytics dashboard during a live event
Peer5’s built-in streaming analytics can also provide many insights into your CDN delivery, and you can use these analytics free of charge even if you don’t use our P2P delivery product. If you have zero analytics today, then this is better than nothing. If you have analytics, then Peer5’s numbers can be compared against your existing vendor’s stats for a sanity check.
#2 Hold your CDN vendor accountable for issues
CDNs are not perfect. They often return HTTP errors and some are notorious for prioritizing important / aggressive customers at the expense of other customers. Get in touch with your CDN, come prepared with data (#1) and demand better performance for your large event.
If your CDN doesn’t fix problems and doesn’t meet your expectations, consider switching! Today, it’s no longer a huge hassle to switch CDNs, and it often only takes a few days to deploy and test a new CDN.
#3 Go multi-CDN if you haven’t done so already
You simply can’t rely on a single vendor to carry your most important event of the year. Redundancy is the key for live streaming and, today, it’s much easier to go multi-CDN. The first step is to use CDNs purely for caching and remove any vendor-locking features such as ingestion. Then it’s very easy to add another CDN (or 2 or 4!). You can then load balance between CDNs and switch to a healthy one in case of a problem.
Even if you are vendor-locked and cannot really move from your CDN, there are ways to use additional CDNs just for caching and redundancy and still use your main CDN exclusively for ingestion / encoding.
#4 Improve your multi-CDN switching logic
How do you currently load balance between CDNs? We see many broadcasters still using very simple, static load balancing schemes. This means that each user is assigned to a single CDN throughout the whole session, which can be a problem. The first thing you MUST test: if one CDN goes down (or is performing poorly), can you switch to a different CDN mid-stream? You can build this functionality yourself, or use third party solutions such as Cedexis or DLVR. Shoot us an email if you need help with this.
#5 Diversify CDNs - Add a serverless CDN
Buy yourself an insurance policy. Peer-to-Peer (P2P) mesh networks have been effective for decades in solving large scale problems. Once Google open-sourced WebRTC, it enabled Peer5 to build a P2P, serverless CDN that requires no end-user download or install, since WebRTC is already baked into the browser. This means Peer5 is completely transparent to the end-user, just like any server-based CDN, and integrates seamlessly with your current streaming infrastructure.
This hybrid approach (traditional CDN + serverless CDN) is unique because it completely diversifies your delivery stack, making you impervious to load. A P2P CDN is the ultimate streaming insurance policy and the best part is that it’s totally free. If everything is normal and your traditional CDNs are performing well, then you’ll never need the P2P capacity and won’t be charged for it. But, if all hell breaks loose, and it looks like you’re about to become a victim of your own success, you really need a solution that will save you from articles like this and this.
Large broadcasters such as FuboTV, Grupo Clarin and YuppTV have added our serverless CDN to their mutli-CDN configuration and are now ready for massive audiences, even when their CDNs are overwhelmed.
#6 Spin up some extra servers
Many companies today run their own data centers, sometimes in conjunction with commercial CDNs. If you run your own CDN, be cautious and spin up some extra servers just in case. Take into account the thundering herd phenomena and spin up more instances before it’s too late.
#7 Be ready to prevent more users from accessing the stream
No one wants to find themselves in this situation, but you may need to throttle connections in areas where your CDN is having issues.
Consider telling users if / when you're experiencing delivery issues. Or, consider removing the video element from the page for some period of time. It’s better to have no video at all instead of a video that is doomed for constant errors and rebuffering.
By taking this step, you’re also relieving your CDN. You really don’t want to scrap error messages or push code changes during a big game. Develop this functionality in advance and make sure you can pull the trigger within a minute if need be.
#8 Be ready for compromises - Add a lower rendition
Many content providers no longer include a 360p rendition in their manifests since the quality is just not good enough. But during large events, you should consider adding them back. A lower rendition means that even if your CDN is under stress and has limited capacity, some end users will still be able to watch.
#9 Perfection is not always a good thing - Remove a higher rendition
An even more interesting approach is the ability to remove the highest rendition in case of an emergency (capacity problem). ABR algorithms are greedy [link to interesting article], meaning each player will try to suck up as much bandwidth as possible from the CDN to reach the highest possible video quality. Each player is independent and doesn’t “know” if the server is under stress or how many other users are currently requesting video segments. It’s very likely that all of them are actually fighting to reach the same highest rendition but eventually none will be able to do it (think prisoner's dilemma).
Again, this is not an ideal scenario, but when bad things are starting to happen, having a fail safe that can eliminate renditions might prove very useful.
#10 Know your limits - Stress test beforehand
Unfortunately, there’s no perfect, global stress test solution available today. There are synthetic solutions like LoadRunner that will create thousands of requests to your CDNs, but these requests will usually originate from a few servers located in the same data center which has great connectivity to your CDN. You can’t really test your CDN at super high scale and in difficult regions. It can also be expensive and time consuming.
The best solution we’ve found (tell us if you know of a better one) is to use real data from previous large events and seriously dig into the weak pain points that are identified by the analytics described in #1.
Good luck and let us know if we can help your scaling initiatives in any way.