At Peer5 we handle big challenges everyday. Building a massive P2P Network means handling an enormous amount of data, scaling, analytics and more, all in real-time. To create our P2P CDN, we harness the power of users by turning them into nodes on the network (a.k.a. peers). Then we help them communicate with each other and share data efficiently, without plugins, using WebRTC. Doing this in real-time requires high performance algorithms and advanced logic. Just how fast does our technology have to work? Fast enough to broadcast a high quality live video stream to hundreds of thousands of users simultaneously.
It’s kind of amazing, but also a bit hard to picture, so today we’re going to do something special. We’re going to show you our network!
So what does a digital network look like? You can't just put electrons on a map - or can you?
The first thing I needed to create was a playground to test animations. I wanted to show the entire network, so I created a map that quickly turned out to be unsatisfying in comparison to what I had in mind. A conventional 2D map didn't have the spark I was aiming for.
Next stop - 3D. I’ve always found that when creating 3D models in JS, the popular library three.js was a good place to start. One 3D object, a camera and a scene, and just a few minutes later I had a nice sphere up and running. To make this visualization feel more real and less like stationary sphere geometry, I added some country borders and stars. I also made it spin.
Now I had my globe, and I was ready to move on to the next step- adding our p2p network.
A peer can be a computer, a phone or even a Smart TV. I decided to represent peers in the network with tiny particles. To give each particle a geographic location I used the peer’s IP address and converted it to coordinates using Geo-IP.
Our network seeks to optimize peer efficiency, so our mapping algorithm connects peers that are expected to have strong connections. In peer-to-peer networks, there’s a strong correlation between physical location to P2P quality. That means that most of the time P2P connections are in the same country, and frequently even in the same city. Areas with a lot of local activity are highlighted using white dots.
Places with many connections in the same location needed special representation because they don’t have to travel very far. In order to help visualize this, I created an animation of standing pillars above these hyperactive locations. When a connection is made between peers that aren’t located in the same place you’ll see an array of colored electrons zooming around the screen.
Sometimes our algorithm decides to connect peers that are far apart. This happens for several reasons. For starters, remember that not all near peers are relevant to each other. It’s possible that two neighbours are connected to our network, but they’re watching different streams. In this case, they might connect to relevant peers on the other side of the globe. In other cases, a distant connection might signify that nearby peers aren’t as stable as ones that are further away. This communication is constantly monitored, tested and adjusted in real time.
To allow you to visually distinguish between near connections and distant ones, each particle is colored according to its travel distance. Distant connections are colored differently as you can see on the legend.
Optimizing the entire world
The network is big: it contains hundreds of thousands of peers that are constantly connecting and exchanging data efficiently. In fact, the network is so big that when I pushed all of the activity data to my browser for the first time, the tab crashed immediately. Filtering out some of the unnecessary data helped, but the globe still struggled to continuously render, even at just 5fps.
How do you render 100,000+ peer particles at > 30fps? The first step when optimizing frame rate is to use a fancy
requestAnimationFrame API that allows the browser to control the load dynamically. If the browser gets momentarily overloaded, it can limit the number of fps to avoid CPU choking. This helped me get the tab to stop crashing, but the fps was still very low - not quite the "progress" I was hoping for. After spending some more time profiling the CPU it seemed like three.js simply couldn’t handle the sheer volume of information.
The first big performance boost didn’t come until after I introduced dynamic particle pools. Using dynamic pools shrunk the overhead needed to create and destroy particles and geometries. The dynamic pools lowered the number of geometries from several thousand to single digits. You can check out the pool module here. Using these pools, as well as applying the profiling conclusions and avoiding heavy operations - non O(1) actions -, led to great progress. Unfortunately, there were still significant performance issues during network peaks and on weaker platforms.
Then it was time for one last optimization. At Peer5, we are always aware of the platform that we are running on. Given how ubiquitous video is, we must handle peers on regular desktops, mobile phones, tablets, ChromeCast, Smart TVs and any other device that you can think of. Since each device has different characteristics, such as CPU power, network and resolution, we have to make some adjustments. At peak times, when the globe may have more than 100,000 particles active simultaneously, a decent desktop PC might be able to handle the load without a problem, but a mobile phone will struggle.
Someone suggested that I optimize using fps as a measuring stick, the idea being that high fps could suggest high performance. Unfortunately, low fps didn’t necessarily suggest poor performance. It's true that low fps could result from poor performance and CPU starvation, but it could also occur because of screen capabilities, GPU power or for any number of other reasons. After testing fps as a performance predictor, it just didn’t seem like an accurate measuring tool.
Execution time to the rescue
Another idea was to use the execution time of known actions to measure the globe’s live performance. I added logic to the execution time of particles being processed so that execution gets capped after reaching a certain limit. Reaching the time limit could be attributed to device speed, number of particles or even temporary computer overload should your computer decide that now is the best time to update itself. The time limit depends on the device and the amount of data used, but either way it shouldn’t exceed 16ms per frame (1000ms/60fps) and should leave some room for rendering overhead (e.g. less than 16ms). Whenever the cap takes effect, the residual particles that haven’t been processed are queued for the next round, offering them time to recover from the temporary overload.The queue has a limit as well and slow devices might drop some particles if they’re unable to process them.
We hope this visualization will help people better understand the nature of our product and of P2P networking in general. Visualizing our network wasn’t easy and neither was optimizing it, but if you understand what Peer5 is up to now, then it was all worth it. Feel free to share it with friends, create a similar tool for yourself and let us know if you have any thoughts or ideas about how we can help you better picture our P2P network.
Take a look at our P2P network globe: Peer5 live P2P network
Peer5’s Serverless CDN supports massively-scaled live and on demand video streaming. As the demand for ever increasing amounts of delivery capacity continues to grow unabated, no single CDN is able to provide all of the capacity, features, geographic coverage and uptime that broadcasters need. Hence, the growing momentum behind a Multi-CDN architecture and the load-balancing, redundancy and feature completeness advantages that this approach affords. Our novel peer-to-peer (P2P) solution solves the peak demand problem by creating a network that actually gets stronger as viewership increases. This means that we perform our best at the exact moment when traditional CDNs struggle the most – making Peer5 the perfect complement to a broadcaster’s existing CDN infrastructure. Register for your own Peer5 account here.