Earlier this month, we released a new batch of results from our VPN speed testing that contained a surprising result: one VPN appeared, on paper, to actually improve internet performance. As with most things, this miraculous result is probably too good to be true, but it gives us the chance to explain how we’ve been doing VPN testing during the COVID-19 era and what that means for the future.
Our Old Testing Methods
Back when standing less than six feet from another person still seemed like a good idea, PCMag tested VPNs very differently. At that point, VPN testing was a marathon of back-to-back sessions that ended with the results being released all at once. You can read about it in our How We Test VPNs article—at least until we update it later this month.
This approach to testing came from a desire to control for as many variables as possible, which is a pillar of all PCMag’s analysis. For VPN testing, we used the same machine, the same network, and the same tools to evaluate performance.
Removing all the variables around speed testing VPNs has been an ongoing struggle. We’ve long dealt with outages, dropped signals, and having to restart the router for no reason. We did sequester the network from other devices, but it was always subject to an enormous amount of interference from the hundreds of devices that were active in the PCMag Labs at any given time.
Of course, there were always issues too large for us to tackle. Variations in use from the entire population of New York City meant that we would see small but noticeable changes in baseline throughput at different times of the day. We also knew that different parts of even the same country likely had greater or fewer VPN servers depending on demand. These are just a few of the reasons why we say that your results will probably differ from ours.
Testing VPN speeds in one big batch was partly to account for all this. By grouping all the tests together, the data would have all come from the same timeframe. It was a snapshot of a particular time and place that let us compare individual products. All in all, testing usually took about 10 days split over two work weeks. Analysis and spot-checking would add another week, after which we’d publish an updated Fastest VPN article.
Adjusting Testing for the Times
In mid-March 2020, the entire PCMag staff started working from home full-time. That meant no more gigabit PCMag Labs test network. My own internet was too unreliable (especially under the initial WFH crunch) and, thanks to our local ISP monopoly, no better alternative was available.
It was clear we needed the PCMag Labs to keep up our high standards, but even after we were allowed back in the office, it was on a case-by-case basis and required coordinating with numerous people. Missing a time slot to go into the Labs meant waiting several days for another shot. That’s not to mention how traveling on the subway in those early days meant wearing two masks and rubber gloves for two hours, worrying about infection along the way.
Our solution was to ditch an all-at-once model for testing. Instead of evaluating results from about 40 products over a few weeks, we tested products in batches and updated the results throughout the year. Most processes remained the same: same machine, same math, same tools.
We made two other changes. First, we decided that if a vendor could demonstrate that another round of speed testing was required, we’d do it as soon as practicable. While we were always open to retesting VPN speeds, we committed to lowering the bar for retesting. Second, we’d present all our results at once, wherever possible. For this, we moved from static graphs to interactive Infogram charts. You can scroll through and even search for any product, wherever this chart appears. In fact, you can see it right here:
A Surprising Result
So far, we’ve been pleased with the results. Our data is fresher, easier to update, and hopefully more useful to you, gentle reader.
This new timeliness also means that we can also comment on our work as it happens. One unusual result from the most recent batch of tests comes from HMA VPN. Typically, you should expect latency to increase when you use a VPN. If you think about it, this expectation makes sense. Your web traffic has to go to an additional machine—the VPN server—before it can be back on its normal path through the internet. Your data’s trip should be longer when the VPN is on, increasing latency. HMA VPN defied expectations by improving latency by -0.02%.
We use the Ookla Speedtest tool for all of our VPN testing, so I showed my results to Marc von Holzen, VP of Software Engineering at Ookla, to get his thoughts. Holzen agreed that the HMA VPN result isn’t enormously significant. “Latency tends to have high variation,” he said. “This can make it difficult to determine whether a small difference between two latency results is significant without a large sample size.”
(Editors’ Note: Ookla is owned by PCMag’s parent company, Ziff Davis.)
The final figure in our results are the percent change between two median results, to make comparisons between services clearer. We run 10 tests with the VPN and 10 without, with the percent change calculated from the median of the two sets. If you look at the actual results, the miracle disappears. When we tested HMA VPN, the connection had a median baseline latency of 4.66 milliseconds without the VPN and a median latency of 4.64 milliseconds with the VPN.
HMA VPN probably will not improve your internet speeds, but it seems clear that it’s doing an excellent job of not making your speeds significantly worse. It definitely deserves credit for that, and it might be a top choice if latency is a major concern for you—if, for example, you’re playing fast-twitch PC games. This will be reflected in its review, which we will update later this month.
Speed Isn’t Everything
As I’ve written in every VPN review for many years: Speed shouldn’t be your only concern. It’s hard to measure reliably, and wildly different depending on when and where you measure it. Keep the results of our speed testing in the back of your mind, but look hard at the cost, the features, and the privacy protections that a VPN service puts in place. The fastest VPN in the world is useless if it’s too expensive, not trustworthy, or a pain to use.