VPN Testing Study by AV-TEST (A Commentary)

The VPN testing recently conducted by AV-TEST provides a rare opportunity to independently examine ExpressVPN vs NordVPN vs PIA (Private Internet Access VPN). 

vpn testing study image

Comprehensive VPN testing was recently conducted by well-respected IT firm, AV-TEST. 

A recent study testing VPN services by AV-TEST, an independent German research institute for IT security, provides a unique opportunity to compare these 3 popular VPN providers.

Here's a direct link to AV-TEST’s full VPN testing report (PDF).

AV-TEST subjected 12 VPN services to a battery of tests including 3 we recommend: ExpressVPN, PIA (Private Internet Access) and NordVPN.

The 2 other VPN services we recommend in general, IPVanish and PureVPN, were not part of AV-TEST’s study. No explanation was provided by AV-TEST about why certain VPN services were included in their tests and others were not. After all, there are now almost 200 VPN services on the market, so why just focus on 12?

Anyway, the rest of this post concentrates only on ExpressVPN vs NordVPN vs PIA (Private Internet Access).

Takeaways from the VPN Testing Study

Read all the details below, but here are our main takeaways from AV-TEST's VPN tests:

  • AV-TEST looked at 12 VPN products in detail, 3 of which we recommend - ExpressVPN, NordVPN and Private Internet Access - as some of the best VPN services
  • The overall winner in the study from the VPN testing was Hotspot Shield Elite but it should be noted that they sponsored the study and we have some major concerns about this conclusion.
  • For the 3 services we recommend, the VPN testing concluded the following:
  • ExpressVPN passed all 6 VPN leak tests, had fast torrent download speeds and had the most countries with VPN servers
  • NordVPN had the easiest set-up, best latency, most total number of servers, fast torrent download speeds and was the VPN with the best features and functionality
  • Private Internet Access passed all 6 VPN leak tests, had the best kill switch and was the best performing VPN (finishing only behind the study's sponsor). 

What Did AV-TEST Evaluate in its VPN Testing?

AV-TEST evaluated the VPN products on 4 main criteria:

  1. usability
  2. privacy and security
  3. performance
  4. functionality

Let’s look at each element in turn with a focus on our 3 recommended VPNs services which were included in the AV-TEST VPN study.

AV-TEST vpn testing study image

Screenshot of the AV-TEST study (highlighting is ours).

1) Usability Including Overall Ease of Use

This part of AV-TEST’s VPN testing study focused on the ease of installing and setting up the VPN application in Windows.

PIA was praised for supporting 18 languages whereas ExpressVPN and NordVPN lost points for only supporting 2 and 1 language, respectively. Of course, if you speak English, this won’t bother you. 🙂

The testing found all three VPNs to have a quick and easy set up process. NordVPN was highlighted for having a very easy 2-step set up process after download.

For all 3 VPNs, manually connecting to VPN servers was found to be very quick, requiring only 2 steps.

VPN usability testing image

VPN testing: usability results.

2) Privacy and Security

In our view, these were the most important tests done in the VPN study. In particular, AV-TEST tested how well the Internet Protocol Address (IP) and by extension the identity of the user (you!) was protected and hidden from outside parties like Internet service providers (ISP) and visited web sites.

AV-TEST acknowledges that "Advertisers, webpage providers, governments and others are attempting to track user and their activity online through varies means."

They go on to say that users do not want their identities unmasked by an unsecured VPN or one that suffers from "leaks": "There are numerous ways such “leaks” may happen and a good VPN will be able to cope with them all"

The study also acknowledged that a VPN's logging policies must be clear, ideally with no personal data or identifying metadata being logged. (Incidentally, on that note, Private Internet Access and ExpressVPN are VPNs with proven non-logging claims.)

Within this context above, AV-TEST tested the VPN products for their privacy and security aspects.

We are pleased to report that, in this category, PIA and ExpressVPN finished among the top scorers. In particular, PIA won on security given its excellent leak protection and fully flawless kill switch.

AV-TEST tested the VPN products against 6 possible data and information leaks. We quote from the study directly:

  • IP leak through missing or non-functioning kill switch tested by deactivating and reactivating the network card as well as disconnecting the network connection for 1, 10 and 60 seconds and determining the IP directly after reconnection.

ExpressVPN, PIA and NordVPN passed all 6 leak tests, with the exception of NordVPN failing the WebRTC leak test. We've already seen a VoidSec study showing ExpressVPN pass WebRTC leak tests in the past. 

vpn leak testing image

VPN testing: leak results.

The study correctly describes the function of a VPN kill switch: "Kill switch protects the user against sudden drops and re-establishing the connection with an unprotected IP".

Importantly, of the 12 VPN products tested by AV-TEST, NordVPN, Private Internet Access, ExpressVPN and only 1 other (Hotspot Shield, the study’s sponsor) were the only products to pass the kill switch test. In other words, the other VPN products tested had IP leaks. Torrent file-sharing users should take note and focus especially on our recommended VPN services.

testing vpn kill switch image

VPN testing: kill switch results.

By its criteria, AV-TEST found ExpressVPN and PIA among the best. In particular, they were impressed that ExpressVPN's kill switch is enabled by default.

However, not to be outshined, in the words of AV-TEST itself, "Private Internet Access is one of the best VPNs when it comes to leak prevention."

The study mentioned that all products provide a 256-bit AES data encryption, which are set as the default option except for Private Internet Access. This is easy to change in PIA's Settings, but good to highlight nonetheless.

3) Performance

AV-TEST correctly emphasizes the importance of performance because speeds will always suffer when using a VPN due to the encryption and tunneling process which means data does not travel by the most direct route but rather via the VPN servers.

This means performance is heavily dependent on the number of servers and where they are located. The more servers and the more evenly spread out they are, the better performance can be expected to be. "The more servers are available the more evenly the network traffic will be distributed and therefore a more reliable connection can be expected". More server locations also provide more advantages for accessing content worldwide which may be geo-blocked.

The VPN testing for this category was performed with quite rigorous standards, all explained in the study. The point was to ensure an equal playing field for the VPN services tested. It is important to note that tests were only performed on Windows.

AV-TEST measured performance of each VPN by testing download speeds, upload speeds, latency, 4K HD video streaming speeds, and torrent file-sharing speeds.

In the general download speeds tests (“downlink”), Private Internet Access, ExpressVPN and NordVPN performed well and were closely clustered together.

vpn testing latency image

VPN testing: latency results.

Upload speed tests (“uplink”) emulated how fast the user can use cloud services, upload backups, host servers or in general share information through, for example, social media. In this test, PIA finished tops only 10% slower than an unprotected connection, ahead of NordVPN and trailed by ExpressVPN.

In the latency tests, NordVPN had the best latency and ExpressVPN and PIA were virtually tied just behind.

For streaming performance, AV-TEST found all the VPN services up to the task. In their words, “In the performed tests there were no perceived issues when playing the 4k video content".

We were very interested (and pleasantly surprised) to see a torrent file-sharing test among the officially tested criteria. It was also surprising to expressly read this statement in the study as a reason for using a VPN for torrenting.

"There are several reasons for that. Some ISPs might block or limit the torrent protocol. Others fear they might get notices from intellectual property right holders."

Wow! You can tell this study was done in Europe and not the USA. 🙂

In AV-TEST's torrent testing, NordVPN and ExpressVPN were tops with performance measured to be about one-third as fast as an unprotected connection. Torrent speeds for PIA were measured as being about half as fast as ExpressVPN and NordVPN.

Taking all of the above tests into account for assessing overall performance, AV-TEST ranked PIA as best performing (2nd among the 12 VPN products tested) followed by NordVPN and ExpressVPN.

4) Functionality

For this category, AV-TEST examined the amount and quality of features available and customizable options for each VPN service. NordVPN was ranked as having the best functionality and was closely followed by PIA and ExpressVPN.

The study pointed out that PIA had among the lowest money-back guarantee periods (7 days) whereas NordVPN and ExpressVPN offered full 30-day money-back guarantee periods.

The study seemed to award points for lifetime subscriptions, but these are not offered by PIA, NordVPN or ExpressVPN.

Regarding payment options, AV-TEST was absolutely correct when it stated that paying by credit card "is probably one of the most convenient ways but not necessarily the one providing the best privacy." It highlighted that ExpressVPN, PIA and NordVPN all accept bitcoin payments and that NordVPN and PIA accept other crypto-currencies too. However, the study fails to mention that PIA also accepts gift card payments (something it praised the sponsor of the study for doing).

But PIA's low prices were also not lost on the study's authors, "An overall very good product, with competitive pricing ..."

ExpressVPN was highlighted for having the most countries with VPN servers (94) while NordVPN was touted as having the most total servers available (4267).

NordVPN and PIA were further highlighted for including socks5 proxies with their services.

PIA and NordVPN also both received praise for their blocking features MACE and CyberSec, respectively. As the study points out, these ad-, tracker- and malware-blocking technologies offered by PIA and NordVPN are especially handy for mobile users because it helps to save network data. Perhaps because of these extra features, the study seemed to favor PIA and NordVPN when it came to VPN features.

We can't be 100% sure, but we think this statement in the AV-TEST report refers to Netflix. "NordVPN ... also allow[s] the circumvention of several geo-blocked streaming content."

In the end, NordVPN wins for features. It was highlighted as being the only VPN product with a "multihop cascading feature for increased anonymity". That's a fancy reference to NordVPN's Double VPN and perhaps Onion over VPN features.

vpn server testing image

VPN testing: servers.

Shortcomings of the AV-TEST VPN Study (IOHO)

Have no doubt: AV-TEST’s VPN testing study is well conceived and comprehensive.

However, it isn’t perfect in our humble opinion (IOHO). We think readers of the study should keep the following things in mind. 

The Sponsor of the Study "Won"

The study was commissioned by Hotspot Shield Elite and, not surprisingly, this VPN service was found to be the "best" VPN of the 12 tested. Coincidence? It’s perhaps a bit like studies funded by the tobacco industry that conclude smoking is good for you.

Now, AV-TEST is a reputable firm, but we would place some big caveats on their overall conclusion that Hotspot Shield Elite is the best VPN product. 

WTF is the Catapult Hydra Protocol?

In particular, the study's big shocker is this revelation about the Hotspot Shield Elite’s protocol (with our emphasis added):

Hotspot Shield Elite is the main exception to the default protocols [i.e., OpenVPN]. They use their own in-house developed protocol Catapult Hydra, which they claim provides much better performance which has been confirmed in the performance test of this report. Unlike OpenVPN, the Catapult Hydra protocol it is not open source and therefore has not been independently reviewed by us but assumed secure until proven otherwise. According to Hotspot Shield, Catapult Hydra it is used by a large number of security vendors worldwide who trust it enough to use it to provide wireless security to their mobile device customers.

What, what, what!? 

Sorry, but that's just not good enough. Especially not for torrenting, activism or other sensitive online activities.

In other words, Hotspot Shield Elite uses its own "unique" VPN protocol called Catapult Hydra. As you can read above, AV-TEST acknowledges this in the study, but doesn't dig any deeper. Our concern is that this "protocol" is not open source and has not been as thoroughly tested as OpenVPN which all our recommended VPN services use. 

Other Quibbles about the VPN Testing Study

In addition, for some of the information about the Features offered by a VPN, they validated them "... through information provided online by the vendors" (page 4). That's not ideal and falls short of what we would expect from a professional testing firm.

In the category of Usability, Hotspot Shield Elite’s name hardly comes up in the study’s narrative but then suddenly, in the Conclusions section, it is touted as being in shared first place?

In general, AV-TEST awards rankings for "first place" but the scoring and weighting they use is not provided for any of the VPN test elements, so we cannot be sure how the rankings were awarded. This is somewhat subjective and may explain how Hotspot Shield Elite finishes in "first place". We would again highlight the big question marks about the VPN protocol Hotspot Shield Elite uses.

We also question some of AV-TEST’s VPN performance tests because they experienced downgraded download speeds of 90% (25% for upload speeds) averaged out among all VPN products. We certainly experience speed drops using our recommended VPN services, but not to that extent! Of course, the figure cited by AV-TEST was an average figure, so very poor performers among the 12 VPNs they tested could be dragging down the overall average.

Also, for the VPN latency test, AV-TEST used a set location rather than allowing each product to choose "the optimal VPN connection suggested by the product". AV-TEST performed its tests from Germany. This is important because the VPN server locations chosen for the tests could have had a big impact on each of the performance tests. That’s also true for consumers: a VPN’s performance will be impacted by where you live. This is why tests such as these or BestVPN's are interesting as an academic exercise but are no substitute for testing them yourself. By taking a free trial and testing the VPN on/at your own system, location and conditions will you know for sure how a VPN performs for you.

On this same subject, Hotspot Shield Elite does not have the option to connect with an optimal VPN server. We think that is a BIG strike against it, but AV-TEST just glosses over this major shortcoming.

Lastly, the study only went into No-Logging claims very superficially. For some VPN users, especially those engaged in torrent file-sharing, this is one of the most important factors in choosing a VPN product.


​AV-TEST's VPN testing study isn't perfect, but it is interesting in many ways. 

From our perspective, we were gratified to see 3 of our recommended VPN services do very well, especially in the categories of performance and privacy & security. The study reinforced why we think ExpressVPN, Private Internet Access and NordVPN are worth your consideration. 

You can sign up for any (or all) of these 3 VPN services with full peace of mind. ​

Let us know your thoughts in the Comments below. ​

March 3, 2019

Leave a Reply

Your email address will not be published.

{"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}