The ultimate guide to VPN encryption, protocols, and ciphers

July 31, 2019  |  Callum Tennent

vpn for security and privacy picture

Introduced to the market nearly two decades ago, Virtual Private Networks (VPNs) are a uniquely enduring cornerstone of modern security. Most large organizations still employ a VPN solution to facilitate secure remote access, while millions of consumers rely on similar products to bolster their online privacy, secure public Wi-Fi connections, and circumvent site blocks.

By now, most of us know that a VPN assigns us a new IP address and transmits our online traffic through an encrypted tunnel. But not all VPNs are created equally. Depending on the protocol in use, a VPN might have different speeds, capabilities, or even vulnerabilities.

Encryption protocols and ciphers are at the heart of VPN technology, determining how your ‘secure tunnel’ is actually formed. Each one represents a different solution to the problem of secure, private, and somewhat anonymous browsing.

Though many of us are aware of how a VPN generally works, it’s common to get lost on the fine details of the technology due to the sheer complexity of the subject. This confusion is reinforced by the fact that many VPN providers can be slapdash to the point of misleading when describing the type of encryption that they use. 

This article will provide a simple point of reference for those who want to explore the technologies driving their VPN service. We’ll review different types of encryption, the main VPN protocols available, and the common ciphers behind them.

In explaining the confusing array of terms commonly used by VPNs and other security products, you will be in a stronger position to choose the most secure protocol and assess the claims made by VPN providers with a much more critical eye.

Types of VPN encryption

At a very basic level, encryption involves substituting letters and numbers to encode data so that only authorized groups can access and understand it.

We now use powerful algorithms called ciphers to perform encryption and decryption. These ciphers simply denote a series of well-defined steps that can be followed repeatedly. The operation of a cipher usually depends on a piece of auxiliary information called a key; without knowledge of the key, it is extremely difficult – if not impossible – to decrypt the resulting data.

When talking about encryption today, we generally refer to a mixture of cipher and key-length, which denotes the number of ‘bits’ in a given key. For example, Blowfish-128 is the Blowfish cipher with a key length of 128 bits. Generally speaking, a short key length means poor security as it is more susceptible to violation by brute-force attacks.

A key length of 256 bits is the current ‘gold standard’. This cannot be brute-forced as it would take billions of years to run through all the possible bit combinations. There are a few key concepts in the world of encryption:

Symmetric-key

This is where the key for encryption and decryption is the same, and both communicating parties must possess the same key in order to communicate. This is the type of encryption used in VPN services.

Public-key

Here, software is used to create sets of public and private keys. The public key is used to encrypt data, which is then sent to the owner of the private key. They then use this private key to decrypt the messages.

Network-based firewall service

Fully managed, cloud-based firewall providing continuous inspection and treatment of internet traffic.

Learn more

Handshake encryption (RSA)

Securely connecting to a VPN server requires the use of public-key encryption through a TLS handshake. While a cipher secures your actual data, this handshake secures your connection. 

This is typically done through the RSA (Rivest-Shamir-Adleman) algorithm, which has essentially been the foundation of internet security for about two decades. Sadly, it now seems probable that RSA-1204 has been cracked by the NSA.

Today, most VPN services have moved on from RSA-1024, but a small minority still incorporate it. These services should be avoided. It’s best to find a service offering RSA-2048, which remains secure.

Secure Hash Algorithm (SHA)

The Secure Hash Algorithm (SHA) is used to authenticate data and SSL/TLS connections. In this process, a unique fingerprint is created to validate the TLS certificate – that is, to check you’re connecting to the server you’re supposed to be. Without this, a hacker could re-route your traffic to their own server instead of your VPN provider’s.

Protocols

VPN protocols represent the processes and instructions that VPN clients rely on to negotiate a secure connection between a device and a VPN server. There are various different protocols commonly supported by commercial VPN services. Here are the main protocols you need to know about:

SSL and TLS

Though not specifically a VPN protocol, this is the type of encryption most people are familiar with. It is used by both OpenVPN and all HTTPS-secured sites. Building on its predecessor – the SSL protocol (Secure Socket Layer) – TLS (Transport Layer Security) uses a mixture of symmetric and public-key encryption to secure network communications.

During this process, your web browser uses public-key encryption to communicate with a website’s server and share the symmetric keys used to secure the transmitted data. The only issue here emerges when a website’s server only uses one private key to initiate all secure sessions. If this key is somehow compromised, a hacker can easily intercept and read any communications with that website. Thanks to Edward Snowden, we now know that the U.S. National Security Agency (NSA) has leveraged this vulnerability to gather swathes of seemingly secure information.

Fortunately, it’s increasingly common for unique private keys to be generated and then disposed of after each TLS session. This is referred to as Perfect Forward Secrecy or using ‘ephemeral encryption keys’. Though nowhere near universal yet, the use of ephemeral keys is on the rise.

OpenVPN

This is the standard protocol used by most commercial VPN providers. It’s widely configurable, very secure, and operational on almost all VPN-capable devices.

OpenVPN is open-source, which means its codebase is publicly available for inspection. Researchers can test for vulnerabilities, substantiate security claims, and refine the product. It uses TLS protocols and the OpenSSL library in combination with a range of other tools to create a reliable and secure VPN connection. Custom OpenVPN applications are available from commercial VPN providers, but the primary source code is created and developed by the OpenVPN Project.

OpenVPN encryption is comprised of two parts: data channel and control channel encryption. The former consists of a cipher and hash authentication – both of which will be discussed later in this article – and is used to secure the data itself. The latter uses TLS encryption to secure the connection between your computer and the VPN server, and consists of handshake encryption, cipher, and hash authentication. For maximum security, both the data and control channel encryption should be as strong as possible.

It’s important to be aware that some VPN providers don’t use the same level of encryption on both of these channels. The stronger the encryption used, the slower the connection will be, so many providers use weaker encryption on the data channel as a cheap shortcut to a faster connection. If you see a VPN service advertising its security merits, be sure to check that these claims apply to both channels.

OpenVPN works on both the UDP (User Datagram Protocol) port and TCP (Transmission Control Protocol) port. The former transmits data packets without requiring confirmation of their arrival before transmitting the next packet, resulting in lower latency. The latter – TCP – is the most common connection protocol on the web. When using this, a confirmation of the packet’s reception is required before the next data packet can be sent.

Normal HTTPS traffic uses TCP port 443. Combining this port with OpenVPN means it’s difficult to distinguish VPN connections from other connections commonly used by email providers, online banking, and e-commerce sites. This means OpenVPN is very hard to block, which is great for those looking to use a VPN in highly-censored countries. 

Ultimately, OpenVPN is the recommended VPN protocol under most circumstances. It is versatile, secure, and open source. While it has no real downsides, it’s important that it is implemented well. This means use of strong encryption with Perfect Forward Secrecy.

IKEv2/IPsec

Internet Key Exchange version 2 (IKEv2) is a security association protocol that sets the foundation for a VPN connection by establishing an authenticated and encrypted connection between two parties. It was created by Microsoft and Cisco, and is natively supported by iOS, Windows 7 and later, and Blackberry.

As part of the IPsec internet security toolbox, IKEv2 uses other IPsec tools to provide comprehensive VPN coverage. This includes the Mobility and Multihoming protocol, which ensures the VPN stays connected as the client moves from one connection to another. This makes IKEv2 a very dependable and stable protocol for mobile devices.

As part of the IPsec suite, IKEv2 works with most leading encryption algorithms, which is testament to its security. That being said, the protocol is less efficient when trying to connect out of a highly censored country. What’s more, IKEv2 uses a method called the Diffie Hellman process to exchange the keys it uses to secure your data. During the Snowden leaks, slides in an NSA presentation revealed they may have found a way to break this procedure. Researchers working on a potential vulnerability estimated that, with hundreds of millions of dollars in computing power investments, the NSA could potentially break into as many as 66% of IPsec connections.

Ultimately, IKEv2 is a reasonably secure and fast protocol. Mobile users in particular may prefer it to OpenVPN due to its stability throughout interrupted internet connections. For users of platforms like Blackberry, it’s essentially the only option available. 

L2TP/IPsec

Layer 2 Tunnelling Protocol (LT2P) is a proprietary Microsoft protocol that creates a connection between you and the other servers in your VPN. Like IKEv2, it relies on other tools in the IPsec suite to encrypt and authenticate your traffic as it can’t perform these services itself. This protocol has a few convenient features, but various issues prevent it from outperforming its competitors.

Unlike other more sophisticated protocols, L2TP doesn’t have any smart ways to bypass firewalls. Surveillance-minded system administrators use firewalls to block VPNs to prevent users from hiding their traffic. People using L2TP here are an easy target as the protocol uses a relatively small number of fixed ports.

L2TP encapsulates data twice; while this can be useful for some applications, it makes it slower compared to other protocols that encapsulate data only once.

Like IKEv2, LT2P is typically used with IPsec. Some VPN providers have been known to administer this combination poorly, often using pre-shared keys that are available to download online. While these are solely used for authentication and not decryption, a malicious actor could use this key to imitate a VPN server. 

For those willing to overlook these issues, LT2P/IPsec is a good choice for non-critical use given the number of different encryption protocols it can accept and the fact it is natively supported by so many platforms. This is especially true for legacy devices that do not support OpenVPN.

SSTP

Secure Socket Tunnelling Protocol (SSTP) is a proprietary Microsoft-owned protocol based on SSL 3.0. Like OpenVPN, it supports AES-256 and can use TCP port 443. SSTP offers most of the advantages of OpenVPN but is primarily only for Windows.

SSTP’s close association with Microsoft means you can be confident that any device using Vista or a later Windows OS will either support SSTP or have it built-in. Likewise, you may be able to receive support from Microsoft if you have problems in implementing the protocol. 

That being said, SSTP is not open-source. It is therefore not possible to analyse or refute suggestions of backdoors and vulnerabilities, or for security researchers to test for potential weaknesses. Microsoft has been known to cooperate with institutions like the NSA in the past, so some suspect the system may include backdoors that allow these agencies to track targeted users. For this reason, the risks of its close integration with Windows largely outweigh any benefits.

WireGuard

WireGuard is a relatively new tunnelling protocol that seeks to offer better performance and faster speeds than OpenVPN.

The protocol is designed to resolve some of the negative issues commonly associated with IPsec and OpenVPN: frequent disconnections, complex setup for users looking to manually configure, extended reconnection times, and heavy codebases which can make it difficult for researchers to spot bugs. 

WireGuard aims to surpass traditional protocols by using more modern ciphers. It’s codebase is only 4000 lines – around 1% of OpenVPN’s and IPsec’s.

Though it’s still in development, preliminary tests suggest that WireGuard is fast, secure and reconnects instantaneously. It has yet to fully ‘prove itself’, but a rising number of VPN services are being applauded for including it with their clients, including IVPN and Mullvad VPN.

PPTP

Designed to create a VPN over dial-up connection, the Point-to-Point Tunnelling Protocol (PPTP) is an old VPN protocol developed in 1999 by a team funded by Microsoft.

It still retains some positives: it’s widely compatible, doesn’t require extra software, and is quite fast.

The biggest issue is with security. PPTP is proven to be insecure and very easy to break. PPTP traffic is also easy to obstruct as it depends on the GRE protocol which is easily stumped by a firewall. Avoid using PPTP unless it is totally necessary for non-critical use.

Choosing a VPN protocol

It’s best to opt for OpenVPN where available, especially when setup is handled by a third-party app. L2TP/IPsec is probably the most widely used alternative and offers the next-best in terms of security. SSTP is a reasonable option for Windows users that trust proprietary tech from Microsoft, while IKEv2 is a fast alternative for the few devices – particularly mobile – that support it. PPTP should only be used as a very last resort.

Ciphers

While key length refers to the amount of numbers or ‘bits’ involved, a cipher is the formula or algorithm used to actually carry out the encryption. The strength of a cipher is dependent on both the key length and the strength of these formulas.

It’s worth mentioning that a larger key length means more calculations are involved, so more processing power is required. This affects how fast data can be encrypted and decrypted. Commercial VPN providers must therefore negotiate the balance between security and usability when choosing an encryption scheme.

AES

The Advanced Encryption Standard (AES) is a symmetric-key cipher established in 2001 by The National Institute of Standards and Technology (NIST). It essentially represents the ‘gold standard’ of the contemporary VPN industry. Though AES-128 is considered secure, AES-256 is preferable as it offers stronger protection.

Blowfish

Blowfish is the default cipher used in OpenVPN, though it can be configured to others. The most commonly used version is Blowfish-128, but it theoretically can range from 32 to 448 bits.

The cipher is generally secure but does sport some vulnerabilities. It should generally only be chosen when AES-256 is unavailable.

Camellia

This cipher is similar to AES. The essential difference is that Camellia is not certified by NIST, the American organization that designed AES.

There is definitely an argument for trusting a cipher that doesn’t sport strong ties to the US government. That being said, Camellia is very rarely available in commercial VPN software, and hasn’t been vetted as thoroughly AES.

Securing your VPN connection

The types of encryption, protocols, and ciphers above should help to strengthen your understanding of what drives and secures your VPN connection. If you’re choosing a consumer or enterprise VPN provider, consider these concepts when assessing the claims that providers make.

Be sceptical. It’s not uncommon to see VPN services advertise their use of ‘military-grade’ 256-bit AES OpenVPN encryption; in reality, this doesn’t reveal all that much. While AES-256 might be a strong cipher, your data simply won’t be secure if other parts of the encryption suite are weak. Consider their use of hash authentication, perfect forward secrecy, and the difference in encryption settings on both the data and control channels.

Share this with others

Get price Free trial