Internet Protocol Security (IPSec) is a powerful set of protocols to safeguard data transmission over IP networks. It operates at the network layer, providing encryption and authentication for every IP packet, effectively securing all applications that pass traffic across the network.
IPSec offers two modes:
- Transport mode encrypts only the payload of the IP packet and is suitable for direct communication between two hosts.
- Tunnel mode encrypts and encapsulates the entire IP packet within a new one, which is ideal for network-to-network communication, like virtual private networks (VPNs).
IPSec tunneling is a common method to secure data passing through a network or over the internet. Encrypting the entire IP packet, including the header, conceals sensitive information like original IP addresses, making it difficult for eavesdroppers to intercept data.
IPSec tunneling supports multiple encryption algorithms, allowing flexibility and customization based on a network's specific security needs. It also ensures data integrity, using cryptographic checksums to prevent tampering. IPSec tunneling also supports perfect forward secrecy (PFS) for even greater security, protecting past and future sessions if a session key is compromised.
Internet Key Exchange version 2 (IKEv2) , which is a tunneling protocol that manages the negotiation of cryptographic keys for secure communication, and Secure Socket Tunneling Protocol (SSTP) , which is a VPN protocol created by Microsoft and uses the same port as Secure Sockets Layer (SSL) / Transport Layer Security (TLS) communication (443), work alongside IPSec. By the way, you can find great information online if you want to learn more about SSL or TLS .
This article explores how IPSec tunneling works, its protocols, the uses of encryption and hashing, and how these technologies combine for a strong network security solution.
Why IPSec Tunneling Matters
When a network is outside of the organization's control, it's an untrusted network . Sensitive data often travels across untrusted networks like the public internet, cloud provider networks, or home networks, and organizations need to encrypt any data that passes through their networks.
IPSec tunneling creates a protected communication channel through untrusted networks. Encrypting the entire IP packet (including the header) masks sensitive information like the original sender and recipient addresses. There are other options for IPSec communication, such as transport mode, but tunneling protects more information if the packets are intercepted.
Your IPSec configuration could choose either IKEv2 or SSTP for the initial connection and the other protocol for fallback. IKEv2 is more secure , and its most recent versions offer options for streamlined communication to reduce overhead. In contrast, SSTP is an acceptable alternative for scenarios when data protection is not as much of a concern or when the devices in communication do not support IKEv2.
For extra protection, IPSec's PFS guarantees that even if a session key is compromised, your past and future data remains secure.
IPSec Tunneling Use Cases
IPSec tunneling is commonly implemented to create site-to-site VPNs, safeguarding all communications between different company branches, even when those communications travel over the internet.
IPSec is also used to safeguard sensitive information. Its powerful encryption ensures the confidentiality of financial, personal, or proprietary company data, even if it's intercepted during transmission.
Additionally, IPSec allows employees to connect to corporate networks from anywhere, securing their communication from interception as they work from home or on the go.
How IPSec Security Works
IPSec tunneling establishes a secure communication channel by authenticating each party and then negotiating mutually acceptable data protection methods. Hashing and encryption are the two types of data transformations that provide data protection through authentication, confidentiality, and integrity. While both these functions rely on mathematical operations to protect data security, their difference is significant for understanding IPSec:
Hashing
Hashing is a one-way, fixed-length, and repeatable method that creates a unique mathematical output. Using the same data and hashing algorithm always produces the same output. In contrast, any changes to the data also change the hash output, and recreating the same hash from a different input is almost impossible. Because the output is repeatable and reliable, hashing proves data integrity.
Encryption
Encryption transforms data so that it's unreadable without the corresponding decryption key. Encryption is reversible; with the right key, the data can be decrypted back to its original form.
Encryption protects data confidentiality. Encryption keys are combined with the data by applying a mathematical transformation.
Shared encryption keys (also called shared secret keys or preshared keys) are encryption keys that can be used both to encrypt and decrypt data. Shared keys must be confidential because they can be used by any party with access to the algorithm and the encrypted output. Using shared keys is called symmetric encryption . Asymmetric encryption uses a preshared public key to encrypt and a second, related private key to decrypt. Sharing the public key does not compromise the private key.
How IPSec Tunneling Works
At its very basic level, IPSec tunneling works by formally initiating communication, establishing a communication channel between two verified parties, encrypting content, and transmitting information.
Important Definitions
Before learning more about the mechanism of tunneling, let's briefly define a few important concepts:
- Initiator: The initiator is the system that needs to send protected data across a network. The initiator triggers the IPSec process.
- Responder: The responder is the system that receives the protected data, processes it, or passes it on. It waits for the communication requests.
- Interesting traffic: Communication requests are centered around interesting traffic. The interesting traffic is initiator-generated network communications that match a preconfigured IPSec security policy.
- Machine automation: Machine authentication occurs when devices prove their identities to each other. The proof can happen through (for instance) digital certificate verification or preshared encryption keys.
While these definitions may be a bit simplistic, they're helpful reference points for what you'll read later, and they'll make it easier to understand communication setup and data exchange in IPSec.
If you're a visual person, this graphic of the IPSec communication steps can help clarify the stages as you read further.
Initiate Communication
The IPSec tunneling process starts with the initiator. When the initiator generates network traffic matching its configured IPSec security policies, the initiator then begins the steps to initiate an IPSec tunnel. The initiator sends a proposal to a designated responder, outlining its supported encryption options, authentication methods, and other parameters. The responder acknowledges the proposal and replies with any compatible choices. This communication allows the machines to create an Internet Security Association and Key Management Protocol (ISAKMP) Security Association (SA) . The ISAKMP (sometimes referred to as SA) is an initial agreement about how the secure data tunnels can be configured.
This is IKE phase 1 . This step does not create the IPSec tunnel itself. Instead, it creates a preliminary secure communication to negotiate options for The IPSec tunnel.
Initial Key Exchange
To encrypt and decrypt the data, the initiator and responder need a shared secret key. This key is generated through a collaborative process.
The systems first agree on a large prime number and a generator (mathematical terms used in cryptography). Then each selects a private key and calculates a corresponding public key. The public keys are exchanged, and each system uses its private key combined with the other's public key to independently calculate the same shared secret key. This method, often based on the Diffie-Hellman key exchange , ensures that even if an attacker intercepts the public keys, they can't figure out the shared secret without the private keys.
Authentication
Before any sensitive data is sent, IPSec verifies the identities of both the initiator and responder. This is essential to prevent unauthorized parties from accessing the secure tunnel.
Authentication is the confirmation that communication is happening between the intended parties and that the exchanged keys are secure. This step typically involves techniques like preshared keys (a secret known by both parties) or digital certificates (such as an electronic ID card issued by a trusted authority).
IKE phase 2 occurs after authentication. The two systems negotiate again, this time agreeing upon the SA of the IPSec tunnel. When this negotiation is complete, the IPSec tunnel is formed.
Encryption
IPSec encryption is powered by two protocols: the Authentication Header (AH) and the Encapsulating Security Payload (ESP) . Either of these two encryption protocols can be used separately, or they can be combined. This increases the work needed to generate the encrypted packet, but it provides all the features available to each protocol—the authenticated headers of AH with the confidentiality of ESP.
AH provides data integrity, authentication, and anti-replay protection by hashing the packet, as defined earlier. This protocol authenticates the entire packet, not just the content, but it does not provide confidentiality.
ESP provides confidentiality, authentication, and other security features through encryption and additional headers. ESP works by encapsulating the data transmitted within an ESP header and trailer. The entire package, including the ESP header and trailer, is then encrypted using a chosen encryption algorithm (such as Advanced Encryption Standard [AES], Data Encryption Standard [DES], or Triple DES [3DES]). This encrypted data travels through the IPSec tunnel. At the receiving end, the data is decrypted using the same encryption algorithm and the shared secret key established previously.
Transmission
With encryption in place, data can now be securely transmitted. IPSec transmission generally uses tunnel mode , where the original IP packet is encrypted and encapsulated within a new packet. This is well-suited for protecting all communication over insecure networks, especially in VPNs. Tunnel mode is preferred for scenarios where there are gateways between the initiator and responder because it protects the content and the header.
Alternatively, the transport mode encrypts the payload of the IP packet but leaves the header untouched. This mode provides end-to-end security between two secure systems and offers faster performance. However, the header information, including source and destination IP addresses, can be intercepted in transit. Although IP addresses may not be highly sensitive, your organization may prefer to keep them private. The information contained in an IP header includes the source and destination IP addresses. The IP addresses are not necessarily sensitive information, but your organization may prefer not to expose them.
Conclusion
IPSec tunneling is an adaptable solution for securing sensitive data over your network, providing a secure channel through otherwise untrusted networks.
Businesses rely on IPSec for secure corporate communications, especially when implementing site-to-site VPNs. IPSec tunneling provides secure remote access to employees, allowing them to work from anywhere while ensuring their data remains protected.
Understanding how IPSec tunneling works is crucial for anyone involved in network security as it is the only option available to safeguard your data within your network infrastructure.