Blog

Network layer security controls have been used frequently for securing communications, particularly over shared networks such as the Internet because they can provide protection for many applications at once without modifying them.

In the earlier chapters, we discussed that many real-time security protocols have evolved for network security ensuring basic tenets of security such as privacy, origin authentication, message integrity, and non-repudiation.

Most of these protocols remained focused at the higher layers of the OSI protocol stack, to compensate for the inherent lack of security in standard Internet Protocol. Though valuable, these methods cannot be generalized easily for use with any application. For example, SSL is developed specifically to secure applications like HTTP or FTP. But there are several other applications that also need secure communications.

This need gave rise to develop a security solution at the IP layer so that all higher-layer protocols could take advantage of it. In 1992, the Internet Engineering Task Force (IETF) began to define a standard ‘IPsec’.

In this chapter, we will discuss how security is achieved at the network layer using this very popular set of protocol IPsec.

Security in Network Layer

Any scheme that is developed for providing network security needs to be implemented at some layer in the protocol stack as depicted in the diagram below −

Layer	Communication Protocols	Security Protocols
Application Layer	HTTP FTP SMTP	PGP. S/MIME, HTTPS
Transport Layer	TCP /UDP	SSL, TLS, SSH
Network Layer	IP	IPsec

The popular framework developed for ensuring security at the network layer is Internet Protocol Security (IPsec).

Features of IPsec

• IPsec is not designed to work only with TCP as a transport protocol. It works with UDP as well as any other protocol above IP such as ICMP, OSPF etc.
• IPsec protects the entire packet presented to IP layer including higher layer headers.
• Since higher layer headers are hidden which carry port number, traffic analysis is more difficult.
• IPsec works from one network entity to another network entity, not from application process to application process. Hence, security can be adopted without requiring changes to individual user computers/applications.
• Tough widely used to provide secure communication between network entities, IPsec can provide host-to-host security as well.
• The most common use of IPsec is to provide a Virtual Private Network (VPN), either between two locations (gateway-to-gateway) or between a remote user and an enterprise network (host-to-gateway).

Security Functions

The important security functions provided by the IPsec are as follows −

• Confidentiality
  • Enables communicating nodes to encrypt messages.
  • Prevents eavesdropping by third parties.
• Origin authentication and data integrity.
  • Provides assurance that a received packet was actually transmitted by the party identified as the source in the packet header.
  • Confirms that the packet has not been altered or otherwise.
• Key management.
  • Allows secure exchange of keys.
  • Protection against certain types of security attacks, such as replay attacks.

Virtual Private Network

Ideally, any institution would want its own private network for communication to ensure security. However, it may be very costly to establish and maintain such a private network over geographically dispersed areas. It would require managing complex infrastructure of communication links, routers, DNS, etc.

IPsec provides an easy mechanism for implementing Virtual Private Network (VPN) for such institutions. VPN technology allows institution’s inter-office traffic to be sent over the public Internet by encrypting traffic before entering the public Internet and logically separating it from other traffic. The simplified working of VPN is shown in the following diagram −

Overview of IPsec

IPsec is a framework/suite of protocols for providing security at the IP layer.

Origin

In the early 1990s, the Internet was used by few institutions, mostly for academic purposes. But in later decades, the growth of the Internet became exponential due to the expansion of networks and several organizations using it for communication and other purposes.

With the massive growth of the Internet, combined with the inherent security weaknesses of the TCP/IP protocol, the need was felt for a technology that can provide network security on the Internet. A report entitled “Security in the Internet Architecture” was issued by the Internet Architecture Board (IAB) in 1994. It identified the key areas for security mechanisms.

The IAB included authentication and encryption as essential security features in the IPv6, the next-generation IP. Fortunately, these security capabilities were defined such that they can be implemented with both the current IPv4 and futuristic IPv6.

Security framework, IPsec has been defined in several ‘Requests for comments’ (RFCs). Some RFCs specify some portions of the protocol, while others address the solution as a whole.

Operations Within IPsec

The IPsec suite can be considered to have two separate operations, when performed in unison, providing a complete set of security services. These two operations are IPsec Communication and Internet Key Exchange.

• IPsec Communication
  • It is typically associated with standard IPsec functionality. It involves encapsulation, encryption, and hashing the IP datagrams and handling all packet processes.
  • It is responsible for managing the communication according to the available Security Associations (SAs) established between communicating parties.
  • It uses security protocols such as Authentication Header (AH) and Encapsulated SP (ESP).
  • IPsec communication is not involved in the creation of keys or their management.
  • IPsec communication operation itself is commonly referred to as IPsec.
• Internet Key Exchange (IKE)
  • IKE is the automatic key management protocol used for IPsec.
  • Technically, key management is not essential for IPsec communication and the keys can be manually managed. However, manual key management is not desirable for large networks.
  • IKE is responsible for creation of keys for IPsec and providing authentication during key establishment process. Though, IPsec can be used for any other key management protocols, IKE is used by default.
  • IKE defines two protocol (Oakley and SKEME) to be used with already defined key management framework Internet Security Association Key Management Protocol (ISAKMP).
  • ISAKMP is not IPsec specific, but provides the framework for creating SAs for any protocol.

This chapter mainly discusses the IPsec communication and associated protocol employed to achieve security.

IPsec Communication Modes

IPsec Communication has two modes of functioning; transport and tunnel modes. These modes can be used in combination or used individually depending upon the type of communication desired.

Transport Mode

• IPsec does not encapsulate a packet received from upper layer.
• The original IP header is maintained and the data is forwarded based on the original attributes set by the upper layer protocol.
• The following diagram shows the data flow in the protocol stack.

• The limitation of transport mode is that no gateway services can be provided. It is reserved for point-to-point communications as depicted in the following image.

Tunnel Mode

• This mode of IPsec provides encapsulation services along with other security services.
• In tunnel mode operations, the entire packet from upper layer is encapsulated before applying security protocol. New IP header is added.
• The following diagram shows the data flow in the protocol stack.

• Tunnel mode is typically associated with gateway activities. The encapsulation provides the ability to send several sessions through a single gateway.
• The typical tunnel mode communication is as depicted in the following diagram.

• As far as the endpoints are concerned, they have a direct transport layer connection. The datagram from one system forwarded to the gateway is encapsulated and then forwarded to the remote gateway. The remote associated gateway de-encapsulates the data and forwards it to the destination endpoint on the internal network.
• Using IPsec, the tunneling mode can be established between the gateway and individual end system as well.

IPsec Protocols

IPsec uses the security protocols to provide desired security services. These protocols are the heart of IPsec operations and everything else is designed to support these protocols in IPsec.

Security associations between the communicating entities are established and maintained by the security protocol used.

There are two security protocols defined by IPsec — Authentication Header (AH) and Encapsulating Security Payload (ESP).

Authentication Header

The AH protocol provides service of data integrity and origin authentication. It optionally caters to message replay resistance. However, it does not provide any form of confidentiality.

AH is a protocol that provides authentication of either all or part of the contents of a datagram by the addition of a header. The header is calculated based on the values in the datagram. What parts of the datagram are used for the calculation, and where to place the header, depends on the mode cooperation (tunnel or transport).

The operation of the AH protocol is surprisingly simple. It can be considered similar to the algorithms used to calculate checksums or perform CRC checks for error detection.

The concept behind AH is the same, except that instead of using a simple algorithm, AH uses a special hashing algorithm and a secret key known only to the communicating parties. A security association between two devices is set up that specifies these particulars.

The process of AH goes through the following phases.

• When IP packet is received from upper protocol stack, IPsec determine the associated Security Association (SA) from available information in the packet; for example, IP address (source and destination).
• From SA, once it is identified that security protocol is AH, the parameters of AH header are calculated. The AH header consists of the following parameters −

• The header field specifies the protocol of packet following AH header. Sequence Parameter Index (SPI) is obtained from SA existing between communicating parties.
• Sequence Number is calculated and inserted. These numbers provide optional capability to AH to resist replay attack.
• Authentication data is calculated differently depending upon the communication mode.
• In transport mode, the calculation of authentication data and assembling of final IP packet for transmission is depicted in the following diagram. In original IP header, change is made only in protocol number as 51 to indicated application of AH.

• In Tunnel mode, the above process takes place as depicted in the following diagram.

Encapsulation Security Protocol (ESP)

ESP provides security services such as confidentiality, integrity, origin authentication, and optional replay resistance. The set of services provided depends on options selected at the time of Security Association (SA) establishment.

In ESP, algorithms used for encryption and generating authenticators are determined by the attributes used to create the SA.

The process of ESP is as follows. The first two steps are similar to the process of AH as stated above.

• Once it is determined that ESP is involved, the fields of ESP packet are calculated. The ESP field arrangement is depicted in the following diagram.

• Encryption and authentication process in transport mode is depicted in the following diagram.

• In case of Tunnel mode, the encryption and authentication process is as depicted in the following diagram.

Although authentication and confidentiality are the primary services provided by ESP, both are optional. Technically, we can use NULL encryption without authentication. However, in practice, one of the two must be implemented to use ESP effectively.

The basic concept is to use ESP when one wants authentication and encryption and to use AH when one wants extended authentication without encryption.

Security Associations in IPsec

Security Association (SA) is the foundation of IPsec communication. The features of SA are −

• Before sending data, a virtual connection is established between the sending entity and the receiving entity, called “Security Association (SA)”.
• IPsec provides many options for performing network encryption and authentication. Each IPsec connection can provide encryption, integrity, authenticity, or all three services. When the security service is determined, the two IPsec peer entities must determine exactly which algorithms to use (for example, DES or 3DES for encryption; MD5 or SHA-1 for integrity). After deciding on the algorithms, the two devices must share session keys.
• SA is a set of above communication parameters that provides a relationship between two or more systems to build an IPsec session.
• SA is simple in nature and hence two SAs are required for bi-directional communications.
• SAs are identified by a Security Parameter Index (SPI) number that exists in the security protocol header.
• Both sending and receiving entities maintain state information about the SA. It is similar to TCP endpoints which also maintain state information. IPsec is connection-oriented like TCP.

Parameters of SA

Any SA is uniquely identified by the following three parameters −

• Security Parameters Index (SPI).
  • It is a 32-bit value assigned to SA. It is used to distinguish among different SAs terminating at the same destination and using the same IPsec protocol.
  • Every packet of IPsec carries a header containing SPI field. The SPI is provided to map the incoming packet to an SA.
  • The SPI is a random number generated by the sender to identify the SA to the recipient.
• Destination IP Address − It can be IP address of end router.
• Security Protocol Identifier − It indicates whether the association is an AH or ESP SA.

An example of SA between two routers involved in IPsec communication is shown in the following diagram.

Security Administrative Databases

In IPsec, there are two databases that control the processing of the IPsec datagram. One is the Security Association Database (SAD) and the other is the Security Policy Database (SPD). Each communicating endpoint using IPsec should have a logically separate SAD and SPD.

Security Association Database

In IPsec communication, the endpoint holds SA state in Security Association Database (SAD). Each SA entry in the SAD database contains nine parameters as shown in the following table −

Sr.No.	Parameters & Description
1	Sequence Number Counter For outbound communications. This is the 32-bit sequence number provided in the AH or ESP headers.
2	Sequence Number Overflow Counter Sets an option flag to prevent further communications utilizing the specific SA
3	32-bit anti-replay window Used to determine whether an inbound AH or ESP packet is a replay
4	Lifetime of the SA Time till SA remain active
5	Algorithm – AH Used in the AH and the associated key
6	Algorithm – ESP Auth Used in the authenticating portion of the ESP header
7	Algorithm – ESP Encryption Used in the encryption of the ESP and its associated key information
8	IPsec mode of operation Transport or tunnel mode
9	Path MTU(PMTU) Any observed path maximum transmission unit (to avoid fragmentation)

All SA entries in the SAD are indexed by the three SA parameters: Destination IP address, Security Protocol Identifier, and SPI.

Security Policy Database

SPD is used for processing outgoing packets. It helps in deciding what SAD entries should be used. If no SAD entry exists, SPD is used to create new ones.

Any SPD entry would contain −

• Pointer to active SA held in SAD.
• Selector fields – Field in incoming packet from upper layer used to decide application of IPsec. Selectors can include source and destination address, port numbers if relevant, application IDs, protocols, etc.

Outgoing IP datagrams go from the SPD entry to the specific SA, to get encoding parameters. Incoming IPsec datagram gets to the correct SA directly using the SPI/DEST IP/Protocol triple, and from there extracts the associated SAD entry.

SPD can also specify traffic that should bypass IPsec. SPD can be considered as a packet filter where the actions decided upon are the activation of SA processes.

Summary

IPsec is a suite of protocols for securing network connections. It is rather a complex mechanism because instead of giving a straightforward definition of a specific encryption algorithm and authentication function, it provides a framework that allows the implementation of anything that both communicating ends agree upon.

Authentication Header (AH) and Encapsulating Security Payload (ESP) are the two main communication protocols used by IPsec. While AH only authenticates, ESP can encrypt and authenticate the data transmitted over the connection.

Transport Mode provides a secure connection between two endpoints without changing the IP header. Tunnel Mode encapsulates the entire payload IP packet. It adds a new IP header. The latter is used to form a traditional VPN, as it provides a virtual secure tunnel across an untrusted Internet.

Setting up an IPsec connection involves all kinds of crypto choices. Authentication is usually built on top of a cryptographic hash such as MD5 or SHA-1. Encryption algorithms are DES, 3DES, Blowfish, and AES being common. Other algorithms are possible too.

Both communicating endpoints need to know the secret values used in hashing or encryption. Manual keys require manual entry of the secret values on both ends, presumably conveyed by some out-of-band mechanism, and IKE (Internet Key Exchange) is a sophisticated mechanism for doing this online.