IP Security Overview

Def: Internet Protocol security (IPSec) is a framework of open standards for protecting
communications over Internet Protocol (IP) networks through the use of cryptographic
security services. IPSec supports network-level peer authentication, data origin
authentication, data integrity, data confidentiality (encryption), and replay protection.

Need for IPSec

In Computer Emergency Response Team (CERT)’s 2001 annual report it listed 52,000
security incidents in which most serious types of attacks included IP spoofing, in which
intruders create packets with false IP addresses and exploit applications that use
authentication based on IP and various forms of eavesdropping and packet sniffing, in
which attackers read transmitted information, including logon information and database
contents. In response to these issues, the IAB included authentication and encryption as
necessary security features in the next-generation IP i.e. IPv6.

Applications of IPSec
IPSec provides the capability to secure communications across a LAN, across private
and public wide area networks (WAN’s), and across the Internet.

• Secure branch office connectivity over the Internet: A company can build a secure
virtual private network over the Internet or over a public WAN. This enables a business to
rely heavily on the Internet and reduce its need for private networks, saving costs and
network management overhead.
• Secure remote access over the Internet: An end user whose system is equipped with IP
security protocols can make a local call to an Internet service provider (ISP) and gain
secure access to a company network. This reduces the cost of toll charges for travelling
employees and telecommuters.
• Establishing extranet and intranet connectivity with partners: IPSec can be used to
secure communication with other organizations, ensuring authentication and
confidentiality and providing a key exchange mechanism.
• Enhancing electronic commerce security: Even though some Web and electronic
commerce applications have built-in security protocols, the use of IPSec enhances that
security.

The principal feature of IPSec enabling it to support varied applications is that it can encrypt
and/or authenticate all traffic at IP level. Thus, all distributed applications, including remote
logon, client/server, e-mail, file transfer, Web access, and so on, can be secured.

1
 2

The following figure shows a typical scenario of IPSec usage. An organization

maintains LANs at dispersed locations. Non secure IP traffic is conducted on each LAN.

The IPSec protocols operate in networking devices, such as a router or firewall that connect
each LAN to the outside world. The IPSec networking device will typically encrypt and
compress all traffic going into the WAN, and decrypt and decompress traffic coming from the
WAN; these operations are transparent to workstations and servers on the LAN. Secure
transmission is also possible with individual users who dial into the WAN. Such user
workstations must implement the IPSec protocols to provide security.

Benefits of IPSec
The benefits of IPSec are listed below:
• IPSec in a firewall/router provides strong security to all traffic crossing the perimeter

• IPSec in a firewall is resistant to bypass

• IPSec is below transport layer(TCP,UDP), hence transparent to applications

• IPSec can be transparent to end users

• IPSec can provide security for individual users if needed (useful for offsite
workers and setting up a secure virtual subnetwork for sensitive applications)

Routing Applications
IPSec also plays a vital role in the routing architecture required for internetworking. It assures
that:
• router advertisements come from authorized routers
• neighbor advertisements come from authorized routers
• redirect messages come from the router to which initial packet was sent
• A routing update is not forged
IP Security Architecture
To understand IP Security architecture, we examine IPSec documents first and then move on
to IPSec services and Security Associations.

IPSec Documents

The IPSec specification consists of numerous documents. The most important of these, issued
in November of 1998, are RFCs 2401, 2402, 2406, and 2408:

• RFC 2401: An overview of a security architecture

• RFC 2402: Description of a packet authentication extension to IPv4 and IPv6
• RFC 2406: Description of a packet encryption extension to IPv4 and IPv6
• RFC 2408: Specification of key management capabilities

Support for these features is mandatory for IPv6 and optional for IPv4. In both cases, the
security features are implemented as extension headers that follow the main IP header. The
extension header for authentication is known as the Authentication header; that for encryption
is known as the Encapsulating Security Payload (ESP) header. In addition to these four RFCs,
a number of additional drafts have been published by the IP Security Protocol Working Group
set up by the IETF. The documents are divided into seven groups, as depicted in following
figure:

• Architecture: Covers the general concepts, security requirements, definitions, and

mechanisms defining IPSec technology
• Encapsulating Security Payload (ESP): Covers the packet format and general issues
related to the use of the ESP for packet encryption and, optionally, authentication.
• Authentication Header (AH): Covers the packet format and general issues related to the
use of AH for packet authentication.

• Encryption Algorithm: A set of documents that describe how various encryption

algorithms are used for ESP.
• Authentication Algorithm: A set of documents that describe how various authentication
algorithms are used for AH and for the authentication option of ESP.
• Key Management: Documents that describe key management schemes.
•
• Domain of Interpretation (DOI): Contains values needed for the other documents to
relate to each other. These include identifiers for approved encryption and authentication
algorithms, as well as operational parameters such as key lifetime.
IPSec Services
IPSec architecture makes use of two major protocols (i.e., Authentication Header and ESP
protocols) for providing security at IP level. This facilitates the system to beforehand choose
an algorithm to be implemented, security protocols needed and any cryptographic keys
required to provide requested services. The IPSec services are as follows:

 Connectionless Integrity:- Data integrity service is provided by IPSec via AH which

prevents the data from being altered during transmission.
 Data Origin Authentication:- This IPSec service prevents the occurrence of replay
attacks, address spoofing etc., which can be fatal
 Access Control:- The cryptographic keys are distributed and the traffic flow is controlled
in both AH and ESP protocols, which is done to accomplish access control over the data
transmission.
 Confidentiality:- Confidentiality on the data packet is obtained by using an encryption
technique in which all the data packets are transformed into ciphertext packets which are
unreadable and difficult to understand.
 Limited Traffic Flow Confidentiality:- This facility or service provided by IPSec
ensures that the confidentiality is maintained on the number of packets transferred or
received. This can be done using padding in ESP.
 Replay packets Rejection:- The duplicate or replay packets are identified and discarded
using the sequence number field in both AH and ESP.

Security Associations

Since IPSEC is designed to be able to use various security protocols, it uses Security
Associations (SA) to specify the protocols to be used. SA is a database record which specifies
security parameters controlling security operations. They are referenced by the sending host
and established by the receiving host. An index parameter called the Security Parameters
Index (SPI) is used. SAs are in one direction only and a second SA must be established for
the transmission to be bi-directional. A security association is uniquely identified by three
parameters:

• Security Parameters Index (SPI): A bit string assigned to this SA and having local
significance only. The SPI is carried in AH and ESP headers to enable the receiving
system to select the SA under which a received packet will be processed.
• IP Destination Address: Currently, only unicast addresses are allowed; this is the address
of the destination endpoint of the SA, which may be an end user system or a network
system such as a firewall or router.
• Security Protocol Identifier: This indicates whether the association is an AH or ESP
security association.
SA Parameters
In each IPSec implementation, there is a nominal Security Association Database that defines
the parameters associated with each SA. A security association is normally defined by the
following parameters:
• Sequence Number Counter: A 32-bit value used to generate the Sequence Number field
in AH or ESP headers
• Sequence Counter Overflow: A flag indicating whether overflow of the Sequence
Number Counter should generate an auditable event and prevent further transmission of
packets on this SA (required for all implementations).
• Anti-Replay Window: Used to determine whether an inbound AH or ESP packet is a
replay
• AH Information: Authentication algorithm, keys, key lifetimes, and related parameters
being used with AH (required for AH implementations).
• ESP Information: Encryption and authentication algorithm, keys, initialization values,
key lifetimes, and related parameters being used with ESP (required for ESP
implementations).
• Lifetime of This Security Association: A time interval or byte count after which an SA
must be replaced with a new SA (and new SPI) or terminated, plus an indication of which
of these actions should occur (required for all implementations).
• IPSec Protocol Mode: Tunnel, transport, or wildcard (required for all implementations).
These modes are discussed later in this section.
• Path MTU: Any observed path maximum transmission unit (maximum size of a packet
that can be transmitted without fragmentation) and aging variables (required for all
implementations).
Transport and Tunnel Modes
Both AH and ESP support two modes of use: transport and tunnel mode.

Transport Mode SA Tunnel Mode SA

AH Authenticates IP payload and Authenticates entire inner IP

selected portions of IP header and packet plus selected portions of
IPv6 extension headers outer IP header

ESP Encrypts IP payload and any IPv6 Encrypts inner IP packet

extesion header

ESP with Encrypts IP payload and any IPv6 Encrypts inner IP packet.
authentication extesion header. Authenticates IP Authenticates inner IP packet.
payload but no IP header

IP sec can be used (both AH packets and ESP packets) in two modes
• Transport mode: the IP sec header is inserted just after the IP header –this contains the
security information, such as SA identifier, encryption, authentication
 Typically used in end-to-end communication
 IP header not protected

• Tunnel mode: the entire IP packet, header and all, is encapsulated in the body of a new
IP packet with a completely new IP header
 Typically used in firewall-to-firewall communication
 Provides protection for the whole IP packet
 No routers along the way will be able (and will not need) to check the content of the
packets

End-to-End versus End-to-Intermediate Authentication

Authentication Header:
The Authentication Header provides support for data integrity and authentication of IP
packets. The data integrity feature ensures that undetected modification to a packet's content
in transit is not possible. The authentication feature enables an end system or network device
to authenticate the user or application and filter traffic accordingly; it also prevents the
address spoofing attacks observed in today's Internet. The AH also guards against the replay
attack. Authentication is based on the use of a message authentication code (MAC), hence the
two parties must share a secret key. The Authentication Header consists of the following
fields:

IPSec Authentication Header

• Next Header (8 bits): Identifies the type of header immediately following this header.
• Payload Length (8 bits): Length of Authentication Header in 32-bit words, minus 2. For
example, the default length of the authentication data field is 96 bits, or three 32-bit
words. With a three-word fixed header, there are a total of six words in the header, and the
Payload Length field has a value of 4.
• Reserved (16 bits): For future use.
• Security Parameters Index (32 bits): Identifies a security association.
• Sequence Number (32 bits): A monotonically increasing counter value, discussed later.
• Authentication Data (variable): A variable-length field (must be an integral number of
32-bit words) that contains the Integrity Check Value (ICV), or MAC, for this packet.

Encapsulating Security Payload:

The Encapsulating Security Payload provides confidentiality services, including
confidentiality of message contents and limited traffic flow confidentiality. As an optional
feature, ESP can also provide an authentication service.

ESP Format
The following figure shows the format of an ESP packet. It contains the following fields:
 IPSec ESP format
• Security Parameters Index (32 bits): Identifies a security association.
• Sequence Number (32 bits): A monotonically increasing counter value; this provides an
anti-replay function, as discussed for AH.
• Payload Data (variable): This is a transport-level segment (transport mode) or IP packet
(tunnel mode) that is protected by encryption.
• Padding (0-255 bytes): This field is used to make the length of the plaintext to be a
multiple of some desired number of bytes. It is also added to provide confidentiality.
• Pad Length (8 bits): Indicates the number of pad bytes immediately preceding this field.
• Next Header (8 bits): Identifies the type of data contained in the payload data field by
identifying the first header in that payload (for example, an extension header in IPv6, or
an upper-layer protocol such as TCP).
• Authentication Data (variable): A variable-length field (must be an integral number of
32-bit words) that contains the Integrity Check Value computed over the ESP packet
minus the Authentication Data field.

Adding encryption makes ESP a bit more complicated because the encapsulation surrounds
the payload rather than precedes it as with AH: ESP includes header and trailer.

fields to support the encryption and optional authentication. It also provides Tunnel and
Transport modes. The IPSec RFCs don't insist upon any particular encryption algorithms, but
we find DES, triple-DES, AES, and Blowfish in common use to shield the payload from
prying eyes. The algorithm used for a particular connection is specified by the Security
Association and this SA includes not only the algorithm, but the key used. Unlike AH, which
provides a small header before the payload, ESP surrounds the payload it's protecting. The
Security Parameters Index and Sequence Number serve the same purpose as in AH, but we
find padding, the next header, and the optional Authentication Data at the end, in the ESP
Trailer.
It's possible to use ESP without any actual encryption (to use a NULL algorithm), which
nonetheless structures the packet the same way. This provides no confidentiality, and it only
makes sense if combined with ESP authentication. Padding is provided to allow block-
oriented encryption algorithms room for multiples of their block size, and the length of that
padding is provided in the pad len field. The next hdr field gives the type (IP, TCP, UDP,
etc.) of the payload in the usual way, though it can be thought of as pointing "backwards" into
the packet rather than forward as we've seen in AH. In addition to encryption, ESP can also
optionally provide authentication, with the same HMAC as found in AH. Unlike AH,
however, this authentication is only for the ESP header and encrypted payload: it does not
cover the full IP packet.

Combining Security Associations:

An individual SA can implement either the AH or ESP protocol but not both. Multiple SAs
must be employed for traffic flow to achieve the desired IPSec services. The term security
association bundle refers to a sequence of SAs through which traffic must be processed to
provide a desired set of IPSec services. The SAs in a bundle may terminate at different
endpoints or at the same endpoints. Security associations may be combined into bundles in
two ways:

• Transport adjacency: Refers to applying more than one security protocol to the same IP
packet, without invoking tunnelling.
• Iterated tunnelling: Refers to the application of multiple layers of security protocols
effected through IP tunnelling. This approach allows for multiple levels of nesting, since
each tunnel can originate or terminate at a different IPSec site along the path.

Authentication Plus Confidentiality

Encryption and authentication can be combined in order to transmit an IP packet that has both
confidentiality and authentication between hosts. There are several approaches for this:

ESP with Authentication Option

In this approach, the encryption is carried out on a data packet prior to its authentication.
This can be represented using the following two cases:

• Transport Mode ESP

• Tunnel Mode ESP

Transport Adjacency
Another way to apply authentication after encryption is to use two bundled transport
SAs, with the inner being an ESP SA and the outer being an AH SA. In this case ESP is used
without its authentication option. Because the inner SA is a transport SA, encryption is
applied to the IP payload. The resulting packet consists of an IP header (and possibly IPv6
header extensions) followed by an ESP. AH is then applied in transport mode, so that
authentication covers the ESP plus the original IP header (and extensions) except for mutable
fields. The advantage of this approach over simply using a single ESP SA with the ESP
authentication option is that the authentication covers more fields, including the source and
destination IP addresses. The disadvantage is the overhead of two SAs versus one SA.

Transport-Tunnel Bundle
The use of authentication prior to encryption might be preferable for several reasons. First,
because the authentication data are protected by encryption, it is impossible for anyone to
intercept the message and alter the authentication data without detection. Second, it may be
desirable to store the authentication information with the message at the destination for
later reference. It is more convenient to do this if the authentication information applies to the
unencrypted message; otherwise the message would have to be reencrypted to verify the
authentication information.

One approach to applying authentication before encryption between two hosts is to

use a bundle consisting of an inner AH transport SA and an outer ESP tunnel SA. In this case,
authentication is applied to the IP payload plus the IP header (and extensions) except for
mutable fields. The resulting IP packet is then processed in tunnel mode by ESP; the result is
that the entire, authenticated inner packet is encrypted and a new outer IP header (and
extensions) is added.

Basic Combinations of Security Associations

The IPSec Architecture document lists four examples of combinations of SAs that must be
supported by compliant IPSec hosts (e.g., workstation, server) or security gateways (e.g.
firewall, router).

case:-1

All security is provided between end systems that implement IPSec. For any two end systems
to communicate via an SA, they must share the appropriate secret keys. Among the possible
combinations:
a) AH in transport mode
b) ESP in transport mode
c) ESP followed by AH in transport mode (an ESP SA inside an AH SA)
d) Any one of a, b, or c inside an AH or ESP in tunnel mode

Case:-2

Security is provided only between gateways (routers, firewalls, etc.) and no hosts implement
IPSec. This case illustrates simple virtual private network support. The security architecture
document specifies that only a single tunnel SA is needed for this case. The tunnel could
support AH, ESP, or ESP with the authentication option. Nested tunnels are not required
because the IPSec services apply to the entire inner packet.
Case-3:-

The third combination is similar to the second, but in addition provides security even to
nodes. This combination makes use of two tunnels first for gateway to gateway and second
for node to node. Either authentication or the encryption or both can be provided by using
gateway to gateway tunnel. An additional IPSec service is provided to the individual nodes
by using node to node tunnel.

Case:-4

This combination is suitable for serving remote users i.e., the end user sitting anywhere in the
world can use the internet to access the organizational workstations via the firewall. This
combination states that only one tunnel is needed for communication between a remote user
and an organizational firewall.

Internet Key Exchange:

The key management portion of IPSec involves the determination and distribution of secret
keys. The IPSec Architecture document mandates support for two types of key management:

• Manual: A system administrator manually configures each system with its own keys
and with the keys of other communicating systems. This is practical for small,
relatively static environments.

•
• Automated: An automated system enables the on-demand creation of keys for SAs
and facilitates the use of keys in a large distributed system with an evolving
configuration.

The default automated key management protocol for IPSec is referred to as ISAKMP/Oakley
and consists of the following elements:
 • Oakley Key Determination Protocol: Oakley is a key exchange protocol based on
the Diffie-Hellman algorithm but providing added security. Oakley is generic in that it
does not dictate specific formats.
• Internet Security Association and Key Management Protocol (ISAKMP):
ISAKMP provides a framework for Internet key management and provides the
specific protocol support, including formats, for negotiation of security attributes.

Oakley Key Determination Protocol

Oakley is a refinement of the Diffie-Hellman key exchange algorithm. The Diffie-Hellman
algorithm has two attractive features:
• Secret keys are created only when needed. There is no need to store secret keys for a
long period of time, exposing them to increased vulnerability.
• The exchange requires no pre-existing infrastructure other than an agreement on
the global parameters.
However, Diffie-Hellman has got some weaknesses:
• No identity information about the parties is provided.
• It is possible for a man-in-the-middle attack

• It is computationally intensive. As a result, it is vulnerable to a clogging attack, in

which an opponent requests a high number of keys.
Oakley is designed to retain the advantages of Diffie-Hellman while countering its
weaknesses.

Features of Oakley
The Oakley algorithm is characterized by five important features:
1. It employs a mechanism known as cookies to thwart clogging attacks.
2. It enables the two parties to negotiate a group; this, in essence, specifies the global
parameters of the Diffie-Hellman key exchange.
3. It uses nonces to ensure against replay attacks.
4. It enables the exchange of Diffie-Hellman public key values.
5. It authenticates the Diffie-Hellman exchange to thwart man-in-the-middle attacks.

In clogging attacks, an opponent forges the source address of a legitimate user and sends a
public Diffie-Hellman key to the victim. The victim then performs a modular exponentiation
to compute the secret key. Repeated messages of this type can clog the victim's system with
useless work. The cookie exchange requires that each side send a pseudorandom number, the
cookie, in the initial message, which the other side acknowledges. This acknowledgment
must be repeated in the first message of the Diffie-Hellman key exchange. The recommended
method for creating the cookie is to perform a fast hash (e.g., MD5) over the IP Source and
Destination addresses, the UDP Source and Destination ports, and a locally generated secret
value. Oakley supports the use of different groups for the Diffie-Hellman key exchange.
Each group includes the definition of the two global parameters and the identity of the
algorithm. Oakley employs nonces to ensure against replay attacks. Each nonce is a locally
generated pseudorandom number. Nonces appear in responses and are encrypted during
certain portions of the exchange to secure their use. Three different authentication methods
can be used with Oakley are digital signatures, public-key encryption and Symmetric-key
encryption.

ISAKMP Exchanges
ISAKMP provides a framework for message exchange, with the payload types serving as the
building blocks. The specification identifies five default exchange types that should be
supported.
1. Base Exchange: allows key exchange and authentication material to be transmitted
together. This minimizes the number of exchanges at the expense of not providing
identity protection.

The first two messages provide cookies and establish an SA with agreed protocol and
transforms; both sides use a nonce to ensure against replay attacks. The last two messages
exchange the key material and user IDs, with an authentication mechanism used to
authenticate keys, identities, and the nonces from the first two messages.

2. Identity Protection Exchange: expands the Base Exchange to protect the users' identities.

The first two messages establish the SA. The next two messages perform key exchange, with
nonces for replay protection. Once the session key has been computed, the two parties
exchange encrypted messages that contain authentication information, such as digital
signatures and optionally certificates validating the public keys.

3. Authentication Only Exchange: used to perform mutual authentication, without a key

exchange

The first two messages establish the SA. In addition, the responder uses the second
message to convey its ID and uses authentication to protect the message. The initiator
sends the third message to transmit its authenticated ID.

4. Aggressive Exchange: minimizes the number of exchanges at the expense of not

providing identity protection.

In the first message, the initiator proposes an SA with associated offered protocol and
transform options. The initiator also begins the key exchange and provides its ID. In the
 second message, the responder indicates its acceptance of the SA with a particular
protocol and transform, completes the key exchange, and authenticates the transmitted
information. In the third message, the initiator transmits an authentication result that
covers the previous information, encrypted using the shared secret session key.

5. Informational Exchange: used for one-way transmittal of information for SA

management.