Securing Data in Transit with IPSec

loyalsockvillemobNetworking and Communications

Oct 27, 2013 (5 years and 2 months ago)


Securing Data in Transit with IPSec

Network security has many facets, and much emphasis is placed (rightly) on keeping intruders and
attackers out of the network via firewalls. However, in today’s business environment, there are also
many instances in

which sensitive data needs to be protected within the local network from users
who have legitimate access to the network

but do not need to have access to the data in question.

The answer in that case is encryption (and encrypting data also provides an

extra layer of security
against intruders who do manage to get into the network). With Microsoft operating systems prior
to Windows 2000, encrypting data required third party software. Now encryption capabilities are
built into the OS. These include the E
ncrypting File System (EFS) and Internet Protocol Security

The type of encryption you need to use depends on the state of the data. File encryption can protect
data residing on disk, but does not protect that data when it’s in transit over the net
work. If you
don’t believe this, send an EFS
encrypted file across the network and capture the packets in transit.
You’ll see that the data is readable. IPSec is needed to encrypt to protect data from network

A Brief History of IPSec

IPSec is an
industry standard set of protocols and services based on cryptography, used to encrypt
data so that it cannot be read or tampered with during its journey across an IP network. There are a
number of RFCs that provide specifications for IPSec and its protoco
ls, as defined by the Internet
Engineering Task Force (IETF). Good starting points are RFC 1825 and 2401, which deal with the
security architecture for IP (

). IPSec can be used with the current IPv4, and is built into
the next generation of IP, IPv6.

What IPSec Does

IPSec is designed to provide authentication (verification of

the identity of the sender), integrity
(assurance that the data was not changed in transit) and confidentiality (encryption of the data so
that it can’t be read by anyone who doesn’t have the correct key).

Because it operates at the network layer of the O
SI model (Layer 3), IPSec has an advantage over
SSL and other methods that operate at higher layers. Applications must be written to be aware of
and use SSL, while applications can be used with IPSec without being written to be aware of it. Thus

occurs transparently to the upper layers.

IPSec protects only IP
based traffic; it is of no use to other network layer protocols such as IPX.
There are also some types of IP traffic (such as Kerberos) that are not protected by Microsoft’s
implementation o
f IPSec by default. Microsoft calls these exemptions.

IPSec Protocols, Modes and Security Associations

IPSec is not a single protocol; rather, it is made up of two protocols, which can be used separately
or together:

AH (Authentication Header). As the name

implies, AH is used to authenticate the
identity of the sender, and to provide integrity of the data to ensure that it hasn’t
been modified. It does not encrypt data, and provides no confidentiality. AH signs the
entire packet.

ESP (Encapsulating Securit
y Payload). ESP can provide confidentiality by encrypting
the data itself, along with authentication and integrity. However, ESP generally
doesn’t sign the entire packet, only the data.

To protect the IP header as well as the data itself, AH and ESP can b
e used together.

There are two modes of operation for both AH and ESP:

Tunnel mode, which is used to create a virtual private network. Tunnel mode provides
gateway to gateway (or server to server) protection.

Transport mode, which is used to encrypt data
inside a tunnel that is created by L2TP
(the layer 2 tunneling protocol). Transport mode provides end
end security, all the
way from the sending computer to the final destination.

The two computers that are communicating via IPSec establish a security
association (SA). This
represents the “agreement” between the two about the way the data will be exchanged and
protected. Thus both of these computers must support IPSec. IPSec support is built into Windows
2000 (Server and Pro) and XP Pro computers and wi
ll be included in Windows Server 2003.

How IPSec Works in Windows

Microsoft and Cisco worked together to develop the implementation of IPSec that is included in
Windows 2000 and later operating systems. Cisco’s ISAKMP/IKE is used along with Microsoft’s IPS

Internet Key Exchange (IKE) negotiates the security associations during two phases: ISAKMP phase
(phase 1) and IPSec phase (phase 2). See RFC 2409 for more information about IKE and its
components, ISAKMP and Oakley. Another IPSec component, th
e Policy Agent, distributes IPSec
polices that are created by the administrator. The IPSec policies can be stored in Active Directory or
in the local configuration policies. The Policy Agent is called IPSec Services in Windows XP.

To use IPSec in Windows 2
000/XP, you must define an IPSec policy that specifies the authentication
method and IP filters to be used. There are three authentication methods to choose from: Kerberos
(the default), certificates, or preshared keys. Preshared keys are not recommended f
or sensitive
environments, because the key is stored as plain text in the database where the IPSec policies are
stored, and thus poses a security risk.

How to Configure a Windows 2000 Pro Computer to use IPSec

Setting up your Windows 2000 computer to use I
PSec is relatively simple. Remember that both the
sending and receiving computers must support IPSec. Also, you must be an administrator to set
IPSec policies. The following steps will configure your system to take advantage of IPSec


Click the

menu, then select
Settings | Network and Dialup Connections


Right click the connection you want to configure for IPSec communications.


. On the

tab, under
Components used by this
, choose
Internet Protocol (TCP/I
. Click the

button (see
Figure 1).

Figure 1


On the TCP/IP properties sheet, click the

button (See Figure 2).

Figure 2


Choose the

tab and select the IP Security optional setting (see Figure 3).

Figure 3


Click the



Check the option button that says
Use this IP Security policy

to enable IPSec
communications (see Figure 4). NOTE: If the options are greyed out and cann
ot be changed, this
usually means the computer belongs to an Active Directory domain and gets its IPSec policies from
Active Directory.

Figure 4


are three predefined IPSec policies you can choose from: Client (respond only), Server
(request security) and Secure Server (require security). The Client policy is used if you do not want
IPSec to be used unless the server with which you are communicatin
g requests it. The Server policy
causes the computer to try to negotiate an IPSec connection, but if the server on the other end
doesn’t support IPSec or isn’t configured to use it, your computer will go ahead and communicate
via unsecured communications.
The Secure Server policy is used if you want the computer to send
and accept IPSec secured communications only. If the computer on the other end can’t use IPSec,
your computer will reject all traffic from it. This is the most highly secure setting.


You may need customized policies to fit your organization’s needs, rather than using the predefined
policies. IPSec policies can be created, changed and managed via the IP Security Policy MMC.
Create a custom MMC and add the IPSec snap
in. You can
also access local IPSec policies using the
Local Security Settings tool in the Administrative Tools menu.

Security Settings

in the left pane, expand the
Public Key Policies

node and click
Security Policies
. You’ll see the three default predefined
policies in the right details pane. Any
custom policies you create will also be listed here (see Figure 5).

Figure 5

Creating an IPSec Policy

To defin
e a new policy, right click
IP Security Policies on Local Machine

in the left console pane.
Create IP Security Policy

(see Figure 6).

Figure 6

IP Security Policy Wizard

will start. The wizard will ask you to provide a name and
description for the new policy. Next, you’ll be asked to decide how the policy should respond to
requests for secure communications. On the next page, you’ll be asked t
o set an initial
authentication method. Kerberos is the default, or you can select certificates (you’ll have to specify
a CA) or preshared key (this is a secret string of characters that must be shared by the two
computers that are communicating via IPSec)
. On the last page of the wizard, click

to create
the policy. You can edit its properties later by double clicking it in the right console pane or right
clicking it and choosing

You can add and edit rules for the policy by selecting the

tab (see Figure 7).

Figure 7


will invoke another wizard, the
Security Rule Wizard
. The steps of this wizard
include the following:


Define whether this rule will cause an IPSec tunnel to be created. IPSec tunneling is
used to create a virtual private network link. If you specify that a tunnel will be
created, you must provide the IP address of the computer that will serve as the
nt of the tunnel.


Select the type of network connection to which the rule is to be applied. You can
choose from the following: all network connections, local area network (LAN)
connections, or remote access connections. (The default setting is all connect


Select an initial authentication method for the rule (Windows 2000 Kerberos,
certificate, or a preshared key).


Choose the type of IP traffic to which the rule will apply. Default choices are: All ICMP
Traffic and All IP Traffic.

You can add addition
al filters by selecting the

button on the IP Filter List screen. This will
invoke the Filter Wizard. Select a filter action for the rule. Default actions you can choose from

Permit (allows unsecured packets to pass through)

Request Security

Optional (negotiates security; will accept unsecured
communications, but always responds using IPSec; will allow unsecured
communications if the other computer is not IPSec

Require Security (will not allow unsecured communications with non IPSec

In most cases, the predefined filter actions will work, but you can also create custom filter actions.
Filter Action Wizard

is used for this purpose. You can choose which IPSec protocol(s) will be
used with the action

ESP, AH, or both.

You can also specify which integrity and encryption
algorithms are to be used and how often a new key is to be generated.

Assigning Policies

Your new policy cannot be used for establishing IPSec connections until it has been assigned. No
policies are assi
gned by default, but it’s easy to assign a policy. Just right click it in the right details
pane of the MMC and select

from the context menu. If you don’t want it to be used any
longer, follow the same procedure and select