Implement IPSec on Windows Server 2003

With IPSec, you can encrypt network traffic and protect it from malicious users. With the first of our two part series, we tell you how to implement it on Windows Server 2003.

 By Drew Bird
Page 1 of 2
Print Article

In this, the latest installment in our series of Windows Server 2003 administration tutorials, we'll take a look at the process of implementing IPSec on a Windows Server 2003 system. We'll also look at some of the procedures you need to be aware of when performing this implementation on a live network. When we're all done, we'll look back at the process and weigh up the pros and cons, seeking to answer one simple question: Is it worth considering an IPSec implementation for your network?

IPSec 101

Before wading into an explanation of how to implement IPSec, we should first take a moment to introduce you to, or refresh your knowledge of, this free and very effective method of securing network transmissions.

On a theoretical level, IPSec is a framework designed to provide security for IP based network traffic. On a practical level, it is a network layer protocol that encrypts data so that it cannot be 'sniffed' from the network and then subsequently read or altered. IPSec achieves this functionality through two protocols called IPSec Authentication Header (AH) and IPSec Encapsulating Security Payload (ESP).

IPSec AH does not actually encrypt data, but it does provide authentication and guaranteed data integrity. In other words, with IPSec AH, someone can read the data in a transmission, but they cannot alter it. Nor can someone fake the source of the data. IPSec ESP, in contrast, focuses on securing the data in the transmission, though it does also provide some authentication, and a measure of data integrity checking.

The good news is that in a Windows Server 2003 IPSec implementation, as with most others, you do not need to choose between IPSec AH and IPSec ESP. You can use both protocols to get the full benefit of IPSec – authentication, guaranteed integrity of data, and encrypted data transfer.

IPSec and Windows Server 2003

Now that we have recapped what IPSec is and does, we can get on with looking at how to implement it.

IPSec functionality is provided on a Windows Server 2003 system through the IPSec Services service. So, the first step in configuring IPSec is to make sure that this is running on your server by looking in the Services MMC. On a domain controller, the Services MMC can be accessed through the Administrative Tools menu. The IPSec service is configured to start automatically by default, so unless it has been stopped or disabled, your check should be nothing more than cursory.

Figure 1.
(Click for a larger image)

The next part of implementing IPSec is choosing and assigning an IPSec policy. IPSec policies, once assigned, define what actions should be performed on incoming network traffic that does or does not meet a specified criteria.

IPSec policies, and their components, are configured through the IP Security Policy Management MMC snap-in. There is no shortcut to this MMC on the Administrative Tools menu, so you'll need to open a blank MMC and then add the snap-in to it. Once you have done this, you'll be able to start working in the IP Security Management MMC snap-in as shown in Figure 1.

Structure of Policies

The properties of an existing rule can be viewed by double-clicking the rule from within the IPSec Security Policies snap-in. The properties page for one of the default policies, which are discussed in a moment, is shown in Figure 2.

Figure 2.
(Click for a larger image)

As discussed, policies define what actions are performed on network traffic when the server is using IPSec. Policies are comprised of rules that define what traffic should be covered by the policy, what kind of authentication mechanism should be used (more about the options for authentication later in this series), and what happens to traffic when it does or does not meet the criteria specified in the policy. This last process is known as the filter action. You can also define whether or not this rule applies to all network connections, just those originating from the LAN, or just from remote connections.

As you can see from Figure 2, there are three rules in this policy. The first defines that security should be requested for all IP traffic, and that the security should use Kerberos (define) for authentication and encryption. The second rule defines that all ICMP traffic (such as that associated with ping and tracert) (define) is permitted with no request for security. The third rule (<default>) defines what happens to traffic that does not conform to either of the first two rules.

This article was originally published on Sep 5, 2007
Get the Latest Scoop with Networking Update Newsletter