Implement IPSec on Windows Server 2003

Enterprise Networking Planet content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

In this, the latest installment in our series of Windows Server 2003 administration tutorials, we’ll take a look at the process of implementing IPSec on a Windows Server 2003 system. We’ll also look at some of the procedures you need to be aware of when performing this implementation on a live network. When we’re all done, we’ll look back at the process and weigh up the pros and cons, seeking to answer one simple question: Is it worth considering an IPSec implementation for your network?

IPSec 101

Before wading into an explanation of how to implement IPSec, we should first take a moment to introduce you to, or refresh your knowledge of, this free and very effective method of securing network transmissions.

On a theoretical level, IPSec is a framework designed to provide security for IP based network traffic. On a practical level, it is a network layer protocol that encrypts data so that it cannot be ‘sniffed’ from the network and then subsequently read or altered. IPSec achieves this functionality through two protocols called IPSec Authentication Header (AH) and IPSec Encapsulating Security Payload (ESP).

IPSec AH does not actually encrypt data, but it does provide authentication and guaranteed data integrity. In other words, with IPSec AH, someone can read the data in a transmission, but they cannot alter it. Nor can someone fake the source of the data. IPSec ESP, in contrast, focuses on securing the data in the transmission, though it does also provide some authentication, and a measure of data integrity checking.

The good news is that in a Windows Server 2003 IPSec implementation, as with most others, you do not need to choose between IPSec AH and IPSec ESP. You can use both protocols to get the full benefit of IPSec – authentication, guaranteed integrity of data, and encrypted data transfer.

IPSec and Windows Server 2003

Now that we have recapped what IPSec is and does, we can get on with looking at how to implement it.

IPSec functionality is provided on a Windows Server 2003 system through the IPSec Services service. So, the first step in configuring IPSec is to make sure that this is running on your server by looking in the Services MMC. On a domain controller, the Services MMC can be accessed through the Administrative Tools menu. The IPSec service is configured to start automatically by default, so unless it has been stopped or disabled, your check should be nothing more than cursory.

Figure 1.
(Click for a larger image)

The next part of implementing IPSec is choosing and assigning an IPSec policy. IPSec policies, once assigned, define what actions should be performed on incoming network traffic that does or does not meet a specified criteria.

IPSec policies, and their components, are configured through the IP Security Policy Management MMC snap-in. There is no shortcut to this MMC on the Administrative Tools menu, so you’ll need to open a blank MMC and then add the snap-in to it. Once you have done this, you’ll be able to start working in the IP Security Management MMC snap-in as shown in Figure 1.

Structure of Policies

The properties of an existing rule can be viewed by double-clicking the rule from within the IPSec Security Policies snap-in. The properties page for one of the default policies, which are discussed in a moment, is shown in Figure 2.

Figure 2.
(Click for a larger image)

As discussed, policies define what actions are performed on network traffic when the server is using IPSec. Policies are comprised of rules that define what traffic should be covered by the policy, what kind of authentication mechanism should be used (more about the options for authentication later in this series), and what happens to traffic when it does or does not meet the criteria specified in the policy. This last process is known as the filter action. You can also define whether or not this rule applies to all network connections, just those originating from the LAN, or just from remote connections.

As you can see from Figure 2, there are three rules in this policy. The first defines that security should be requested for all IP traffic, and that the security should use Kerberos (define) for authentication and encryption. The second rule defines that all ICMP traffic (such as that associated with ping and tracert) (define) is permitted with no request for security. The third rule (<default>) defines what happens to traffic that does not conform to either of the first two rules.

Default IPSec Policies

Because the configuration of the IPSec rules can be somewhat complex, and because many environments will have the same level of need for their IPSec implementations, Microsoft includes three IPSec policies with Windows Server 2003. These policies are worth looking at before going to the effort of creating your own brand new policies. The three policies are Client (Respond Only), Server (Request Security) and Secure Server (Require Security). Each contains certain rules that define exactly how that policy affects IPSec traffic, but they can be summarized quite simply as follows:

When Client (Respond Only) is used, a Windows Server 2003 system will allow and provide for an IPSec connection if a client system requests it. All other connections will be allowed without IPSec. When Server (Request Security) is used, the Windows Server 2003 system will ask all incoming client connections to use IPSec. If the client is able to use IPSec, then it does precisely that.

If the client is not able to use IPSec, for example if the connection originates from an operating system that does not have an IPSec client, then the connection is till permitted to continue, though obviously without the security provided by IPSec. As the name suggests, when the Secure Server (Require Security) policy is used, all incoming requests to the server must be able to use IPSec encryption. If they cannot, then that connection is refused. The only exception in the rules that make up the Secure Server (Require Security) policy is that ICMP traffic is allowed to connect without encryption.

For the purposes of our discussion, we’ll be using the Client (Respond Only) policy, but in a real-world scenario, you might find that you need to consider creating your own IPSec policies from scratch. As discussed, this is quite an involved process, and beyond the scope of this article. However, there are numerous excellent sources of information on this process, including Microsoft knowledgebase articles.

In Part Two of this article we continue our look at the practical side of IPSec implementation, including the assignment of an IPSec policy, and the process of configuring client systems to use encrypted communications. We’ll also show you how you can determine whether or not your network traffic is being encrypted with IPSec. Until then!

Get the Free Newsletter!

Subscribe to Daily Tech Insider for top news, trends, and analysis.

Latest Articles

Follow Us On Social Media

Explore More