Build an IPSEC VPN Without Losing Your Mind - Page 2

Why This Is Difficult

First of all, if you are using iptables to filter or NAT packets, the above test may not have been successful. The quick and easy way around NAT is to modify your MASQUERADE rule, on both ends, to exclude the internal destination on the opposite. Traffic destined for the subnet behind the other NAT cannot be NAT'd, as NAT attempts to modify layer 4 headers, which are encrypted. Everything in the IP data section, including TCP and UDP packets in their entirety, are encrypted and cannot be changed. Furthermore, if you are attempting to start an IPSEC connection from a box that lives behind (a possibly secret) NAT device, it will fail. The NAT device can be configured to allow IPSEC passthrough for you, though.

Filtering traffic with IPSEC in the mix gets messy. When the packet arrives, everything but the first part of the IP header is encrypted. Filter rules designed to detect TCP/UDP ports, or even specific upper layer protocols, will not be able to see that data until the packet is decrypted. You can mark packets, or use policy match rules; both are convoluted, but have been discussed at length on various mailing lists including the Openswan list.

One other reason for IPSEC failing to start could be blocked ports. First get it working without iptables in the way, to make sure IPSEC is configured and working, then re-enable the firewall. Port 500 needs to be reachable by both IPSEC endpoints, for IKE, the key exchange protocol.

We can't stress enough that configuring IPSEC is actually quite painless--until you add NAT or firewalls into the picture. You absolutely must test the IPSEC connection outside of any firewall or NAT presence to ensure it is working, before adding back the challenging piece (if you absolutely must). Your sanity will be preserved, and the installation process time dramatically reduced.