Build an IPSEC VPN Without Losing Your Mind - Page 2
Why This Is Difficult
First of all, if you are using iptables to filter or NAT packets, the above test may not have been successful. The quick and easy way around NAT is to modify your MASQUERADE rule, on both ends, to exclude the internal destination on the opposite. Traffic destined for the subnet behind the other NAT cannot be NAT'd, as NAT attempts to modify layer 4 headers, which are encrypted. Everything in the IP data section, including TCP and UDP packets in their entirety, are encrypted and cannot be changed. Furthermore, if you are attempting to start an IPSEC connection from a box that lives behind (a possibly secret) NAT device, it will fail. The NAT device can be configured to allow IPSEC passthrough for you, though.
Filtering traffic with IPSEC in the mix gets messy. When the packet arrives, everything but the first part of the IP header is encrypted. Filter rules designed to detect TCP/UDP ports, or even specific upper layer protocols, will not be able to see that data until the packet is decrypted. You can mark packets, or use policy match rules; both are convoluted, but have been discussed at length on various mailing lists including the Openswan list.
One other reason for IPSEC failing to start could be blocked ports. First get it working without iptables in the way, to make sure IPSEC is configured and working, then re-enable the firewall. Port 500 needs to be reachable by both IPSEC endpoints, for IKE, the key exchange protocol.
When he's not writing for Enterprise Networking Planet or riding his motorcycle, Charlie Schluting works as the VP of Strategic Alliances at the US Division of LINBIT, the creators of DRBD. He also operates OmniTraining.net, and recently finished Network Ninja, a must-read for every network engineer.