IPSec is set of standards to ensure private, secure communications over IP networks through the use of cryptographic. It can provide authentication, integrity and confidentiality for ip traffic on a network. It can secure LAN, WAN and dial-up connections and the basics are described in RFC 2401 to 2411.
IPSec is enabled at the network level. (OSI layer 3) It can use an authentication header (AH) protocol and/or an encapsulated security protocol. (ESP) These protocols define the protocol, the payload header format and the services they provide. (RFC 2401)
Authentication Header (AH) is used for source authentication, data integrity and anti-replay. It adds additional bytes to the packet that contains a hash of the packet as it was negotiated by the isakmp protocol. AH uses the hashing algorithms HMAC-MD5 (RFC 2403) and HMAC-SHA (RFC 2404) for authentication and integrity. Anti-replay services are provided by increasing sequence numbers. AH is described in RFC 2402.
Encapsulated Security Payload (ESP) is used to provide confidentiality via DES-CBC (RFC 2405), 56-bit DES or 3DES but it can also provide data integrity via HMAC-MD5 (RFC 2403) and HMAC-SHA (RFC 2404). Anti-replay service are provided via increasing sequence numbers. ESP is described in RFC 2406.
Keep in mind that both AH and ESP do not provide the actual cryptographic algorithms but they leverage existing crypthograpic and authentication algorithms like DES and MD5.
IP-sec uses the following industry standards :
| Diffy-Hellman (DH) technique. This is a public key cryptography
algorithm that allows two communicating entities to agree on a shared key. |
| HMAC-MD5. Hashed Message Authentication Code Message Digest
function 95. This is a secret key mechanism used to provide data integrity
and authentication. It uses a hash function that produces a 128-bit key. |
| HMAC-SHA. Hashed Message Authentication Code Secure Hash Algorithm
(SHA1) . This is a secret key mechanism used to provide data integrity and
authentication. It produces a 160-bit key which is more secure, but slower,
than MHAC-MD5. |
| DES-CBC. Data Encryption Standards - Cipher Block Chaining. This is a secret key algorithm used for confidentiality. You can use DES that uses a one 56-bit key or 3DES (triple DES) that uses three 56 bit keys. |
IPSec uses by default Kerberos to identify and trust communicating computers but it can also use certificates or pre-shared keys.
IP Security process :
You cannot secure the following types of IP traffic with ipsec in transport mode:
| Broadcasts |
| Multicast |
| Resource Reservation Protocol (RSVP, port 46) |
| Internet Key Exchange (IKE, UDP port 500) |
| Kerberos (Port 88) |
In tunnel-mode, RSVP, IKE and Kerberos are secured with IPSec. See article Q253169.
This agent runs on the clients. It starts when the machine starts and performs the following tasks at intervals specified at the IPSec policy :
When facing connection problems that probably are IPSec related, stop the IPSec policy agent to check if it is causing the problems.
This service runs on each client and is started by the policy agent. It
generates a security association between two computers requiring secure
communication. This security association contains the settings of the secure
connection like keys and security properties.
First the service establishes a secure channel between the two computers by
authentication the computer identities and exchanging data about the connection.
After this a security association is created that is passed to the IPSec
drivers. ISAKMP/Oakley has the ability to dynamically change keys.
Oakley is a key determination protocol, which uses the Diffie-Hellman key
exchange algorithm. Oakley supports Perfect Forward Secrecy (PFS), which ensures
that if a single key is compromised, it permits access only to data protected by
a single key. It never reuses the key that protects communications to compute
additional keys and never uses the original key-generation material to compute
another key.
The IPSec driver (ipsec.sys) runs on all clients to monitor the IP datagrams. The datagrams are compared with the filter list to check if secure connections are requested or required. If a datagram matches a filter list entry, it is encrypted by using the security association and the shared key before it is put on the network. The IPSec driver also requests new security associations and updates and deletes them when required. The IPSec driver is started by the IPSec policy agent.
IP sec policies can be set in the group policy object\Computer
Configuration\Windows Settings\Security Settings\IP Security Policies. A shorter
way is to open the IP Security Policy Management snap-in to the MMC.
Each policy contains rules that tell when and how the policy is applied and
settings about key exchange. You can set filter lists, filter actions and
additional properties.
There a three predefined policies :
| Client (respond only). All communications will be in plain text but the client will accept encryption request of other machines. |
| Server (request security). The client will try to establish a secure connection. If this is not possible the data will be transferred in plain text |
| Secure server (require security). Unsecured connections are not allowed. Only outgoing broadcasts, multicasts, rsvp- and isakmp packets may be insecure. |
For a policy you can set the following rules and key exchange settings :
| Ip filter list. Each rule can have one ip filter list. Within a
filter list you can set to which incoming or outgoing packets the policy
should apply. You can set the incoming or outgoing ip address, (my ip
address, any ip address, a specific ip address, a specific subnet) the
protocol (any, egp, hmp, icmp, other, raw, rdp, rvd, tcp, udp, xns-idp) and
the source and destination port. You can select the mirror-option to
automatically secure connections in both ways. Keep in mind that when you
choose 'My ip address', only the first ip address on a multi-homed (RRAS)
client is used. If you specify a DNS name in a filter instead of an ip-adress you have to add an extra rule if the DNS server is not ip-sec enabled. Add a rule with the source address of My ip address and a destination address of the DNS server, enable the Mirrored option and set Do not allow secure communication on port 53. | ||||||||||||||||||
| Filter actions. Set which actions should be taken when the criteria
of the IP filter list are met. The default options are Permit, Request
security and Require security. These default options can be changed and new
options can be add. Each of the options contains the following items :
|
| Authentication methods. On this tab you can set how the computers
should authenticate each other. There are three options that can be
selected. It is possible to select more than one option :
|
The hashing algorithms used (SHA1/MD5) are defined at the filter action tab.
| Connection type. Set for which type of connection the policy should be used. (All network connections, Local area network or remote access) |
| Tunnel settings. If you select the option 'This rule does not specify an IPSec tunnel', ipsec runs in transport mode (default). To run ipsec in tunnel mode, enter the ip address of the tunnel end point. |
These are the phases in the IPSec process :
| An IP packet matches an IP filter that is part of an IP Security policy. |
| The IP Security policy is applied to secure the packet. The IPSec driver uses the ISAKMP service to negotiate a security method and security key between the two machines. |
| The security key negotiated by the ISAKMP service is passed to the IPSec driver. |
| The security method and key become the IPSec Security Association (SA). The IPSec driver stores this SA in its database. This happens on both machines. |
| The security method is applied to the IP packet by the IP sec driver on both machines. |
Transport mode is the default IPSec mode that provides end-to-end security
between hosts. In this mode the endpoint for communication is the same as the
cryptographic endpoint. When operating in transport mode, ipsec leaves most of
the packet unchanged and adds some new headers. Transport mode uses less
capacity of the clients and is in general used when communicating on an internal
network. You cannot use tunnel mode with NAT.
Tunnel mode regenerates the original packet and puts the original packet in the
payload of the new packet. This costs more processing power of the clients as
packets are re-packed when send an unpacked when received. Tunnel mode is mostly
used when transferring data on an external network. The ip adress of the 'end of
the tunnel' must be a fixed address.
IPSec does not work through NAT and proxies when used in transport mode as IPSec does not allow fields in the packet to be changed.
Routers and switches will normally forward encrypted or authenticated ip packets but on a firewall or filtering router you must enable ip forwarding for the following ip protocols and udp ports :
| IP port 50 for inbound/outbound ESP traffic. |
| IP port 51 for inbound/outbound to pass AH traffic. |
| UDP port 500 for isakmp traffic. |
As some firewalls analyze packet payload data, ESP may not work.
Within Windows 2000 you can use two VPN solutions, PPTP and L2TP. Both protocols do not encrypt data by themselves but PPTP can use MPPE to encrypt data. MPPE is based on RSA/RC4. L2TP can use IPSec to encrypt data within a tunnel to create a VPN.
If you use L2TP and IPSec, the following ports must be open on the firewall :
| TCP 50 |
| TCP 51 |
| UDP 500 |
| UDP 1701 (L2TP) |
| TCP 88 (required for domain controllers to communicate) |
Only the predefined policies Secure initiator and Lockdown do secure SNMP traffic. You can enable secure SNMP traffic in a policy by adding two filters with the following settings :
| In the addressing tab set the source address to the SNMP management system and the destinition address to My ip address and enable the Mirrored option. |
| In the protocol page set the protocol type to TCP or UDP and the port to 161. |
The second filter is the same, but the port is set to 162 to secure the traps.
There are various ways to monitor IPSec :
Policy agent and IPSec driver events are logged in the system log, Oakley events in the application log. ISAKMP events are stored in the security log if logon auditing is enabled.
Network monitor 2.0 contains parsers for isakmp, ah, esp and IPSec. If the packets are encrypted, the contents is not visible. If it is only authenticated you can view the contents. Packets with ESP are shown under the ESP protocol but packets with an AH header are not reported under the AH protocol.
The ipsecmon utility can be used to monitor IPSec information like SA's, negotiation errors and ip statistics. It monitors the following IPSec statistics on local or remote machines :
| Active associations. The number of active SA's. |
| Confidential bytes send/received. Bytes send and received with the ESP protocol. |
| Authenticated bytes send/received. Bytes send and received with the AH protocol. |
| Bad SPI packets. The Security Parameters Index is used the match inbound packets with SA's. The packet is concerned as bad if the SA has expired. |
| Packets not decrypted. The number of packets that could not be decrypted by the Ipsec driver. |
| Packets not authenticated. The number of packets that could not be authenticated by the Ipsec driver. |
| Key additions. The number of keys the isakmp service has send to the ipsec driver. |
IPSecmon also displays isakmp statistics :
| Oakley main modes. The number of successful isakmp's created during phase 1 negotiations. |
| Oakley quick modes. The number of succesfull isakmp's created during phase 2 negotiations. |
| Soft associations. The number of phase 2 negotiations that resulted in agreements to send using clear text. |
| Authentication failures. The total number of authentication
failures. |
| Ipsec usage will cost processor cycles. AH uses less than ESP as no encryption is done. |
| Encryption is not allowed in every country, consider this when using an ESP protocol. |
| Compression on encrypted data is less effective. Activate compression before encryption as the encryption process eliminates semantic patterns that are used for compression. |
| Ipseccmd.exe utility. Can be used to view and modify the policies and properties of IPSec. (WH1205S020701) |
| IPSec monitoring improvements. Replaces IPSecmon (WH0371N020701) |
Last update : 17 August 2001
