IPSEC(4) Device Drivers Manual IPSEC(4)

ipsec — IP security protocol

#include <sys/types.h>
#include <netinet/in.h>
#include <netinet6/ipsec.h>

ipsec is a security protocol in Internet Protocol layer. ipsec is defined for both IPv4 and IPv6 (inet(4) and inet6(4)). ipsec consists of two sub-protocols, namely ESP (encapsulated security payload) and AH (authentication header). ESP protects IP payload from wire-tapping by encrypting it by secret key cryptography algorithms. AH guarantees integrity of IP packet and protects it from intermediate alteration or impersonation, by attaching cryptographic checksum computed by one-way hash functions. ipsec has two operation modes: transport mode and tunnel mode. Transport mode is for protecting peer-to-peer communication between end nodes. Tunnel mode includes IP-in-IP encapsulation operation and is designed for security gateways, like VPN configurations.

ipsec is controlled by key management engine and policy engine, in the operating system kernel.

Key management engine can be accessed from the userland by using PF_KEY sockets. The PF_KEY socket API is defined in RFC2367.

Policy engine can be controlled by extended part of PF_KEY API, setsockopt(2) operations, and sysctl(3) interface. The kernel implements extended version of PF_KEY interface, and allows you to define IPsec policy like per-packet filters. setsockopt(2) interface is used to define per-socket behavior, and sysctl(3) interface is used to define host-wide default behavior.

The kernel code does not implement dynamic encryption key exchange protocol like IKE (Internet Key Exchange). That should be implemented as userland programs (usually as daemons), by using the above described APIs.

The kernel implements experimental policy management code. You can manage the IPsec policy in two ways. One is to configure per-socket policy using setsockopt(2). The other is to configure kernel packet filter-based policy using PF_KEY interface, via setkey(8). In both cases, IPsec policy must be specified with syntax described in ipsec_set_policy(3).

With setsockopt(2), you can define IPsec policy in per-socket basis. You can enforce particular IPsec policy onto packets that go through particular socket.

With setkey(8) you can define IPsec policy against packets, using sort of packet filtering rule. Refer to setkey(8) on how to use it.

In the latter case, “default” policy is allowed for use with setkey(8). By configuring policy to default, you can refer system-wide sysctl(8) variable for default settings. The following variables are available. 1 means “use”, and 2 means “require” in the syntax.

Type Changeable
net.inet.ipsec.esp_trans_deflev integer yes
net.inet.ipsec.esp_net_deflev integer yes
net.inet.ipsec.ah_trans_deflev integer yes
net.inet.ipsec.ah_net_deflev integer yes
net.inet6.ipsec6.esp_trans_deflev integer yes
net.inet6.ipsec6.esp_net_deflev integer yes
net.inet6.ipsec6.ah_trans_deflev integer yes
net.inet6.ipsec6.ah_net_deflev integer yes

If kernel finds no matching policy system wide default value is applied. System wide default is specified by the following sysctl(8) variables. 0 means “discard” which asks the kernel to drop the packet. 1 means “none”.

Type Changeable
net.inet.ipsec.def_policy integer yes
net.inet6.ipsec6.def_policy integer yes

The following variables are accessible via sysctl(8), for tweaking kernel IPsec behavior:

Type Changeable
net.inet.ipsec.ah_cleartos integer yes
net.inet.ipsec.ah_offsetmask integer yes
net.inet.ipsec.dfbit integer yes
net.inet.ipsec.ecn integer yes
net.inet.ipsec.debug integer yes
net.inet6.ipsec6.ecn integer yes
net.inet6.ipsec6.debug integer yes

The variables are interpreted as follows:

If set to non-zero, the kernel clears type-of-service field in the IPv4 header during AH authentication data computation. The variable is for tweaking AH behavior to interoperate with devices that implement RFC1826 AH. It should be set to non-zero (clear the type-of-service field) for RFC2402 conformance.
During AH authentication data computation, the kernel will include 16bit fragment offset field (including flag bits) in IPv4 header, after computing logical AND with the variable. The variable is for tweaking AH behavior to interoperate with devices that implement RFC1826 AH. It should be set to zero (clear the fragment offset field during computation) for RFC2402 conformance.
The variable configures the kernel behavior on IPv4 IPsec tunnel encapsulation. If set to 0, DF bit on the outer IPv4 header will be cleared. 1 means that the outer DF bit is set regardless from the inner DF bit. 2 means that the DF bit is copied from the inner header to the outer. The variable is supplied to conform to RFC2401 chapter 6.1.
If set to non-zero, IPv4 IPsec tunnel encapsulation/decapsulation behavior will be friendly to ECN (explicit congestion notification), as documented in draft-ietf-ipsec-ecn-02.txt. gif(4) talks more about the behavior.
If set to non-zero, debug messages will be generated via syslog(3).

Variables under net.inet6.ipsec6 tree has similar meaning as the net.inet.ipsec counterpart.

The ipsec protocol works like plug-in to inet(4) and inet6(4) protocols. Therefore, ipsec supports most of the protocols defined upon those IP-layer protocols. Some of the protocols, like icmp(4) or icmp6(4), may behave differently with ipsec. This is because ipsec can prevent icmp(4) or icmp6(4) routines from looking into IP payload.

ioctl(2), socket(2), ipsec_set_policy(3), icmp6(4), intro(4), ip6(4), setkey(8), sysctl(8)

Daniel L. McDonald, Craig Metz, and Bao G. Phan, PF_KEY Key Management API, Version 2, RFC, 2367.

D. L. McDonald, A Simple IP Security API Extension to BSD Sockets, internet draft, draft-mcdonald-simple-ipsec-api-03.txt, work in progress material.

The implementation described herein appeared in WIDE/KAME IPv6/IPsec stack.

The IPsec support is subject to change as the IPsec protocols develop.

There is no single standard for policy engine API, so the policy engine API described herein is just for KAME implementation.

AH and tunnel mode encapsulation may not work as you might expect. If you configure inbound “require” policy against AH tunnel or any IPsec encapsulating policy with AH (like “esp/tunnel/A-B/use ah/transport/A-B/require”), tunnelled packets will be rejected. This is because we enforce policy check on inner packet on reception, and AH authenticates encapsulating (outer) packet, not the encapsulated (inner) packet (so for the receiving kernel there's no sign of authenticity). The issue will be solved when we revamp our policy engine to keep all the packet decapsulation history.

Under certain condition, truncated result may be raised from the kernel against SADB_DUMP and SADB_SPDDUMP operation on PF_KEY socket. This occurs if there are too many database entries in the kernel and socket buffer for the PF_KEY socket is insufficient. If you manipulate many IPsec key/policy database entries, increase the size of socket buffer.

January 29, 1999 macOS 15.0