At our workplace we use all of the three major OSes: Windows, Linux and Mac OS X. Finding a VPN solution that works on all those platforms was quite hard. Our first solution was OpenVPN, but it required just too much configuring, it’s own client and didn’t even always work. After that we bought a Netgear FVS336v2 -box. SSL-VPN worked fine with Windows and somewhat ok with 32-bit linux, but not at all with Mac OS X nor 64-bit Linux. And netgear’s own IPsec.. Well, simply put: there was always some strange problem.

So, what we wanted was a simple VPN-solution, preferably usable by built-in-client from every OS, and which actually would work.

Why is setting up a VPN server so hard?

Finally after many hours or even days of googling and testing things, I ended up testing Openswan. Getting Openswan and xl2tpd to work with Windows built-in IPsec-client was quite a handfull, but it finally started to work perfectly. Now after few weeks of using it, I am very pleased.

Next thing was to get Openswan work with Mac OS X built-in IPsec-client. It didn’t work out-of-the-box with same server settings as Windows did. I found many articles telling how to fix some Mac OS X bug and get it to work with Openswan. Finally after testing and testing again with different configurations, it started working (that isn’t actually all, more specifics later). But, like always it seems, there is a catch. It didn’t work with Windows any more! We tried configuring the server with more and more obscure ways, tried dancing and checked the angle of the moon, nothing worked. The difference in Openswan configuration files was minimal between Windows and Mac OS X. Basically for Mac OS X you had to force NAT-mode on and for windows you had to force it off.

Lesson learned and believe me: You cannot get Openswan IPSec server to work with Windows and Mac OS X at the same time. At least not with current versions. Solution: Two virtual machines with those tiny differences in Openswan configuration! By the way: this guide is about how to setup Openswan to work with Window or with Mac OS X, the basic stuff you need to figure out yourself.

Original Openswan guide I used: Setting Up an IPSec L2TP VPN server on Ubuntu for Windows clients

Prerequisites

First of all, we use currently Ubuntu 10.04 as our primary linux distribution. So for other distributions everything might not work as I explain.

You need to install atleast following (plus maybe some other things):

• Openswan
• xl2tpd
• ppp

And in order to get Openswan work with Mac OS X, it has to be atleast version 2.6.32. Version 2.6.23 which is shipped with Ubuntu 10.04 won’t work. It will complain about some XFRM_.. data size etc. So compile and install Openswan manually, if this is the case. Also this guide assumes you are using 192.168.0.0/24 as your LAN subnet.

ppp

Copy /etc/ppp/options to /etc/ppp/options.x2ltpd. You need to add/change following lines:

# this is a must. well you can use noauth-option, but...
auth

# this is better explained in the original options-file
name l2tpd

# If you need/want to use your internal DNS-server(s)
# otherwise comment out
ms-dns 192.168.0.x
ms-dns 192.168.0.y

Also add users in /etc/ppp/chap-secrets if you want to use chap for authentication (simplest way).

xl2tpd

This is quite straightforward. Simple configuration file /etc/xl2tpd/xl2tpd.conf:

[global]
ipsec saref = yes
# change address below to your public server ip
listen-addr = <public_server_ip>

[lns default]
# change ip-range below, it will be used for xl2tpd dhcp
ip range = 192.168.0.<from>-192.168.0.<to>
# server's LAN address
local ip = 192.168.0.<lan_ip>
refuse chap = yes
refuse pap = yes
require authentication = yes
ppp debug = yes
pppoptfile = /etc/ppp/options.xl2tpd
length bit = yes

Openswan

Set pre-shared key into /etc/ipsec.secrets (if you are not using certificates):

%any: PSK "yourpresharedkey"

Openswan and Windows

Openswan-config (/etc/ipsec.conf) is somewhat following:

# basic configuration
config setup
 nat_traversal=no
 # exclude networks used on server side by adding %v4:!a.b.c.0/24
 virtual_private=%v4:10.0.0.0/8,%v4:192.168.0.0/16,%v4:172.16.0.0/12,%v4:!192.168.0.0/24
 oe=off
 protostack=auto

conn L2TP-PSK
 authby=secret
 pfs=no
 auto=add
 keyingtries=3
 rekey=no
 ikelifetime=8h
 keylife=1h
 type=transport

 left=<public_server_ip>
 leftprotoport=17/1701
 right=%any
 rightprotoport=17/1701

Openswan and Mac OS X

Unresolved: Mac OS X sleepmode disconnect-reconnect error

Openswan (/etc/ipsec.conf) is somewhat following:

# basic configuration
config setup
 nat_traversal=yes
 # exclude networks used on server side by adding %v4:!a.b.c.0/24
 virtual_private=%v4:10.0.0.0/8,%v4:192.168.0.0/16,%v4:172.16.0.0/12,%v4:!192.168.0.0/24
 oe=off
 protostack=auto

conn L2TP-PSK
 authby=secret
 pfs=no
 auto=add
 keyingtries=3
 rekey=no
 ikelifetime=8h
 keylife=1h
 type=transport
 forceencaps=yes

 left=<public_server_ip>
 leftprotoport=17/1701
 right=%any
 rightprotoport=17/%any
 rightsubnet=vhost:%priv,%no