OpenVPN + ufw + cloud servers = goodness April 28th, 2011

It’s been a number of years since I’ve had my sysadmin hat on for real and I was happy today to find out that the world has changed for the better.  The last time I did this, I had to deal with setting up actual equipment starting with an ethernet cable in one datacenter and two incoming T1′s at another.  I have vague recollections and nightmares of trying to decipher the meaning of blink starts and esoteric T1 setup stuff while trying to balance the Cisco book open on top of a chair in the datacenter supporting a monitor and keyboard.  And of course the cell phone calls to SBC.  ”Ok, try now…  No it’s still blinking.”  And then once I got the routers online, I had to navigate setting up a basic PKI to establish the vpn between sites so that things could actually talk to each other.  Then came the servers, and the monitoring systems… I managed through it but there was trauma.  All in all, the experience was worthwhile in that same way I imagine basic training is to people going into the army.  I would *never* do it again, but it gave me a perspective and some skills that few software developer types have.

But now, thankfully, its 2011 and things are easier.  Of course, everyone knows that just being able to login to an account at your favorite cloud providers and provision instances is a god-send, but the thing that has always bugged me about provisioning small numbers of internet connected servers is the feeling like I just walked out of my front door without any clothes on.  I say small numbers because if I were setting up an orderly install with any bulk, I would spend some time making sure it was all seamless, secure and accessible.  But more often these days, it’s small numbers of servers for one task or another and spending any time making things manageable isn’t usually in the cards.  There are services out there that are more turnkey, but I’ve never really used them.

So, continuing my naked outside analogy, the first thing I always do is grab a towel off the nearby clothes line (ie. lock down SSH with public key auth and a few other security bits).  This creates a single server that is more or less secure, but often then I want to connect to my private GIT repo, mirror files onto it or just generally manage it by connecting to administrative ports, whether it be for the database server, memcache, file sharing, jconsole, etc.  I don’t want any of these ports flapping out on the internet, but I also get tired of doing one off ssh tunnels to get at them from my workstation.

Today, I finally did spend a few hours working on this problem.  This post is somewhere between a howto and notes to myself should I need to do this again.  The tools I used were:

I’ve built OpenVPN + iptables management backplanes before but what really surprised me today was how easy this has gotten to be under recent Ubuntu installs.  I’ve got one permanent VPS that I run with Linode that I used as the VPN concentrator.  All I had to do was “apt-get install openvpn ufw” to get the bits.  Then on the concentrator, I took the following steps:

  1. Expand the OpenVPN “easy-rsa” files (/usr/share/doc/openvpn/examples/easy-rsa) into a fresh GIT repo and run the commands to generate a server cert
  2. Copy the server cert and config file to /etc/openvpn (sample config is at /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz)
  3. /etc/init.d/openvpn restart

Then I added the following “mkclient” script to my easy-rsa git repo:

die() {
	echo "$*"
	exit 1

td=$(dirname $0)
if [ -z "$name" ]; then
	echo "Need client name"
	exit 1

mkdir -p $cdir

if ! [ -f $td/keys/$name.key ]; then
	# Generate
	source $td/vars
	$td/build-key $name

cp $td/client.conf $cdir
cp $td/keys/ca.crt $cdir || die "Could not find ca.crt"
cp $td/keys/$name.crt $cdir/client.crt || die "Could not find $name.crt"
cp $td/keys/$name.key $cdir/client.key || die "Could not find $name.key"

I also put a client.conf (example is at /usr/share/doc/openvpn/examples/sample-config-files/client.conf ) in the directory that is the default configuration file for any connecting client. Then setting up a client is just a matter of:

  1. ./mkclient myclient
  2. scp clients/myclient/* root@myclient:/etc/openvpn
  3. ssh root@myclient /etc/init.d/openvpn restart

If everything works, your client should pop onto the network.  I treated my workstation as a client as well, but you can use any OpenVPN gui to do the same thing given a conf file and keys.

I’ve configured OpenVPN before so I didn’t really follow any instructions on modifying the confs.  The basic process is to take the stock server.conf example and client.conf example and make the following changes:

  • Keep the bridging bits commented out.  You want a routed network.
  • Keep the default to use UDP
  • Change the “server” directive to be a private subnet of your own choosing
  • Make sure that “ifconfig-pool-persist ipp.txt” is not commented.  This will make your clients keep the same IP addresses over time.
  • Uncomment “client-to-client” which allows all clients to see each other as well as see the server (this differs from a typical road-warrior config because our “clients” are mostly servers that we are trying to create a management network for)
  • Potentially tweak keepalive.  Keep in mind, though that if you have firewall filtering on UDP 1194 traffic, this will most commonly be stateful and if your keepalive is longer than the firewall timeout, you will start to drop packets

Ok, so far so good.

What I was after from this point was to have host based firewall configs that are default deny with holes punched through for internet accessible services.  But I want the management OpenVPN network to be able to access anything on the host.  This would be the security model of a hard candy shell with a soft gooey center.  I wouldn’t necessarily recommend it as-is for production installs, but for dev/test systems or non-sensitive prod systems, you can’t beat how easy it is to get at everything.  If anyone gets onto any one of the hosts, they are going to be able to access privileged ports on any of the hosts.  This is more or less just like any office LAN.

This is where my memory was telling me to buckle down and get cozy with some obtuse iptables configs, but things have changed.  Ubuntu now comes with UFW, which stands for Uncomplicated Firewall.  It’s built to make the process of administrating host-base firewalls brain dead simple.  Perfect.  To get it going on a host, run “ufw –force enable && ufw default allow”.  Note I put it together into one command to enable the firewall and set its default policy to allow so that I don’t run the risk of dropping my ssh connection until I’ve got my rules in place.

As an example for one of my hosts, I then ran the following commands:

# Allow openvpn (this really only applies to the server)
ufw allow 1194/udp
ufw allow 1194/tcp

# Allow ssh
ufw allow 22/tcp

# Allow web
ufw allow 80/tcp
ufw allow 443/tcp

# Allow all from private management network (from OpenVPN conf)
ufw allow from

# Change default to deny everything else
ufw default deny

And that’s it. You can see the config by running “ufw status verbose” or its bigger brother “iptables -L”. You should also do some poking at it from the internet side and the vpn side to ensure that your private ports really are private.

The only gotcha I experienced was already alluded to: The firewall config was blocking some OpenVPN traffic.  In order to diagnose traffic drop issues, run “ufw log on” and “tail -f /var/log/messages”.  An example of the problem is below:

Apr 28 17:21:27 client kernel: [UFW BLOCK] IN=eth0 OUT= MAC=xxxxxx SRC={vpnserverip} DST={clientip} LEN=129 TOS=0×00 PREC=0×00 TTL=63 ID=0 DF PROTO=UDP SPT=1194 DPT=43152 LEN=109

Here we see that the kernel is blocking UDP packets from the vpn server to the client originating from port 1194.  This is because our firewall on the client is initiating the vpn “connection” by sending a UDP packet to port 1194 on the vpn server from a randomly assigned port on the client.  If this exchange happens after the firewall rules are flushed or if the firewall’s stateful packet filter doesn’t remember the translation, it’s just going to deny the traffic.  If you see this right after resetting/enabling the firewall, just bounce the openvpn service (/etc/init.d/openvpn restart) to have it reestablish a connection that the firewall will remember.  You should also make sure that the “keepalive” directive on the server is set sufficiently low to keep things from timing out.  Mine is set at “30 120″.  Finally, if all else fails, you could switch everything to tcp or add a rule on the client that allows any traffic from udp port 1194.  I’m not listing that rule here because while it looks simple enough, in actuality it opens all udp ports on your host to an attacker.

Once I put a little mileage on this setup, I’ll probably also block ssh (port 22) at the host firewall.  I want to make sure I don’t lock myself out first though!

This entry was posted on Thursday, April 28th, 2011 at 10:57 am and is filed under geeky. You can follow any responses to this entry through the RSS 2.0 feed.You can leave a response, or trackback from your own site.

No Responses

Leave a Reply