A redundant load-balancing firewall system, using FreeBSD.

Goal: Configure two or more redundant IPF-based firewalls, which will also act as load-balancers (henceforth referred to as "FWLBs") for an internet services cluster. If one firewall fails, the second will take over as the firewall/load balancer. Barring any catastrophes, this should result in a highly-available, almost zero downtime environment. This hopefully saves you loads of money on expensive dedicated hardware devices. We'll end up with a configuration like the following:
~~~ ( net ) ~~~ | | | ---------| switch | ---------/ \ / VIP \ ---------------| fwlb1 | | fwlb2 | ---------------\ VIP / \ / --------| switch | --------/ \ / \ ----------------| server1 | | server2 | -----------------

Where the first VIP (Virtual IP) is the publicly known IP of your site, and the second is an RFC 1918 private address that your servers will use as a default route and NAT through.

Prerequisites: - Two dual-homed FreeBSD-STABLE boxes - Two switches/hubs

- Two or more real servers for a cluster - /usr/ports/net/pen - /usr/ports/net/freevrrpd - /usr/ports/sysutils/daemontools or /usr/ports/sysutils/runit - kernel compiled with IPFILTER and IPFILTER_LOG

is a simple but flexible load balancer. FreeVRRPd is what will be handling our failover for us. Daemontools(or Gerrit Pape's excellent, more licensing-friendly and featureful workalike, runit) is not strictly necessary, but it's a tool I find quite useful for process control and supervision, so I've used it in the examples here.
Pen

Procedure: PART 1: The Firewall Create an IPF(or IPFW, or PF, for that matter) ruleset, allowing traffic in on the port of the service you'll be balancing. Optionally, allow the cluster to be able to NAT through this box, if they'll be needing to initiate outbound connections. IPF/IPNAT configuration is beyond the scope of this document, but there's been plenty written on the subject. The two points important to keep in mind are to make sure you allow free multicast communication between both firewalls(VRRP requires it), and to make sure other hosts can't(or you'll run into some unpleasant security possibilities). For more IPF info, please refer to:
http://www.nwo.net/ipf/ipf-howto.html http://coombs.anu.edu.au/~avalon/ip-filter.html

Short story - build a kernel with the options above, add the following to /etc/rc.conf: ipfilter_enable="YES" ipfilter_rules="/etc/ipf.conf" ipnat_rules="/etc/ipnat.conf" ipnat_flags="-CF" ipmon_enable="YES" And put your rulesets in the proper places. PART 2: The balancer 1) Create the necessary users and directories. mkdir -p /etc/supervise/pen/log

mkdir -p /var/chroot/pen mkdir -p /var/log/pen pw useradd pen -s /bin/false -d /var/chroot/pen pw useradd penlog -s /bin/false -d /var/chroot/pen chown penlog:pen /var/log/pen 2) Create the runfiles for pen. cd /etc/supervise/pen cat << _EOF_ > run #!/bin/sh exec 2>&1 exec pen -d -u pen -j /var/chroot/pen -C localhost:8888 -f -r 80 hostname1 hostname2 _EOF_ chmod 755 run cd log cat << _EOF_ > run #!/bin/sh exec /usr/local/bin/setuidgid penlog /usr/local/bin/multilog s999999 n20 /var/log/pen _EOF_ chmod 755 run

This will configure pen to run chrooted in /var/chroot/pen, with a control port of 8888. It will be balancing port 80 incoming to port 80 on hostname1 and hostname2. This is configured for round-robin balancing - if you require sticky sessions, remove the "-r" flag. This example has pen logging somewhat verbosely, to aid in debugging. You may wish to remove the "-d" in a production environment. 3) Start up the load-balancing services. cd /service ln -s /etc/supervise/pen echo "csh -cf '/usr/local/bin/svscanboot &'" >> /etc/rc.local csh -cf '/usr/local/bin/svscanboot &' sleep 5 && svstat pen

You should now be able to point your browser, dns resolver, etc to either of the IPs of these machines, and see it balancing out to your real servers. You can confirm this by tailing /var/log/pen/current on both machines. PART 3: Redundancy 1) First, configure syslog to log VRRP info to its own file. touch /var/log/freevrrpd.log cat << _EOF_ >> /etc/syslog.conf !freevrrpd *.* _EOF_

/var/log/freevrrpd.log

2) Configure FreeVRRPd Until this point, both machines have been equal. Now, you need to choose which FWLB is going to be your primary. On this machine, Copy /usr/local/etc/freevrrpd.conf.sample to /usr/local/etc/freevrrpd.conf. Edit the file, and configure it along the following lines: # public-facing VRID [VRID] serverid = 1 interface = fxp0 priority = 255 addr = 198.123.111.1/32 password = vrid1 vridsdep = 2 # backend VRID [VRID] serverid = 2 interface = fxp1 priority = 255 addr = 10.0.0.1/32 password = vrid2 vridsdep = 1 This results in 2 VRIDs being created - one for the front-facing

network, and one for the rear-facing one that the cluster will be using. In this example, both VRIDs are configured to consider this host the master server during VRRP elections. Note that both VRIDs depend on the other, specified by the "vridsdep" field. This is important - it means that if one of the interfaces in a FWLB fails, the other one will automatically go into backup mode, failing both interfaces to the slave FWLB. This avoids the backend servers trying to route through a machine with a dead front-end connection. You should now copy this file over to the slave FWLB, and change both priority fields to be 100. Change the password field on both to something more original, but certainly don't rely on VRRP passwords as a security measure. If another box outside of this cluster is in a position to communicate with it over VRRP, you've got a problem. 3) Start FreeVRRPd You can now start up freevrrpd on both boxes: cp /usr/local/etc/rc.d/freevrrpd.sh{.sample,} /usr/local/etc/rc.d/freevrrpd.sh start

PART 4: Failover testing Now you just need to verify that this whole setup works in the case of a failure. First, configure both FWLB boxes to start an SSH daemon, so we have something to connect to to verify that the interfaces fail properly. Try the following scenarios: - From one of the machines in your cluster, ssh to 10.0.0.1, and log in. Verify the hostname of the machine is the hostname of the master FWLB. - While watching /var/log/pen/current on FWLB1, connect to 198.123.111.1, port 80 from a machine on the front-end network. Verify that you see the connection occur. - Pull the front-end interface of FWLB1. - Watch the logs on FWLB2. Connect to 198.123.111.1, port 80, and verify that the connection occurred. - SSH again to 10.0.0.1. You should see the hostname of FWLB2 in the SSH banner.

- Reconnect the front-end interface of FWLB1. Verify that both interfaces on FWLB1 recover back to master state. Now conduct the same tests when unplugging the backend interface. For fun, you may want to just hit the reset button on FWLB1 while actively hitting the web servers.

Notes: Removing servers from a pool: Pen cannot permanently remove servers from a pool, but if you need to have it ignore a server while doing upgrades or such, you can just do: penctl localhost:8888 server $servername blacklist 99999 This will blacklist the server for 99999 seconds, giving adequate time to seed the server. When the server is back in service, just do: penctl localhost:8888 server $servername blacklist 1 This will reset the blacklist timeout to 1 second, bringing it back into the pool. Permanently adding or removing servers to/from a pool: In the case that more servers need to be added or removed, the /service/pen/run file will need to be edited. Simply add the hostname to the end of the pen command, and do: svc -t /service/pen This will TERM and restart pen. While this may not cause interruption to the user, it's probably wise to do this while in maintenance mode or off hours.

References:
http://www.faqs.org/rfcs/rfc2338.html http://www.bsdshell.net/hut_fvrrpd.html http://siag.nu/pen/ http://cr.yp.to/daemontools.html http://smarden.org/runit

 2004 David Thiel --- lx [@ at @] redundancy.redundancy.org Updated Sun Mar 21 18:23:28 PST 2004