Prodution Architecture and Deployment with Fabric

- Andy McCurdy @andymccurdy

Whiskey Media

Whiskey Sites

Your First Django App

Basic Config (web)

Apache mod_wsgi
use daemon mode threads more efficient processes if you're unsure of thread safety

WSGIDaemonProcess my-site python-path=/home/code/ processes=2 threads=150 maximum-requests=5000 WSGIProcessGroup my-site WSGIScriptAlias / /home/code/my-site/deploy/wsgi/my-site.wsgi

Basic Config (media)

Nginx
Use Nginx to proxy traffic to Apache Meanwhile Nginx serves media
upstream my-site { server 127.0.0.1:8000; } server { listen 80; location ~ ^/media/ { root /home/code/my-site; expires 30d; } location / { proxy_pass http://my-site; proxy_set_header X-Real-IP $remote_addr; }

Basic Config (db)

Databases PostgreSQL
Frank Wiles @ www.revsys.com

MySQL
Percona @ www.mysqlperformanceblog.com

Basic Config (cache)

Use Memcached! Even 16mb will significantly help against a Digg or being Slashdot'ed It's incredibly easy...
# settings.py MIDDLEWARE_CLASSES = ( 'django.middleware.cache.UpdateCacheMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.cache.FetchFromCacheMiddleware')

CACHE_BACKEND = 'memcached://127.0.0.1:11211/' CACHE_MIDDLEWARE_SECONDS = 60*5

# 5 minutes

Basic Deployment
Copy Code & Media (rsync or scp) Run DB migrations / syncdb Bounce wsgi daemons

Managing Growth (db)

Move DB to its own server As much RAM as possible Write-heavy (>10%)? Get fast disks Tune your config file

Managing Growth (web)

Add more web servers Use a resource monitoring tool like Munin to understand if your app is CPU or memory bound

Even More Growth

Replicated or sharded Databases Multiple load balancers for redundancy Message queues Crons Search daemons (Solr, Sphinx) etc...

(not so) Basic Deployment

image credit: Brad Fitzpatrick

Deployment Reqs
Copy media to CDN Maintenance splash page Run DB migrations Install/Upgrade Python dependencies Add a new web server to the cluster Execute arbritrary commands for sysadmin maintenance tasks

Deployment Options
Capistrano
+ Out of box support for common use cases + Hooks to customize tasks + Source control integration + Threaded deployment to multiple hosts

- Ruby :(

Deployment Options
Fabric
+ Very simple, tasks are just Python functions + Easy to chain together tasks to create complex scripts out of bite size pieces - No source control integration - No out of box support - Some bugs, although fairly easy to work around, and new maintainer is working on fixes

Fabric Basics
sudo easy_install fabric need a fabfile.py from fabric.api import * be mindful of tasks that may fail each remote command starts fresh changing directories

Core Functionality
local() - Run a command locally run() - Run a command remotely sudo() - Run a command remotely as
another user

put() - Copy a file from local to remote get() - Copy a file from remote to local many more helper-ish commands

Authentication
Relies on SSH model Use SSH keys Control access to root user via sudoers When you have to revoke access, you just turn off their SSH account

Configuration
Fabric environment (env) -- it's just a dictionary
Hosts and Roles Code Repositories Whatever you need

~/fabricrc
Global settings or all Fabric deployments SSH username

Example Config
# fabfile.py from fabric.api import * env.roledefs = { 'web' : ['10.1.1.1', '10.1.1.2'], 'db' : ['10.1.1.3'], 'lb' : ['10.1.1.4'], } env.repositories = {...}

Tasks
# fabfile.py def uptime(): run('uptime')

$> fab uptime -H 10.1.1.3 [10.1.1.3] run: uptime [10.1.1.3] out: 05:20:39 up 88 days, 12:00, 0 users, load average: 0.03, 0.03, 0.00

Mapping Roles to Tasks

# fabfile.py @roles('web') def uptime(): run('uptime') $> fab uptime [10.1.1.1] run: uptime [10.1.1.1] out: 05:20:39 up 88 days... [10.1.1.2] run: uptime

Decorator Problems
Some problems with Fabric's role management Can't override decorated tasks at command line as docs suggest
def default_roles(*role_list): def selectively_attach(func): if not env.roles and not env.hosts: return roles(*role_list)(func) else: if env.hosts: func = hosts(*env.hosts)(func) if env.roles: func = roles(*env.roles)(func) return func

All better now

#fabfile.py @default_roles('web', 'db') def uptime(): run('uptime') $> fab uptime # runs on all hosts in the 'web' and 'db' roles

$> fab uptime --roles lb

Dealing with Failures

By default Fabric dies if a task fails Use a context manager when failures are anticipated
# fabfile.py from __future__ import with_statement py2.5 def symlink_me(): with settings(warn_only=True): run('rm /path/to/symlink') run('ln -s /home/andy /path/to/symlink') #

Easy sys-admin
Make an "invoke" command Great for sys-admin and one-off tasks
# fabfile.py @default_roles('all') def invoke(command): "Invoke an arbritrary command" sudo(command) # install new packages on all hosts in one command

Real World Tasks

$> fab --list Available commands:

bounce_wsgi_procs deploy deploy_media invoke migrate reload_nginx splash_off splash_on update_repositories update_dependencies

Bounce the WSGI procs by touching the files Full deployment Push media to S3 Invoke an arbritrary command Run any migrations via South Update Nginx's running config Configure Nginx to serve the site Configure Nginx to serve a downed-site page Push code to servers Update dependencies to third party libs

Whiskey's Deployment
def deploy(splash='no'): "Full deployment" deploy_media() update_cached_repositories() update_dependencies() generate_releases() if splash == 'yes': splash_on() _symlink_code() migrate() bounce_wsgi_procs() if splash == 'yes': splash_off()

$> fab deploy:splash=yes

Questions?