agile-ajax

Automated Deployments Rock

Jason Sendelbach, Thursday, November 20, 2008 @ 5:30 pm

I have been a developer for many moons, and have taken many systems to production. I must say that production deployments were never on my list of things that make me warm and fuzzy. This was due to the fact that they weren't automated, or the automation was poor. On a recent java project, we had a series of simple sh scripts that did the work of deployment. We deployed war files to 12 application servers and to about 60 JBoss nodes. Before the automated scripts, a developer who typed wicked fast did all of the deployments. Without him, deployments would have been a nightmare.

I learned Ruby on Rails last year, and was delighted when I found Capistarno. I wanted to share the things I love and hate about capistrano.

Nice things:

  • The deployment script is the same for each environment with a configuration file for each environment. It's really easy to see what's different in each environment.
  • The deployment can specify a SVN tag/branch. Our configurations usually fail if you don't specify a branch to avoid the old "oops i just deployed trunk to production...." type errors. (Let's just say that I almost made that mistake, and added that configuration the next day)
  • Copies a system maintenance page out, which our apache configuration uses to halt traffic to the application servers during deployment
  • We actually build our apache configuration file in our deployment script. We can then test out new redirects and other apache specific changes. We point apache to these conf files with symbolic links, and our last deployment step is to restart apache. We also configure our mongrel cluster with configuration parameters, so we can easily modify these parameters by simply changing the configuration and deploying. Of course on the projects that we have moved to passenger don't need this.
  • We have a development deployment target, so we can deploy locally, and get as close to the production environment as possible. A coworker who develops on Windows (yuck) has had some issues matching production, as he can't run passenger (and apparently support for windows is not on the road map). Luckily he runs VMware, so he has a Linux VM that he uses as a local development environment that alleviates most of his windows "issues." Another nifty feature with that approach is that his rake runs like a hundred thousand times faster in linux compared to windows.
  • Symlinks are great. If you need to rollback, you can manually switch some symlinks until you figure out what the hell happened.
  • Keeps the last n deployments. We keep 5, and unless you do a number of screwed up deployments in a row, this usually suffices.
  • At first I hated the fact that there really is no rollback support, but after doing a few tests in staging to make sure I trusted capistrano, I found that adding rollback functionality isn't possible, as you wouldn't really be able to tell what went wrong, and programatically determine how to fix it. What I like about capistrano, is that it leaves the maintenance page up, and gives you a log of the problem. From there you can figure out what needs to be done to fix the deployment, or you can quickly change some symlinks to revert the deployment. This is the only reason the deployment to production needs to be done by a developer and not some non-technical resource.
  • The deployment directories are standardized, so it's not difficult to go find an installation on a new project.
  • It is pretty quick to add a new application server or get a deployment to a new server in the event of hardware failure. This helps with growth.
  • As the scripts are pretty easy to run, we can have our QA team actually do the deployments. This keeps our developers free to geek out and not need to be nagged to deploy releases all the time.

! Nice things:

  • Capistrano exports from SVN, tars the export, copies it to each application server, and then does its fun symlink magic (and anything else you configure). Luckily, it sends the system maintenance page out before it starts any of this. So my site is down during export, tar and copy over the network. Before I had a central deployment server, I would deploy from my laptop, and this would often be from home at night. Luckily my crappy DSL is limited on its upload bandwidth, so deployments would take almost 3 times as long from home. I looked into changing this, but I was getting deep into capistrano, so decided to just deal with it and use a deployment server at the office with the nice big interweb tubes.
  • The documentation is good, but the formatting on the site makes me curse it every time I find myself on their site.
  • Looking at a few of the configuration files for some of our production systems, I have noticed that the before and after configurations can be quite difficult to read through and figure out exactly what is happening and when.

I must say that capistrano is not perfect, but it rocks. In general any automation of deployment tasks is worth their weight in gold.

Topics: ,

agile ajax Agile Ajax

Comments: 1 so far

  1. Hey Jason,

    I’m on the capistrano documentation team, and we’re always looking to make things easier to understand, better formatted, etc. Join us if you like, it would be helpful to have your input:

    http://groups.google.com/group/capdoc
    http://wiki.capify.org

    - Scott

    Comment by Scott Becker, Monday, November 24, 2008 @ 7:43 pm

Leave a comment

Powered by WP Hashcash

Who is Pathfinder?

Categories

Topics

Blogroll

Search

WordPress

Comments about this site: info@pathf.com