Ryan Kanno: The diary of an Enginerd in Hawaii

Two events spurred me to write this blog.

First, my 2 year old “Subversion + Dreamhost + Post-Commit” blog still gets quite a number of hits. Second, after the latest Dreamhost outage move, I’m beginning to feel a little more vigilant about backing up my data.

As a standard disclaimer, if you’re not familiar with the Unix shell, I highly suggest you not try this unless under the supervision of someone who reads Perl books for fun. By accessing your Dreamhost shell, you can seriously f-up your account and I will not fix it for you. You have been warned. (Don’t you just love smileys?)

Setup

There are a few prerequisites to being able to back up your SVN repository.

1. First and foremost, you must have already installed a SVN repository into your Dreamhost account via the control panel.
2. Second, you must know how to SSH into your Dreamhost account. As a FYI, you sorta-kinda-need to know what that means in order to follow this tutorial.

Grabbing the backup script

Wait, you didn’t think I was writing my own right? In any case, if you actually installed/compiled Subversion on your own, it would’ve contained this file, hotbackup.py. Fortunately for us, Dreamhost has this file conveniently available at: /usr/bin/svn-hot-backup, but it’s an older version of the backup script. There are some subtle differences like being unable to pass in the number of backups you want the script to manage. Personally, I like to be on the edge, so let’s get the latest version. Execute the following commands from your home directory.

$ cd ~
$ mkdir scripts
$ cd scripts
$ wget http://svn.collab.net/repos/svn/trunk/tools/backup/hot-backup.py.in
$ mv hot-backup.py.in svn-hot-backup.py

The commands issued above created a directory called scripts in your home directory, switched into the directory, downloaded the latest hot-backup.py file from CollabNet, and renamed it to svn-hot-backup.py. Now that you have the file, you’ll need to make a few edits. Personally, I’m accustomed to vi, but pick your poison (pico, nano, text editor of your choice) and find these two values (they should be close to the top of the file in consecutive lines).

# Path to svnlook utility
svnlook = r"@SVN_BINDIR@/svnlook"

# Path to svnadmin utility
svnadmin = r"@SVN_BINDIR@/svnadmin"

and change them to the following:

# Path to svnlook utility
svnlook = r"/usr/bin/svnlook"

# Path to svnadmin utility
svnadmin = r"/usr/bin/svnadmin"

(If you’re wondering, if and when you compile/install Subversion yourself, these two variables would have been automagically filled in for you.)

The python script we downloaded not only performs a hotcopy of your svn directory, but also can archive it and manage a set number of copies. Pretty neat right?

Preparing for the backups

Before you can actually back up your SVN repository, you’ll want to create a directory structure to manage your backups. Execute the following commands from your home directory.

$ cd ~
$ mkdir backup
$ cd backup
$ mkdir svn
$ cd ~/scripts

The commands issued above created a directory called backup in your home directory, switched into the directory, and created another directory called svn within the backup directory. We’ll be using this directory to store all your backups. Finally, we switched back into the scripts directory created in the previous steps. Now that we have the backup script and directory structure to manage the back ups, let’s test it out!

Before you can back up your repository, you’ll have to know the name of the Subversion repository you’re trying to back up. To find the name of your repository, you can either look in the svn directory in your home directory, or you can check out the ID value in your Subversion Goodies control panel. In any case, remember the name of your SVN repository and issue the following commands.

$ cd ~/scripts/
$ python2.4 svn-hot-backup.py --archive-type=zip --num-backups=10 ~/svn/REPOSITORY_NAME_HERE/ ~/backup/svn/

You should see the following in the console:

Beginning hot backup of '/home/USERNAME/svn/lkg/'.
Youngest revision is REVISION_NUMBER
Backing up repository to '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701'...
Done.
Archiving backup to '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701.zip'...
Archive created, removing backup '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701'...

Voila! (But there’s more)

Automating the backups

Now that you actually have the script backing up your SVN repository, let’s automate them! To do so, we’ll use the handy cron daemon. Cron has similarities to the Windows task scheduler in that it provides a service that enables a user to execute commands at a specified date/time or set intervals. To tell cron the tasks you want to execute, you’ll need to load a configuration file called a crontab. You can read more about it here and here. In any case, here’s what my crontab configuration file looks like.

MAILTO=ryankanno@CHANGE_TO_YOUR_EMAIL.com
# minute (0-59),
# |      hour (0-23),
# |      |       day of the month (1-31),
# |      |       |       month of the year (1-12),
# |      |       |       |       day of the week (0-6 with 0=Sunday).
# |      |       |       |       |       commands
  0      0       *       *       *      /usr/bin/python2.4 /home/USERNAME/scripts/svn-hot-backup.py --archive-type=zip --num-backups=10 /home/USERNAME/svn/REPOSITORY_NAME/ /home/USERNAME/backup/svn/

Create a file in your scripts directory called svn_backup_once_a_day.cron and copy the contents above into your file. I’ve setup my crontab to backup my svn repository once a day.

Once you have this file called svn_backup_once_a_day.cron in your scripts directory, load the file into your crontab by issuing the following command:

$ crontab svn_backup_once_a_day.cron

As a FYI, this will replace your old crontab. If you have other items already running on cron, it’s a good idea to list them via the crontab -l command first. If you want to make sure that your cron will run, you can test it out by setting the values in the crontab to the time you want it to run. I’ll leave this as an exercise to the reader.

Storing your backups

Though out of scope of this blog, you’ll still have to store your backups somewhere. Please just don’t leave them in your Dreamhost account. Your best bet is probably to get an Amazon S3 account and store your backups there. Personally, I like to run another script immediately after the hotcopy finishes that pushes the backup to my S3 account. Other options include scp/sftp’ing the backups to your home machine. Here’s a link to read more about that option.

Voila! Enjoy!

Popularity: 43% [?]

Now that my MBA class is finally done for the summer, I can focus on more important things… like upgrading my Python installation on Dreamhost for my Django application. Seeing as how Dreamhost is still behind the Python times, with Python 2.4 hidden in Dreamhost obscurity, I figured I’d blog about updating your Dreamhost Python installation (and subsequent MySQLdb libraries) to Python 2.5.

The very first thing I did was search Google. You know, I really don’t know how people lived pre-Googs. In any case, I found this blog posting describing exactly what I wanted to do. Thanks Ben! Since I’m not a big fan of running one large batch script people create in their blogs, I’ll break it down for the non-*nix fans out there.

Before I begin, I’m assuming that you already have Django running on Dreamhost. If you’re having a “wtf” moment, make sure to stop by Jeff’s blog and read “Setting up Django on Dreamhost“. (This is how I set mine up). To follow my short tutorial, you’ll need shell access to your Dreamhost account.

After ssh’ing into your Dreamhost account, you should be in your home directory (/home/username). According to the Filesystem Hierarchy Standard, the /opt dir “is reserved for the installation of add-on application software packages.” With that said, issue the following commands:

$ mkdir opt
$ mkdir downloads
$ cd downloads
$ wget http://www.python.org/ftp/python/2.5.1/Python-2.5.1.tgz
$ tar xvzf Python-2.5.1.tgz

First, create a directory named opt. Next, create a directory named downloads for all your files. Change into the downloads directory, then download the latest Python from http://www.python.org. Finally, unzip and untar the package into the download directory. Everything will extract into a directory named Python-2.5.1.

$ cd Python-2.5.1
$ ./configure --prefix=$HOME/opt/ --enable-unicode=ucs4
$ make
$ make install

Change into the Python-2.5.1 directory and type in the following configure command. Basically, configure prepares your installation for compilation. The –prefix flag will install machine-independent data files in subdirectories of the specified directory. The default is to install in /usr/local, but it’s overwritten with the opt directory created earlier. Finally, run make and make install which will install your custom Python installation.

$ cd ..
$ rm -rf Python-2.5.1

Finally, delete the Python-2.5.1 directory. Before you can use this Python installation, you have to add the /opt/bin directory to your path. To do this, add /opt/bin to your .bash_profile file in your home directory. To do so, you’ll have to add the following to your .bash_profile.

export PATH=$HOME/opt/bin/:$PATH

Basically, this allows you to type ‘python’ in your shell and reach the custom Python 2.5.1 installation instead of the Dreamhost one. To make sure that our Python installation is working, type the following in your home directory (cd ~):

$ source .bash_profile
$ python --version

After the last command, you should see the following: Python 2.5.1. If that displays, your upgrade was successful! After upgrading your Python installation, you’re not done yet. Since Dreamhost uses an old MySQL-Python installation, we’ll upgrade that as well. Type the following in your home directory:

$ cd downloads
$ wget http://internap.dl.sourceforge.net/sourceforge/mysql-python/MySQL-python-1.2.2.tar.gz
$ tar xvzf MySQL-python-1.2.2.tar.gz
$ cd MySQL-python-1.2.2
$ python setup.py install

First, change into the downloads directory and issue the wget command to download the latest MySQL-Python files from Sourceforge. Once you’ve received the files, unzip and untar the package. All the files will extract into a directory called MySQL-python-1.2.2. Change into this directory and install the files by typing python setup.py install. If you followed the custom Python installation, the files should build and extract into the ~/opt/lib/python2.5/site-packages/ directory as an egg file.

You now have a custom Python installation and a MySQL-Python upgrade!

Voila!

Update: Just so you don’t get caught up in the same mistake that I made, to be sure that your Django fcgi installation is using your custom Python installation, make sure the dispatch.fcgi file reads as such:

#!/home/USERNAME_GOES_HERE/opt/bin/python
import sys

sys.path += ['WHATEVER_PATHS_YOU_NEED']

from fcgi import WSGIServer
from django.core.handlers.wsgi import WSGIHandler
import os

os.environ['DJANGO_SETTINGS_MODULE'] = 'wegoeat.settings'
WSGIServer(WSGIHandler()).run()

Popularity: 38% [?]

Since I haven’t put out a technical article in a while, this blog will explain how I’ve setup Dreamhost + Django + Subversion to play nicely together in a seamless development environment via a shared hosting provider. Hopefully – someone, somewhere can find this information useful and insightful in their own development environment.

The very first thing I did was unleash my first Django web application on Dreamhost. Thanks to an excellent tutorial from Jeff Croft, a detailed explanation about FastCGI contained within the Django documentation, and a few helpful pointers on the Dreamhost wiki, I was able to get my application deployed in a matter of a few hours.

You can check it out here!

However, after going through Jeff’s excellent tutorial, I still wasn’t completely satisfied with my Django deployment on Dreamhost. Something was missing. There wasn’t a seamless way to continue development on my home machine, deploy to a test environment, and still keep my live site intact. After all, I’m a true believer in the open source dictum of ‘release early, release often‘, and without a way to test my application on a live server, I wasn’t happy with my configuration management.

Ideally, I envisioned having a live web application (i.e. http://www.wegoeat.com/) and another url that I could deploy my beta releases to (i.e. http://beta.wegoeat.com/). From a configuration management standpoint, I would tag major release builds and to maintain that release over its life (via bug fixes, minor enhancements), I would create a branch of the tag. Thus, the live site would be updated from the branches directory, while the beta url would update from the trunk in my Subversion repository. So to summarize the ‘extra’ steps I did to ensure a smoother deployment cycle, I’ve conjured up the following action list.

1. The very first thing I did was follow Jeff’s tutorial – instead of creating a single directory in my django_projects directory, I created two. One was named ‘project_live’ and the other ‘project_beta’.
2. Next, I checked out the appropriate source files from the appropriate locations in my Subversion repository. The ‘project_live’ directory came from my branches directory and represents my ‘live’ site. The ‘project_beta’ directory came from the trunk and represents my ‘beta’ site. Obviously, the settings.py file for the Django applications as well as the configuration files for FastCGI were different according to the directories. Since my settings will probably be very different then your settings, I’ll leave this as an exercise to the reader.
3. Note, as far as Dreamhost goes, I created two domain entries, one @ http://www.wegoeat.com that will host my live site, and another @ http://beta.wegoeat.com that will be my beta site.
4. I followed my own tutorial and created a post-commit hook to update the appropriate Dreamhost directories when I committed to the repository.

And voila! We’re done.

Now, I can develop on my home machine where I’ve checked out the trunk of my Subversion repository. Whenever I commit, the post-commit hook updates the project_beta directory on my Dreamhost account, and all the while, my live site is still functioning.

Stay tuned for my next blog where I discuss how to get Custom PHP + MediaWiki + EAccelerator playing nicely together on Dreamhost!

Popularity: 24% [?]

Update

According to Jeff, Dreamhost has disabled setuid for home directories – which they probably should’ve done years ago. As a side note, some other alternatives include the following:

• Setting the directory permissions to +777 (probably not a good idea)
• Setup cron to perform the same task. If you need an introduction to cron, check out my blog about backing up your Subversion respository

In any case, I’ve since moved half my sites to WebFaction so if you know of a solution, post it in the comments and I’ll write a new blog post about it.

Since Dreamhost has integrated Subversion into their services, I finally decided to take the leap and move my local repository to my hosted account. After all, I was tired of always having to remember to transfer my updated files via scp. With my repository moved over to Dreamhost, I figured I would take advantage of Subversion’s post-commit hooks to automagically update my live beta site.

After reading a myriad of solutions (like here and here ), I found the Dreamhost wiki entry the most helpful. So without further adieu, here’s a small, quick and dirty tutorial on how to automagically get post-commit working with Subversion on your Dreamhost account.

The setup

The very first thing you need to do is create a directory on your Dreamhost account to checkout the source files to (I’m assuming you’ve already created your Subversion repository, have checked files into it, and are ssh’ed into your Dreamhost account.). In my situation, I created a subdomain in Dreamhost called beta.localkinegrinds.com. Dreamhost automatically creates a directory named beta.localkinegrinds.com within the root of the admin account. I cd’ed into that directory and issued the following command:

svn co PATH_TO_FILES_IN_REPOSITORY . --username USERNAME --password PASSWORD

Once checked out, I switched into my Subversion repository directory; it was named svn in the root of my admin account. From there, I went into my named repository, and then into the ‘hooks’ directory.

Here’s where all the fun begins!

First, I created a script called ‘post-commit.script’ that looked like the following:

#!/bin/sh
/usr/bin/svn up PATH_TO_DIR --username SVN_USERNAME --password PASSWORD

I exited my text editor, then issued the following command:

chmod +x post-commit.script

From reading all the forums, the main issue with the post-commit hooks is that svnserve will run the hooks as the Apache user; in Dreamhost’s case, the dhapache user. Thus, we have the following options as described here. We can:

1. Run password-less sudo (which we don’t have access to on Dreamhost)
2. Change the group permissions (which we don’t have access to on Dreamhost)
3. Checkout the files as the dhapache user from a post-commit hook, then change it to update as described here (which I thought was a pain in the rear, not to mention having the directory be world writeable)
4. Compile a small C program with its setuid bit set

Being the programmer that I am, I chose the last option. Basically, as Wikipedia describes, “when a binary executable file has been given the setuid attribute, normal users on the system can execute this file and gain the privileges of the user who owns the file (commonly root) within the created process.” This means that even though the Apache user will run the post-commit hook, it’ll assume the privileges of the user who owns the file (my account) and be able to execute the post-commit.script file we just created!

Neat!

So… taking a look at the Dreamhost wiki, I created the following c program named post-commit.c

#define PATH_TO_POST_COMMIT_SCRIPT "/PATH/TO/POST_COMMIT.SCRIPT"
#include <sys/types.h>
#include <unistd.h>
main( ac, av ) char **av;
{
  execv( PATH_TO_POST_COMMIT_SCRIPT, av );
}

I then issued the following command to compile the program into a binary called post-commit:

gcc -o post-commit post-commit.c

And then to set the setuid bit on the post-commit file:

chmod 4755 post-commit


Voila!

Assuming you set all the correct paths, on a commit to your Subversion repository – your live site will be updated!

As a side note, most of the sites previously mentioned didn’t have any troubleshooting tips. Here’s a few that I ran into along the way:

1. Always test the scripts out first, ie, execute them from the command line and make sure they are doing what you want them to.
2. Apparently, according to the Subversion book, “Subversion repository executes hook scripts with an empty environment.” This is something good to know.
3. If everything was done right, and it’s still not working, try to update your checked out Subversion directory. Sometimes you need to clean up before any other updates can occur. That problem took me a whole half-an-hour to figure out.

In any case, if done correctly, you’ll be on your way to a better development environment!

Popularity: 42% [?]