Ryan Kanno: The diary of an Enginerd in Hawaii

Everything you've ever thought, but never had the balls to say.

My LinkedIn Profile
Follow @ryankanno on Twitter
My Feed

Tag Archive » ‘Dreamhost’

Backing up your Subversion (SVN) repository on Dreamhost with cron

Two events spurred me to write this blog.

First, my 2 year old “Subversion + Dreamhost + Post-Commit” blog still gets quite a number of hits. Second, after the latest Dreamhost outage move, I’m beginning to feel a little more vigilant about backing up my data.

As a standard disclaimer, if you’re not familiar with the Unix shell, I highly suggest you not try this unless under the supervision of someone who reads Perl books for fun. By accessing your Dreamhost shell, you can seriously f-up your account and I will not fix it for you. You have been warned. :) (Don’t you just love smileys?)

Setup

There are a few prerequisites to being able to back up your SVN repository.

  1. First and foremost, you must have already installed a SVN repository into your Dreamhost account via the control panel.
  2. Second, you must know how to SSH into your Dreamhost account. As a FYI, you sorta-kinda-need to know what that means in order to follow this tutorial.

Grabbing the backup script

Wait, you didn’t think I was writing my own right? In any case, if you actually installed/compiled Subversion on your own, it would’ve contained this file, hotbackup.py. Fortunately for us, Dreamhost has this file conveniently available at: /usr/bin/svn-hot-backup, but it’s an older version of the backup script. There are some subtle differences like being unable to pass in the number of backups you want the script to manage. Personally, I like to be on the edge, so let’s get the latest version. Execute the following commands from your home directory.

$ cd ~
$ mkdir scripts
$ cd scripts
$ wget http://svn.collab.net/repos/svn/trunk/tools/backup/hot-backup.py.in
$ mv hot-backup.py.in svn-hot-backup.py

The commands issued above created a directory called scripts in your home directory, switched into the directory, downloaded the latest hot-backup.py file from CollabNet, and renamed it to svn-hot-backup.py. Now that you have the file, you’ll need to make a few edits. Personally, I’m accustomed to vi, but pick your poison (pico, nano, text editor of your choice) and find these two values (they should be close to the top of the file in consecutive lines).

# Path to svnlook utility
svnlook = r"@SVN_BINDIR@/svnlook"

# Path to svnadmin utility
svnadmin = r"@SVN_BINDIR@/svnadmin"

and change them to the following:

# Path to svnlook utility
svnlook = r"/usr/bin/svnlook"

# Path to svnadmin utility
svnadmin = r"/usr/bin/svnadmin"

(If you’re wondering, if and when you compile/install Subversion yourself, these two variables would have been automagically filled in for you.)

The python script we downloaded not only performs a hotcopy of your svn directory, but also can archive it and manage a set number of copies. Pretty neat right?

Preparing for the backups

Before you can actually back up your SVN repository, you’ll want to create a directory structure to manage your backups. Execute the following commands from your home directory.

$ cd ~
$ mkdir backup
$ cd backup
$ mkdir svn
$ cd ~/scripts

The commands issued above created a directory called backup in your home directory, switched into the directory, and created another directory called svn within the backup directory. We’ll be using this directory to store all your backups. Finally, we switched back into the scripts directory created in the previous steps. Now that we have the backup script and directory structure to manage the back ups, let’s test it out!

Before you can back up your repository, you’ll have to know the name of the Subversion repository you’re trying to back up. To find the name of your repository, you can either look in the svn directory in your home directory, or you can check out the ID value in your Subversion Goodies control panel. In any case, remember the name of your SVN repository and issue the following commands.

$ cd ~/scripts/
$ python2.4 svn-hot-backup.py --archive-type=zip --num-backups=10 ~/svn/REPOSITORY_NAME_HERE/ ~/backup/svn/
Notice, change the value of REPOSITORY_NAME_HERE to the id of the SVN repository you want backed up.

You should see the following in the console:

Beginning hot backup of '/home/USERNAME/svn/lkg/'.
Youngest revision is REVISION_NUMBER
Backing up repository to '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701'...
Done.
Archiving backup to '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701.zip'...
Archive created, removing backup '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701'...
If you see the following, the backup was a success! You can even check on the file by changing into the backup/svn directory!

Voila! (But there’s more)

Automating the backups

Now that you actually have the script backing up your SVN repository, let’s automate them! To do so, we’ll use the handy cron daemon. Cron has similarities to the Windows task scheduler in that it provides a service that enables a user to execute commands at a specified date/time or set intervals. To tell cron the tasks you want to execute, you’ll need to load a configuration file called a crontab. You can read more about it here and here. In any case, here’s what my crontab configuration file looks like.

MAILTO=ryankanno@CHANGE_TO_YOUR_EMAIL.com
# minute (0-59),
# |      hour (0-23),
# |      |       day of the month (1-31),
# |      |       |       month of the year (1-12),
# |      |       |       |       day of the week (0-6 with 0=Sunday).
# |      |       |       |       |       commands
  0      0       *       *       *      /usr/bin/python2.4 /home/USERNAME/scripts/svn-hot-backup.py --archive-type=zip --num-backups=10 /home/USERNAME/svn/REPOSITORY_NAME/ /home/USERNAME/backup/svn/

Create a file in your scripts directory called svn_backup_once_a_day.cron and copy the contents above into your file. I’ve setup my crontab to backup my svn repository once a day.

Notice, change the value of ryankanno@CHANGE_TO_YOUR_EMAIL.com to your email address (or comment the line out with a # if you don’t want emails sent to you), USERNAME to your Dreamhost username, and REPOSITORY_NAME to your Subversion repository.

Once you have this file called svn_backup_once_a_day.cron in your scripts directory, load the file into your crontab by issuing the following command:

$ crontab svn_backup_once_a_day.cron

As a FYI, this will replace your old crontab. If you have other items already running on cron, it’s a good idea to list them via the crontab -l command first. If you want to make sure that your cron will run, you can test it out by setting the values in the crontab to the time you want it to run. I’ll leave this as an exercise to the reader. :)

Storing your backups

Though out of scope of this blog, you’ll still have to store your backups somewhere. Please just don’t leave them in your Dreamhost account. Your best bet is probably to get an Amazon S3 account and store your backups there. Personally, I like to run another script immediately after the hotcopy finishes that pushes the backup to my S3 account. Other options include scp/sftp’ing the backups to your home machine. Here’s a link to read more about that option.

Voila! Enjoy!

Tagged: , , , , , , , , .


Custom Python installation for Django on Dreamhost

Now that my MBA class is finally done for the summer, I can focus on more important things… like upgrading my Python installation on Dreamhost for my Django application. Seeing as how Dreamhost is still behind the Python times, with Python 2.4 hidden in Dreamhost obscurity, I figured I’d blog about updating your Dreamhost Python installation (and subsequent MySQLdb libraries) to Python 2.5.

The very first thing I did was search Google. You know, I really don’t know how people lived pre-Googs. In any case, I found this blog posting describing exactly what I wanted to do. Thanks Ben! Since I’m not a big fan of running one large batch script people create in their blogs, I’ll break it down for the non-*nix fans out there.

Before I begin, I’m assuming that you already have Django running on Dreamhost. If you’re having a “wtf” moment, make sure to stop by Jeff’s blog and read “Setting up Django on Dreamhost“. (This is how I set mine up). To follow my short tutorial, you’ll need shell access to your Dreamhost account.

After ssh’ing into your Dreamhost account, you should be in your home directory (/home/username). According to the Filesystem Hierarchy Standard, the /opt dir “is reserved for the installation of add-on application software packages.” With that said, issue the following commands:

$ mkdir opt
$ mkdir downloads
$ cd downloads
$ wget http://www.python.org/ftp/python/2.5.1/Python-2.5.1.tgz
$ tar xvzf Python-2.5.1.tgz

First, create a directory named opt. Next, create a directory named downloads for all your files. Change into the downloads directory, then download the latest Python from http://www.python.org. Finally, unzip and untar the package into the download directory. Everything will extract into a directory named Python-2.5.1.

$ cd Python-2.5.1
$ ./configure --prefix=$HOME/opt/ --enable-unicode=ucs4
$ make
$ make install

Change into the Python-2.5.1 directory and type in the following configure command. Basically, configure prepares your installation for compilation. The –prefix flag will install machine-independent data files in subdirectories of the specified directory. The default is to install in /usr/local, but it’s overwritten with the opt directory created earlier. Finally, run make and make install which will install your custom Python installation.

$ cd ..
$ rm -rf Python-2.5.1

Finally, delete the Python-2.5.1 directory. Before you can use this Python installation, you have to add the /opt/bin directory to your path. To do this, add /opt/bin to your .bash_profile file in your home directory. To do so, you’ll have to add the following to your .bash_profile.

export PATH=$HOME/opt/bin/:$PATH

Basically, this allows you to type ‘python’ in your shell and reach the custom Python 2.5.1 installation instead of the Dreamhost one. To make sure that our Python installation is working, type the following in your home directory (cd ~):

$ source .bash_profile
$ python --version

After the last command, you should see the following: Python 2.5.1. If that displays, your upgrade was successful! After upgrading your Python installation, you’re not done yet. Since Dreamhost uses an old MySQL-Python installation, we’ll upgrade that as well. Type the following in your home directory:

$ cd downloads
$ wget http://internap.dl.sourceforge.net/sourceforge/mysql-python/MySQL-python-1.2.2.tar.gz
$ tar xvzf MySQL-python-1.2.2.tar.gz
$ cd MySQL-python-1.2.2
$ python setup.py install

First, change into the downloads directory and issue the wget command to download the latest MySQL-Python files from Sourceforge. Once you’ve received the files, unzip and untar the package. All the files will extract into a directory called MySQL-python-1.2.2. Change into this directory and install the files by typing python setup.py install. If you followed the custom Python installation, the files should build and extract into the ~/opt/lib/python2.5/site-packages/ directory as an egg file.

You now have a custom Python installation and a MySQL-Python upgrade!

Voila!

Update: Just so you don’t get caught up in the same mistake that I made, to be sure that your Django fcgi installation is using your custom Python installation, make sure the dispatch.fcgi file reads as such:

#!/home/USERNAME_GOES_HERE/opt/bin/python
import sys

sys.path += ['WHATEVER_PATHS_YOU_NEED']

from fcgi import WSGIServer
from django.core.handlers.wsgi import WSGIHandler
import os

os.environ['DJANGO_SETTINGS_MODULE'] = 'wegoeat.settings'
WSGIServer(WSGIHandler()).run()

Tagged: , , , , , , .


My Dreamhost + Django + Subversion Setup

Since I haven’t put out a technical article in a while, this blog will explain how I’ve setup Dreamhost + Django + Subversion to play nicely together in a seamless development environment via a shared hosting provider. Hopefully – someone, somewhere can find this information useful and insightful in their own development environment.

The very first thing I did was unleash my first Django web application on Dreamhost. Thanks to an excellent tutorial from Jeff Croft, a detailed explanation about FastCGI contained within the Django documentation, and a few helpful pointers on the Dreamhost wiki, I was able to get my application deployed in a matter of a few hours.

You can check it out here!

However, after going through Jeff’s excellent tutorial, I still wasn’t completely satisfied with my Django deployment on Dreamhost. Something was missing. There wasn’t a seamless way to continue development on my home machine, deploy to a test environment, and still keep my live site intact. After all, I’m a true believer in the open source dictum of ‘release early, release often‘, and without a way to test my application on a live server, I wasn’t happy with my configuration management.

Ideally, I envisioned having a live web application (i.e. http://www.wegoeat.com/) and another url that I could deploy my beta releases to (i.e. http://beta.wegoeat.com/). From a configuration management standpoint, I would tag major release builds and to maintain that release over its life (via bug fixes, minor enhancements), I would create a branch of the tag. Thus, the live site would be updated from the branches directory, while the beta url would update from the trunk in my Subversion repository. So to summarize the ‘extra’ steps I did to ensure a smoother deployment cycle, I’ve conjured up the following action list.

  1. The very first thing I did was follow Jeff’s tutorial – instead of creating a single directory in my django_projects directory, I created two. One was named ‘project_live’ and the other ‘project_beta’.
  2. Next, I checked out the appropriate source files from the appropriate locations in my Subversion repository. The ‘project_live’ directory came from my branches directory and represents my ‘live’ site. The ‘project_beta’ directory came from the trunk and represents my ‘beta’ site. Obviously, the settings.py file for the Django applications as well as the configuration files for FastCGI were different according to the directories. Since my settings will probably be very different then your settings, I’ll leave this as an exercise to the reader.
  3. Note, as far as Dreamhost goes, I created two domain entries, one @ http://www.wegoeat.com that will host my live site, and another @ http://beta.wegoeat.com that will be my beta site.
  4. I followed my own tutorial and created a post-commit hook to update the appropriate Dreamhost directories when I committed to the repository.

And voila! We’re done.

Now, I can develop on my home machine where I’ve checked out the trunk of my Subversion repository. Whenever I commit, the post-commit hook updates the project_beta directory on my Dreamhost account, and all the while, my live site is still functioning.

Stay tuned for my next blog where I discuss how to get Custom PHP + MediaWiki + EAccelerator playing nicely together on Dreamhost!

Tagged: , , , , , , , , , , , , .


Backing up your WordPress installation.

Wordpress

I’ve found that many bloggers with WordPress installations seem to overlook the simple task of backing up their data. Rather, they put their blogging fate into the hands of their trustworthy web-hosting provider. Not that I don’t trust my provider, but I like to sleep knowing full well that my data is resting in a few safe places – rather than putting my eggs all in one basket. So you can either learn the hard way like here, here, and here, or you can follow the simple backup recipe below.

Backing up your WordPress blog calls for the installation of two very simple WordPress plugins, WP-DB Backup and WP-Cron, and a dummy Gmail account. After going through the painstaking task of downloading, installing, and configuring each plugin on my hosting provider, I come to find out that other users have reportedthat these plugins have stopped working for WordPress 2.0.4, the exact version I have installed on my hosting provider. After searching around a bit (how did people live without Googs?), I found this blog posting that outlined the exact versions of the plugins you’ll need to get backups working with WordPress 2.0.4. In any case, I’ll walk you through the procedure I followed to get my WordPress installation backed up.

The setup

If you’ve somehow managed to SSH/telnet into your hosting account, you can issue the wget command to retrieve the two WordPress plugins. I highly suggest creating a temp directory, then cd’ing into the directory to execute these commands.

wget URL/TO/LATEST/WP-DB-BACKUP.zip
wget URL/TO/LATEST/WP-CRON.zip

Note: I didn’t include the URL’s since they are likely to change depending on when this post is read.

The install

After downloading these two files, you’ll need to move them into your WordPress plugins directory using the following command:

mv *.zip WP_INSTALL_DIRECTORY/wp-content/plugins

Cd into the directory and extract the two zip files using the following commands:

unzip LATEST_WP-CRON.zip
unzip LATEST_WP-DB-BACKUP.zip

Once these files are unzipped, click here to find the files you need to overwrite. This can be done by cutting and pasting the file contents from the blog over to the same files on your hosted account. You can even delete the files on your hosted site and create a new file with the contents from the aforementioned blog.

You pick your poison.

Once installed, login to your WordPress admin site and find the Plugins link atop the administration panel. You’ll have to ‘activate‘ your plugins via the administration panel. Once you’ve activated both WP-Cron and WP-DB Backup, click on the Manage link atop the administration panel. In the sub panel, you’ll find a new link that reads ‘Backup’.

The config

I highly, highly recommend that before setting up your backup to execute nightly, immediately test to see if you can backup your current WordPress installation. This can be done by selecting the ‘Email backup to:’ option (fill in an email address) and pressing the Backup button. Once you know that the backups are working (check your email address)… fill in the Scheduled backup section by selecting Daily and filling in an email address. Make sure to add any extra tables (if there are any) that you would like to backup.

You’re probably wondering what the Gmail account was for. Since Google mail offers a sweet 2 GBs of free storage, create an appropriately named dummy Gmail account such as ‘my-blog-backup@gmail.com’ and send all your backups to this address. Periodically, download the backups to your home machine by accessing Gmail via POP, but also leave a few backups sitting in the actual account. If you ever approach the 2 GB storage limit, log into your account via the web and delete the extraneous backups.

Voila! Blog on!

Now bloggers can rest assured that Googs, your hosting provider, and your local machine will have a copy of your current blog.

Tagged: , , , , , , , , , , .


Google Apps

I’ve finally decided to switch my mail server over to Google Apps. Since I have all of ‘1‘ user on my domain – I figure it should be now or never. Seeing how my last Google experience was FUBAR’ed, I figured this would be another annoyingly painful process.

Yet, to my amazement, it was quite simple.

After signing up my domain, the first thing I had to do was verify that I was the owner. The verification process consisted of uploading a unique html document to the root of my domain. Contained within the document was a unique key that identified to Googs that I was indeed the domain’s owner. After FTP’ing the html document, I went back to Google Apps and indicated that I was ready for the verification process. Once verified, I was allowed to move my mail servers over.

Since my email was hosted on Dreamhost, in order to prevent an interruption in email service, the first thing I did was to create the exact same accounts I had on Dreamhost over on Google’s servers. That way, when my mail servers were finally switched over, all emails in transit would still be delivered to the right inbox.

To actually change my mail servers, I had to edit my Dreamhost MX record to point to Google App’s mail server. Googs even provided Dreamhost specific instructions to accomplish this task.

Nice.

After changing the MX record, I configured each account to allow POP access. Since I’m a Thunderbird user, I followed these instructions. After configuring Thunderbird, I clicked on ‘Get All Mail‘ and…

Voila!

On a difficulty scale of 1-10, with 10 being the most difficult, I’d rate this process of moving your domain over to Google Apps a 3. Not only that, but there were quite a number of other features like enabling Google Calendar, Google Start Page, and Google Talk for my domain.

Good job Googs!

Tagged: , , , , , , , , , , .


Powered by Wordpress. Stalk me.