Ryan Kanno: The diary of an Enginerd in Hawaii

Update

If you like to stay on the edge, check out my latest post describing how to use Capistrano 2.5 to deploy Rails 2.3 to Webfaction!

There’s nothing I love more than sweet automation.

After spending the better part of an hour searching the great Googs, there was only a single blog I could find describing how to use Capistrano to deploy to WebFaction. Unfortunately, Justin was describing a Capistrano 1.4 deployment. I found a few posts on the WebFaction forums, but nothing concrete. So after a few hours fiddling with the technology, here’s how I configured my Rails 2.1.1 project to use Capistrano 2.5 to deploy to WebFaction.

Assumptions

Before getting started, I’m going to assume the following:

• I’m assuming you’ve already used the one-click WebFaction goodness to create a brand new Rails application in ~/webapps/<application_name>. If you don’t know what I’m referring to, make sure to check out the Rails and Typo Demo screencast. Make sure you have a domain, application, and website configured.
• I’m also going to assume that your nifty Rails application is safely stored away in either a Subversion or Git repository and you’ve frozen Rails in your application.
• Finally, I’m going to assume you setup your database via WebFaction’s control panel.

Installing Capistrano

The very first thing you have to do is install Capistrano on your local machine by issuing the following command:

$ gem install -y capistrano

After installing Capistrano, the first thing you have to do is to “capify” your local Rails project. Change into your project’s root directory and issue the following command:

$ capify .

This configures your Rails project to play nicely with Capistrano. Two files should’ve been created; Capfile in the project root and config/deploy.rb. The deploy.rb file contains the Rails project application-specific deployment configuration.

Configuring WebFaction

Jumping back to WebFaction, I followed a few of the steps in Justin’s blog. First thing’s first, ssh into your WebFaction account and create a directory called webapps-releases in your home directory. This directory is where we’re going to deploy the application to.

Since you’ve already configured a Rails application at ~/webapps/<application_name>, change into that directory. You should see a standard Rails project with the exception of an extra file called autostart.cgi. Remove everything in the directory except the autostart.cgi file by issuing the following commands:

$ cd ~/webapps/<application_name>
$ mv autostart.cgi ~/
$ rm -rf *
$ mv ~/autostart.cgi .

Once the directory is clear, create a symlink to the log directory that will be in the webapps-releases directory we created earlier.

$ ln -s ~/webapps-releases/<application_name>/shared/log ~/webapps/<application_name>/log

Note: I’m assuming here that the WebFaction app and the Rails application have identical names.

Next, open up your favorite editor of choice (*cough*Vi*cough*) and edit the autostart.cgi file. Jump to the end of the file and comment out the following line:

1
2

 
# os.system('/usr/local/bin/mongrel_rails start -d -e production -P /home/<webfaction_username>/webapps/<webfaction_app_name>/log/mongrel.pid -p <port>')

and right below it, cut and paste the following:

1
2

 
  os.system('/usr/local/bin/mongrel_rails start -c /home/<webfaction_username>/webapps-releases/<webfaction_app_name>/current -d -e production -P /home/<webfaction_username>/webapps/<webfaction_app_name>/log/mongrel.pid -p <port>')

Creating your custom deploy.rb

After configuring WebFaction, we have to configure the Capistrano application deployment configuration. On your local machine, find the file config/deploy.rb and replace it with the one below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

 
set :webfaction_username, "<webfaction_username>"
set :webfaction_db_type, "<webfaction_db_type>"
set :webfaction_db, "<webfaction_db>"
set :webfaction_db_username, "<webfaction_db_username>"
set :webfaction_port, "<webfaction_port (get from autostart.cgi)>"
set :database_yml_template, "database.example.yml"
 
set :application, "test"
set :deploy_to, "/home/#{webfaction_username}/webapps-releases/#{application}"
 
set :scm, :subversion
set :scm_user, "<scm_username>"
set :scm_password, Proc.new { Capistrano::CLI.password_prompt("Subversion password for #{scm_user}: ") }
set :repository, Proc.new { "--username #{scm_user} --password #{scm_password} --no-auth-cache <http://path/to/your/svn/goes/here/>"} 
 
set :user, "#{webfaction_username}"
set :use_sudo, false 
 
set :domain, "<webfaction_domain>"
 
role :app, domain
role :web, domain
role :db,  domain, :primary => true
 
desc "Symlink public to what webfaction expects the webroot to be"
task :after_symlink, :roles => :web do
  run "ln -nfs #{release_path}/public /home/#{webfaction_username}/webapps/#{application}/"
end
 
namespace :deploy do
 
  # Taken from http://jonathan.tron.name/2006/07/15/capistrano-password-prompt-tips 
  # Thanks Jonathan! :)
  desc "Creates the database configuration on the fly"
  task :create_database_configuration, :roles => :app do
    require "yaml"
    set :production_db_password, proc { Capistrano::CLI.password_prompt("Remote production database password: ") }
 
    db_config = YAML::load_file("config/#{database_yml_template}")
    db_config.delete('test')
    db_config.delete('development')
 
    db_config['production']['adapter'] = "#{webfaction_db_type}"
    db_config['production']['database'] = "#{webfaction_db}"
    db_config['production']['username'] = "#{webfaction_db_username}"
    db_config['production']['password'] = production_db_password
    db_config['production']['host'] = "localhost"
 
    put YAML::dump(db_config), "#{release_path}/config/database.yml", :mode => 0664
  end
 
  after "deploy:update_code", "deploy:create_database_configuration"
 
  desc "Redefine deploy:start"
  task :start, :roles => :app do
    invoke_command "/usr/local/bin/mongrel_rails start -c #{deploy_to}/current -d -e production -P /home/#{webfaction_username}/webapps/#{application}/log/mongrel.pid -p #{webfaction_port}", :via => run_method
  end
 
  desc "Redefine deploy:restart"
  task :restart, :roles => :app do
    invoke_command "/usr/local/bin/mongrel_rails restart -c #{deploy_to}/current -P /home/#{webfaction_username}/webapps/#{application}/log/mongrel.pid", :via => run_method
  end
 
  desc "Redefine deploy:stop"
  task :stop, :roles => :app do
    invoke_command "/usr/local/bin/mongrel_rails stop -c #{deploy_to}/current -P /home/#{webfaction_username}/webapps/#{application}/log/mongrel.pid", :via => run_method
  end
end

Props out to Jonathan for the fantastic Capistrano tips!

After copying the deploy.rb file and editing the appropriate variables, run the following command in your Rails project’s root directory:

$ cap deploy:setup

This command creates the appropriate directory structure for Capistrano on the deployment server based upon values set in your deploy.rb. Next, run the following command to check your dependencies.

$ cap deploy:check

If everything is successful, you should see a message that reads something like…

You appear to have all necessary dependencies installed

Next, push your code out to the server using the following command:

$ cap deploy:update

Finally, to start up your application run the following Capistrano command:

$ cap deploy:start

Now, you should be able to run the standard Capistrano tasks to deploy your application to WebFaction!

Explanation

Most techies like to have an explanation of what’s going on with the Capistrano deploy.rb. I could probably write another blog about it, but I’m lazy (and pressed for time). The :create_database_configuration task basically writes the database.yml production configuration on the fly (courtesy of this blog posting).

The basic gyst of the rest of the script is that WebFaction is proxying a Mongrel instance. The Capistrano deploy.rb override the original deploy:start, deploy:stop, and deploy:restart tasks to run Mongrel commands that WebFaction can understand. Typically, the default Capistrano tasks run script/spin and reaper, but it was easier just to redefine the task. If anyone has any tips/suggestions to improve the script, I’m all ears!

Voila! (Enjoy)

Popularity: 28% [?]

Two events spurred me to write this blog.

First, my 2 year old “Subversion + Dreamhost + Post-Commit” blog still gets quite a number of hits. Second, after the latest Dreamhost outage move, I’m beginning to feel a little more vigilant about backing up my data.

As a standard disclaimer, if you’re not familiar with the Unix shell, I highly suggest you not try this unless under the supervision of someone who reads Perl books for fun. By accessing your Dreamhost shell, you can seriously f-up your account and I will not fix it for you. You have been warned. (Don’t you just love smileys?)

Setup

There are a few prerequisites to being able to back up your SVN repository.

1. First and foremost, you must have already installed a SVN repository into your Dreamhost account via the control panel.
2. Second, you must know how to SSH into your Dreamhost account. As a FYI, you sorta-kinda-need to know what that means in order to follow this tutorial.

Grabbing the backup script

Wait, you didn’t think I was writing my own right? In any case, if you actually installed/compiled Subversion on your own, it would’ve contained this file, hotbackup.py. Fortunately for us, Dreamhost has this file conveniently available at: /usr/bin/svn-hot-backup, but it’s an older version of the backup script. There are some subtle differences like being unable to pass in the number of backups you want the script to manage. Personally, I like to be on the edge, so let’s get the latest version. Execute the following commands from your home directory.

$ cd ~
$ mkdir scripts
$ cd scripts
$ wget http://svn.collab.net/repos/svn/trunk/tools/backup/hot-backup.py.in
$ mv hot-backup.py.in svn-hot-backup.py

The commands issued above created a directory called scripts in your home directory, switched into the directory, downloaded the latest hot-backup.py file from CollabNet, and renamed it to svn-hot-backup.py. Now that you have the file, you’ll need to make a few edits. Personally, I’m accustomed to vi, but pick your poison (pico, nano, text editor of your choice) and find these two values (they should be close to the top of the file in consecutive lines).

# Path to svnlook utility
svnlook = r"@SVN_BINDIR@/svnlook"

# Path to svnadmin utility
svnadmin = r"@SVN_BINDIR@/svnadmin"

and change them to the following:

# Path to svnlook utility
svnlook = r"/usr/bin/svnlook"

# Path to svnadmin utility
svnadmin = r"/usr/bin/svnadmin"

(If you’re wondering, if and when you compile/install Subversion yourself, these two variables would have been automagically filled in for you.)

The python script we downloaded not only performs a hotcopy of your svn directory, but also can archive it and manage a set number of copies. Pretty neat right?

Preparing for the backups

Before you can actually back up your SVN repository, you’ll want to create a directory structure to manage your backups. Execute the following commands from your home directory.

$ cd ~
$ mkdir backup
$ cd backup
$ mkdir svn
$ cd ~/scripts

The commands issued above created a directory called backup in your home directory, switched into the directory, and created another directory called svn within the backup directory. We’ll be using this directory to store all your backups. Finally, we switched back into the scripts directory created in the previous steps. Now that we have the backup script and directory structure to manage the back ups, let’s test it out!

Before you can back up your repository, you’ll have to know the name of the Subversion repository you’re trying to back up. To find the name of your repository, you can either look in the svn directory in your home directory, or you can check out the ID value in your Subversion Goodies control panel. In any case, remember the name of your SVN repository and issue the following commands.

$ cd ~/scripts/
$ python2.4 svn-hot-backup.py --archive-type=zip --num-backups=10 ~/svn/REPOSITORY_NAME_HERE/ ~/backup/svn/

You should see the following in the console:

Beginning hot backup of '/home/USERNAME/svn/lkg/'.
Youngest revision is REVISION_NUMBER
Backing up repository to '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701'...
Done.
Archiving backup to '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701.zip'...
Archive created, removing backup '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701'...

Voila! (But there’s more)

Automating the backups

Now that you actually have the script backing up your SVN repository, let’s automate them! To do so, we’ll use the handy cron daemon. Cron has similarities to the Windows task scheduler in that it provides a service that enables a user to execute commands at a specified date/time or set intervals. To tell cron the tasks you want to execute, you’ll need to load a configuration file called a crontab. You can read more about it here and here. In any case, here’s what my crontab configuration file looks like.

MAILTO=ryankanno@CHANGE_TO_YOUR_EMAIL.com
# minute (0-59),
# |      hour (0-23),
# |      |       day of the month (1-31),
# |      |       |       month of the year (1-12),
# |      |       |       |       day of the week (0-6 with 0=Sunday).
# |      |       |       |       |       commands
  0      0       *       *       *      /usr/bin/python2.4 /home/USERNAME/scripts/svn-hot-backup.py --archive-type=zip --num-backups=10 /home/USERNAME/svn/REPOSITORY_NAME/ /home/USERNAME/backup/svn/

Create a file in your scripts directory called svn_backup_once_a_day.cron and copy the contents above into your file. I’ve setup my crontab to backup my svn repository once a day.

Once you have this file called svn_backup_once_a_day.cron in your scripts directory, load the file into your crontab by issuing the following command:

$ crontab svn_backup_once_a_day.cron

As a FYI, this will replace your old crontab. If you have other items already running on cron, it’s a good idea to list them via the crontab -l command first. If you want to make sure that your cron will run, you can test it out by setting the values in the crontab to the time you want it to run. I’ll leave this as an exercise to the reader.

Storing your backups

Though out of scope of this blog, you’ll still have to store your backups somewhere. Please just don’t leave them in your Dreamhost account. Your best bet is probably to get an Amazon S3 account and store your backups there. Personally, I like to run another script immediately after the hotcopy finishes that pushes the backup to my S3 account. Other options include scp/sftp’ing the backups to your home machine. Here’s a link to read more about that option.

Voila! Enjoy!

Popularity: 43% [?]

This blog is for all my Hawaii television addicts.

Since I rarely have time to watch live television, my Oceanic Time Warner DVR is constantly filled to max capacity. This means I’m always battling my inner demons on what shows I have to erase… Rock of Love, A Shot at Love, Flavor of Love… you know, all the good stuff. To solve my problem, I’ve finally decided to invest the $150 to upgrade my DVR and increase its total number of recording hours.

Luckily for you, I’ll walk you through the steps to upgrade your own DVR!

As a standard disclaimer, if you attempt to upgrade your own DVR and f-it up, I can’t and won’t fix it. So… if technology scares you, please parents, do not try this unless supervised by your technology-oriented youngster. If you don’t understand what SATA, external enclosures, or hard drives mean, do not, and I repeat do not try this at home!

The setup

Before you can upgrade your DVR, you’ll need to make sure that you have the Scientific Atlanta Explorer 8300HD. Just match what your DVR looks like to the one in the picture. It’s not that hard. This is what mine looks like: the front and the back. I do know for a fact that Oceanic has a few versions of their cable boxes out in the wild. I’m pretty sure you can upgrade (some of) the other models as well, but I’ve personally only upgraded the 8300HD. So if you want to be ballsy and upgrade a different cable box, feel totally free – just be warned that this guide won’t apply to you. I’m not even sure if you can still turn in your old cable box because of the demand for HDTV in Hawaii, but calling up Oceanic can’t hurt.

Aside from owning an 8300HD, you’ll need three additional components to make this upgrade work. I’ve included links to where I purchased the following items. Fear not, I don’t make any commissions on these links so feel free to buy these products from anywhere you see fit.

• 1 external SATA enclosure (I used a KingWin from Fry’s for 30 bucks.)
• 1 SATA/IDE harddrive (I used a Maxtor 500 GB for about a 100.)
• 1 SATA to eSATA cable (I used a 6 ft cable from Monoprice.com for 5 bucks.)

Here are a few pictures of the aforementioned items.

The results

First, make sure your 8300HD is turned off. Place the hard drive into the external enclosure. Next, after connecting the external SATA enclosure to the 8300HD (with the SATA to eSATA cable), power the external hard drive before turning the DVR box back on. Note, it’s extremely important that the external SATA enclosure be turned on prior to the cable box being powered on. Once booted, the 8300HD should recognize a new, external data source and prompt you to format the new drive. The following message should appear:

Once formatted, you should see a success message:

Voila! DVR Upgraded!

The benes

There are numerous benefits to increasing your DVR’s total recording time.

• No more having to rush home because you forgot the DVR is full.
• No more making those life-altering decisions about what movies to delete.
• Being able to store almost a year’s worth of reality crap is fun!

Of course, there’s the almost 4X increase in the DVR’s recording time as you can see by the following before and after pictures. Not bad!

The cons

There’s no such thing as a free pass in life… so here are a few of the cons.

• As I wrote earlier, the external hard drive needs to be powered on before your cable box. This means one of two things. Either you always turn the external drive on first or leave it on permanently. Since I know I could never remember to do the former, I’ve decided to leave the device on permanently – meaning a slightly larger electricity bill. As someone trying to get off the grid, that makes me sad.
• You can’t rip the recorded video off the external hard drive. Unfortunately, the data is encrypted. Unless you’re a cryptographic expert, worked on the 8300HD, or have a few Beowulf clusters, deal with it. You won’t be able to share your recordings.
• $150 bucks is a lot to spend on easing one’s mind, but I think it’s money well spent considering the prices here and here.

Some linkage

Of course I couldn’t have upgraded my DVR without the Internet. Here’s a link to the forums and guides I read to assist me along the way. Check them out, some of them are quite interesting.

• Scientific Atlanta 8300HD User Guide
• AVSForum’s SA 8300 HD Tips & Tricks — SARA thread
• Scientific Atlanta 8300HD SARA eSATA Database

Finally, check out my flickr set if you need to see any more pictures!

Enjoy!

Popularity: 38% [?]

Since I actually used this method to reduce the number of Update:”explicit” SQL calls made in WeGoEat, I figured I’d write a little blog explaining the context in which it was used, and maybe, just maybe, it’ll help shed some light on how others can take advantage of this neat little function.

Background

As a Django “proof-of-concept”, I’m working on a local restaurant review site for my home state of Hawai`i. (I actually just released it yesterday). For each restaurant, I want to be able to calculate the average of all reviews and display this listing in a paginated view. (Yes, I do realize there’s no average rating, but that has to do with there being no users. ;P).

The Problem

Having a serious “wtf was I thinking moment”, I initially wrote a Restaurant model function that returned the average (review) rating for each restaurant instance. Little did I realize that when I actually displayed the restaurant’s average reviews, I would be making an additional SQL avg() call for every restaurant. Though I’m paging “n” records at a time, this function added an additional “n” SQL calls for every view that contained a restaurant listing, just to name a few.

In pseudo-code, my initial naive function resembled the following: (I’m sure we’re all guilty of writing something of the sort… ok, fine, I know I was. ;P)

1
2
3
4
5
6

     def get_average_review(self):
         query = 'QUERY TO GET AVERAGE (SELECT AVG(rating)...); (I have the query below)'
         # Get cursor from connection
        cursor = connection.cursor()
        cursor.execute(query)
        return cursor.fetchall()

Duh.

Here’s a picture of the number of queries it took:

The “extra()” solution

After profiling my application and realizing what a bone-headed mistake I made, I began researching the extra() Queryset modifier. Yes, I realize that these extra lookups aren’t the most portable and often violate the DRY principle, but it’ll probably suffice for most of all my personal projects.

Since I’m already retrieving a list of Restaurants and filtering them via letter, island, and what not, I figured I could add an average rating subquery. The entire call looks as such:

1
2
3
4
5
6
7

     restaurants = Restaurant.objects.filter(name__istartswith = letter).extra(
             select={'<strong>avg_rating</strong>': 'SELECT AVG(overall_rating) FROM restaurants_restaurant as res, reviews_review, django_content_type \
                                          WHERE restaurants_restaurant.id = res.id \
                                          AND res.id = reviews_review.object_id \
                                          AND reviews_review.content_type_id = django_content_type.id \
                                          AND django_content_type.model = \'restaurant\''},
                       )

As you can see, I’m exploiting the fact that restaurants_restaurant will be available from the Restaurant.objects.filter() call. (I know, I know… bad for portability).

But voila!

Now, in my templates, when I iterate over the restaurants, I can get issue the following:

1
2
3
4
5
6
7
8
9
10
11

{% for restaurant in restaurant_list %}
<tr>
    <td><a href="{{restaurant.get_absolute_url}}">{{ restaurant.name }}</a></td>
    <td>{% if restaurant.avg_rating %}
	   {% load show_stars %} 
           <span class="average-rating">
	   {% show_stars <strong>restaurant.avg_rating</strong> of 5 round to quarter %}
           &lt;/span&gt;
           {% endif %}&lt;/td&gt;
&lt;/tr&gt;
{% endfor %}

Notice how I used my show_stars template tag that I blogged about a few weeks ago to display the average restaurant rating. (Cheap shameless plug, but damn effective! ) I’d link to a page in action, but since I just opened up my site to a few select users, I’ll update this post when I actually have any reviews.

Oh, and before I forget, thanks to my co-worker Stephen for assisting me with my SQL issues!

Here’s a picture of the final result:

Note:

As an added bonus, I also realized a few other ’spots’ where the .extra() Queryset modifier would come in handy. Since I’m also using the wonderful django-voting application from Jonathan Buchanan, I came across this post about accessing a dictionary via a template in the Django-users Google Group.

Basically, I had come across the same issue as the poster. Since I allow users to vote on reviews (similar to Amazon, Yelp, etc.), I wanted to retrieve the score of each Review instance to display on a paginated listing of all Reviews. Using the same extra() modifier, I was able to inject the total number of votes and the score when I retrieved all Reviews as such:

Btw, I just injected most of the code from Jonathan’s template tag.

1
2
3
4
5
6
7
8
9
10
11

.extra(select={'total_votes': 'SELECT COUNT(vote) FROM votes as v, reviews_review as rev, django_content_type \
                                        WHERE reviews_review.id = rev.id \
                                        AND v.object_id = reviews_review.id \
                                        AND v.content_type_id = django_content_type.id \
                                        AND django_content_type.model = \'review\'', 
 
                                        'score': 'SELECT SUM(vote) FROM votes as v, reviews_review as rev, django_content_type \
                                        WHERE reviews_review.id = rev.id \
                                        AND v.object_id = reviews_review.id \
                                        AND v.content_type_id = django_content_type.id \
                                        AND django_content_type.model = \'review\''},)

Pretty neat right?

Now, when iterating through the reviews, I can use the following:

1
2
3
4
5
6
7
8
9
10
11
12
13

{% for review in object_list %}
	<tr>
		<td><a href="{{review.content_object.get_absolute_url}}">{{ review.content_object.name }}</a></td>
		<td><a href="{% url profile-detail username=review.user.username %}">{{ review.user.username }}</a></td>
		<td><nobr>{% load show_stars %}
			<span class="rating">{% show_stars review.overall_rating of 5 round to half %}</span>
			</nobr>
		</td>
		<td>"<span style="font-weight:bold; color:#092e20;">{{ review.get_recommendation_display }}</span>"</td>
		<td><span style="font-size:.875em;">{{ review.submit_date|timesince }} ago</span></td>
		<strong>&lt;td&gt;Total of {{ review.score|default:0 }} from {{ review.total_votes }} {{  review.total_votes|pluralize:"person,people" }}.&lt;/td&gt;</strong>
	&lt;/tr&gt;
{% endfor %}

Hope y’all learned something like I did! Oh, and before I forget my standard disclaimer, “since this is on my blog, feel free to take/use/steal/distribute/copy/modify any code you see fit, but if you find any bugs, have any comments, or think the code can be cleaner, I’d love to hear from you.”

Enjoy!

Popularity: 44% [?]

Update (7/6/2009):

Check the comments as Scott has provided an excellent link to a breakdown of the Merovingian’s various ties from the Matrix series!

I’ve always wondered how the Merovingian in the Matrix tied his tie. It’s bothered me for years… I guess it’s that attention to detail that keeps me up coding at 2am in the morning.

And then (as if through divine intervention) I was sent this video link on youtube…

Yes, I know… I’m a nerd. (But you still love me)

Update: As Rudy commented, in the Matrix, the tie is actually ‘backwards’. Apparently, there are nerdier people out there than me because I totally forgot about that detail.

Thanks Rudy!

In any case, a link was conveniently provided… (though, personally, I still like the other video.)

Popularity: 13% [?]