Ryan Kanno: The diary of an Enginerd in Hawaii

Update

After finally getting time to play around with the Google App Engine Django helpers, here’s a few more steps to integrate nicely with the helper suite.

• Move the appengine installation from C:\AppEngine\ to where the Windows installer would have installed it to: C:\Program Files\Google\google_appengine (make sure to clean up your .pyc files)
• Add the following to your PYTHONPATH system variable: %APPENGINE%\;%APPENGINE%\lib;%APPENGINE%\lib\yaml\lib;%APPENGINE%\lib\webob;

After following the instructions, you should be good to go with Django + AppEngine! FTW! Whee.

I sense some pure, unadultered haterade. (j/k)

Since Python is one of those insert_any_synonym_for_fun languages that just works, here’s how to get the Google App Engine SDK working in Win2K.

• Download the Linux/Other platform package and unzip to somewhere neat.
• Add a System Environment variable called ‘APP_ENGINE_HOME’ that points to your App Engine installation. (Notice, I installed mine into C:\AppEngine)

  

• Add the System Environment variable to your System Path so the Windows shell can execute the included Python files.
• Make sure you have .py files associated with the python.exe executable located in your Python installation. (Check file types under folder options)
• Follow the tutorials: here and here, or learn with others - just to name a few.
• Oh, and before I forget, if you develop an application and realize that you can’t kill the development appserver (dev_appserver.py) by pressing Ctrl-C, I found a solution here. Basically, press Ctrl-C, hit the server with your browser one more time and voila, the development application server dies. Thanks Frank!

As an added bonus…

Last but not least…

Big ups to Mr. Fitz for solving all my Google App Engine issues and thanks to Mr. Harper for causing them.

Voila! (Enjoy)

Three things spurred me to release django-yui-layout-templates.

1. I’ve always wanted to see what GoogleCode offered in relation to SourceForge / RubyForge.
2. I’m so caught up in corporate America staring at Java / Ruby code all day, not only haven’t I blogged about anything Django related in quite a while, but it’s nice to get some commentary from the community, i.e. “your code sucks”. (Brings me back to reality)
3. I found myself using the same templates on a variety of projects and figured that I could do my part and help eliminate unncessary cruft/duplication.

So without further adieu, check out the project here. I know, I know - nothing revolutionary here, but I figure since Django is picking up some steam, these templates might help a Djangonaut get a head start on their next million dollar idea.

Voila! Enjoy!

Two events spurred me to write this blog.

First, my 2 year old “Subversion + Dreamhost + Post-Commit” blog still gets quite a number of hits. Second, after the latest Dreamhost outage move, I’m beginning to feel a little more vigilant about backing up my data.

As a standard disclaimer, if you’re not familiar with the Unix shell, I highly suggest you not try this unless under the supervision of someone who reads Perl books for fun. By accessing your Dreamhost shell, you can seriously f-up your account and I will not fix it for you. You have been warned. (Don’t you just love smileys?)

Setup

There are a few prerequisites to being able to back up your SVN repository.

1. First and foremost, you must have already installed a SVN repository into your Dreamhost account via the control panel.
2. Second, you must know how to SSH into your Dreamhost account. As a FYI, you sorta-kinda-need to know what that means in order to follow this tutorial.

Grabbing the backup script

Wait, you didn’t think I was writing my own right? In any case, if you actually installed/compiled Subversion on your own, it would’ve contained this file, hotbackup.py. Fortunately for us, Dreamhost has this file conveniently available at: /usr/bin/svn-hot-backup, but it’s an older version of the backup script. There are some subtle differences like being unable to pass in the number of backups you want the script to manage. Personally, I like to be on the edge, so let’s get the latest version. Execute the following commands from your home directory.

$ cd ~
$ mkdir scripts
$ cd scripts
$ wget http://svn.collab.net/repos/svn/trunk/tools/backup/hot-backup.py.in
$ mv hot-backup.py.in svn-hot-backup.py

The commands issued above created a directory called scripts in your home directory, switched into the directory, downloaded the latest hot-backup.py file from CollabNet, and renamed it to svn-hot-backup.py. Now that you have the file, you’ll need to make a few edits. Personally, I’m accustomed to vi, but pick your poison (pico, nano, text editor of your choice) and find these two values (they should be close to the top of the file in consecutive lines).

# Path to svnlook utility
svnlook = r"@SVN_BINDIR@/svnlook"

# Path to svnadmin utility
svnadmin = r"@SVN_BINDIR@/svnadmin"

and change them to the following:

# Path to svnlook utility
svnlook = r"/usr/bin/svnlook"

# Path to svnadmin utility
svnadmin = r"/usr/bin/svnadmin"

(If you’re wondering, if and when you compile/install Subversion yourself, these two variables would have been automagically filled in for you.)

The python script we downloaded not only performs a hotcopy of your svn directory, but also can archive it and manage a set number of copies. Pretty neat right?

Preparing for the backups

Before you can actually back up your SVN repository, you’ll want to create a directory structure to manage your backups. Execute the following commands from your home directory.

$ cd ~
$ mkdir backup
$ cd backup
$ mkdir svn
$ cd ~/scripts

The commands issued above created a directory called backup in your home directory, switched into the directory, and created another directory called svn within the backup directory. We’ll be using this directory to store all your backups. Finally, we switched back into the scripts directory created in the previous steps. Now that we have the backup script and directory structure to manage the back ups, let’s test it out!

Before you can back up your repository, you’ll have to know the name of the Subversion repository you’re trying to back up. To find the name of your repository, you can either look in the svn directory in your home directory, or you can check out the ID value in your Subversion Goodies control panel. In any case, remember the name of your SVN repository and issue the following commands.

$ cd ~/scripts/
$ python2.4 svn-hot-backup.py --archive-type=zip --num-backups=10 ~/svn/REPOSITORY_NAME_HERE/ ~/backup/svn/

You should see the following in the console:

Beginning hot backup of '/home/USERNAME/svn/lkg/'.
Youngest revision is REVISION_NUMBER
Backing up repository to '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701'...
Done.
Archiving backup to '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701.zip'...
Archive created, removing backup '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701'...

Voila! (But there’s more)

Automating the backups

Now that you actually have the script backing up your SVN repository, let’s automate them! To do so, we’ll use the handy cron daemon. Cron has similarities to the Windows task scheduler in that it provides a service that enables a user to execute commands at a specified date/time or set intervals. To tell cron the tasks you want to execute, you’ll need to load a configuration file called a crontab. You can read more about it here and here. In any case, here’s what my crontab configuration file looks like.

MAILTO=ryankanno@CHANGE_TO_YOUR_EMAIL.com
# minute (0-59),
# |      hour (0-23),
# |      |       day of the month (1-31),
# |      |       |       month of the year (1-12),
# |      |       |       |       day of the week (0-6 with 0=Sunday).
# |      |       |       |       |       commands
  0      0       *       *       *      /usr/bin/python2.4 /home/USERNAME/scripts/svn-hot-backup.py --archive-type=zip --num-backups=10 /home/USERNAME/svn/REPOSITORY_NAME/ /home/USERNAME/backup/svn/

Create a file in your scripts directory called svn_backup_once_a_day.cron and copy the contents above into your file. I’ve setup my crontab to backup my svn repository once a day.

Once you have this file called svn_backup_once_a_day.cron in your scripts directory, load the file into your crontab by issuing the following command:

$ crontab svn_backup_once_a_day.cron

As a FYI, this will replace your old crontab. If you have other items already running on cron, it’s a good idea to list them via the crontab -l command first. If you want to make sure that your cron will run, you can test it out by setting the values in the crontab to the time you want it to run. I’ll leave this as an exercise to the reader.

Storing your backups

Though out of scope of this blog, you’ll still have to store your backups somewhere. Please just don’t leave them in your Dreamhost account. Your best bet is probably to get an Amazon S3 account and store your backups there. Personally, I like to run another script immediately after the hotcopy finishes that pushes the backup to my S3 account. Other options include scp/sftp’ing the backups to your home machine. Here’s a link to read more about that option.

Voila! Enjoy!

Since I actually used this method to reduce the number of Update:”explicit” SQL calls made in WeGoEat, I figured I’d write a little blog explaining the context in which it was used, and maybe, just maybe, it’ll help shed some light on how others can take advantage of this neat little function.

Background

As a Django “proof-of-concept”, I’m working on a local restaurant review site for my home state of Hawai`i. (I actually just released it yesterday). For each restaurant, I want to be able to calculate the average of all reviews and display this listing in a paginated view. (Yes, I do realize there’s no average rating, but that has to do with there being no users. ;P).

The Problem

Having a serious “wtf was I thinking moment”, I initially wrote a Restaurant model function that returned the average (review) rating for each restaurant instance. Little did I realize that when I actually displayed the restaurant’s average reviews, I would be making an additional SQL avg() call for every restaurant. Though I’m paging “n” records at a time, this function added an additional “n” SQL calls for every view that contained a restaurant listing, just to name a few.

In pseudo-code, my initial naive function resembled the following: (I’m sure we’re all guilty of writing something of the sort… ok, fine, I know I was. ;P)

1
2
3
4
5
6

     def get_average_review(self):
         query = 'QUERY TO GET AVERAGE (SELECT AVG(rating)...); (I have the query below)'
         # Get cursor from connection
        cursor = connection.cursor()
        cursor.execute(query)
        return cursor.fetchall()

Duh.

Here’s a picture of the number of queries it took:

The “extra()” solution

After profiling my application and realizing what a bone-headed mistake I made, I began researching the extra() Queryset modifier. Yes, I realize that these extra lookups aren’t the most portable and often violate the DRY principle, but it’ll probably suffice for most of all my personal projects.

Since I’m already retrieving a list of Restaurants and filtering them via letter, island, and what not, I figured I could add an average rating subquery. The entire call looks as such:

1
2
3
4
5
6
7

     restaurants = Restaurant.objects.filter(name__istartswith = letter).extra(
             select={'<strong>avg_rating</strong>': 'SELECT AVG(overall_rating) FROM restaurants_restaurant as res, reviews_review, django_content_type \
                                          WHERE restaurants_restaurant.id = res.id \
                                          AND res.id = reviews_review.object_id \
                                          AND reviews_review.content_type_id = django_content_type.id \
                                          AND django_content_type.model = \'restaurant\''},
                       )

As you can see, I’m exploiting the fact that restaurants_restaurant will be available from the Restaurant.objects.filter() call. (I know, I know… bad for portability).

But voila!

Now, in my templates, when I iterate over the restaurants, I can get issue the following:

1
2
3
4
5
6
7
8
9
10
11

{% for restaurant in restaurant_list %}
<tr>
    <td><a href="{{restaurant.get_absolute_url}}">{{ restaurant.name }}</a></td>
    <td>{% if restaurant.avg_rating %}
	   {% load show_stars %} 
           <span class="average-rating">
	   {% show_stars <strong>restaurant.avg_rating</strong> of 5 round to quarter %}
           &lt;/span&gt;
           {% endif %}&lt;/td&gt;
&lt;/tr&gt;
{% endfor %}

Notice how I used my show_stars template tag that I blogged about a few weeks ago to display the average restaurant rating. (Cheap shameless plug, but damn effective! :P) I’d link to a page in action, but since I just opened up my site to a few select users, I’ll update this post when I actually have any reviews.

Oh, and before I forget, thanks to my co-worker Stephen for assisting me with my SQL issues!

Here’s a picture of the final result:

Note:

As an added bonus, I also realized a few other ’spots’ where the .extra() Queryset modifier would come in handy. Since I’m also using the wonderful django-voting application from Jonathan Buchanan, I came across this post about accessing a dictionary via a template in the Django-users Google Group.

Basically, I had come across the same issue as the poster. Since I allow users to vote on reviews (similar to Amazon, Yelp, etc.), I wanted to retrieve the score of each Review instance to display on a paginated listing of all Reviews. Using the same extra() modifier, I was able to inject the total number of votes and the score when I retrieved all Reviews as such:

Btw, I just injected most of the code from Jonathan’s template tag.

1
2
3
4
5
6
7
8
9
10
11

.extra(select={'total_votes': 'SELECT COUNT(vote) FROM votes as v, reviews_review as rev, django_content_type \
                                        WHERE reviews_review.id = rev.id \
                                        AND v.object_id = reviews_review.id \
                                        AND v.content_type_id = django_content_type.id \
                                        AND django_content_type.model = \'review\'', 
 
                                        'score': 'SELECT SUM(vote) FROM votes as v, reviews_review as rev, django_content_type \
                                        WHERE reviews_review.id = rev.id \
                                        AND v.object_id = reviews_review.id \
                                        AND v.content_type_id = django_content_type.id \
                                        AND django_content_type.model = \'review\''},)

Pretty neat right?

Now, when iterating through the reviews, I can use the following:

1
2
3
4
5
6
7
8
9
10
11
12
13

{% for review in object_list %}
	<tr>
		<td><a href="{{review.content_object.get_absolute_url}}">{{ review.content_object.name }}</a></td>
		<td><a href="{% url profile-detail username=review.user.username %}">{{ review.user.username }}</a></td>
		<td><nobr>{% load show_stars %}
			<span class="rating">{% show_stars review.overall_rating of 5 round to half %}</span>
			</nobr>
		</td>
		<td>"<span style="font-weight:bold; color:#092e20;">{{ review.get_recommendation_display }}</span>"</td>
		<td><span style="font-size:.875em;">{{ review.submit_date|timesince }} ago</span></td>
		<strong>&lt;td&gt;Total of {{ review.score|default:0 }} from {{ review.total_votes }} {{  review.total_votes|pluralize:"person,people" }}.&lt;/td&gt;</strong>
	&lt;/tr&gt;
{% endfor %}

Hope y’all learned something like I did! Oh, and before I forget my standard disclaimer, “since this is on my blog, feel free to take/use/steal/distribute/copy/modify any code you see fit, but if you find any bugs, have any comments, or think the code can be cleaner, I’d love to hear from you.”

Enjoy!

Since my last post was quite popular (by my simple blogging standards), I’ve decided to post another Django snippet that I’ve used while coding WeGoEat. (I know, I know… one of these days I’ll finish. I’ve just been really, really lazy busy.)

After reading Leah’s blog (of Pownce fame) about “rounding to the nearest half”, I was immediately struck by two sobering realities:

1. I would need that exact same algorithm for my current site.
2. A few years out of grad school, and I couldn’t even begin to fathom a solution. Thank you, Internet.

In any case, I figured I’d wrap my solution into a template tag that people could use/steal/copy. Here’s a generated test page that illustrates what my template tag would display.

You can even customize the display a bit:

I know, I know… I can do a lot better. In any case, here’s the code snippet that I placed in a file called show_stars.py:

import math
 
from django.template import Library, Node, TemplateSyntaxError, VariableDoesNotExist, resolve_variable
from django.conf import settings
 
register = Library()
 
IMG_TEMPLATE = '<img src="%s" alt="%s"/>'
 
PATH_TO_WHOLE_STAR = IMG_TEMPLATE % (settings.MEDIA_URL + 'img/stars/star.png', "Whole Star")
PATH_TO_THREE_QUARTER_STAR = IMG_TEMPLATE % (settings.MEDIA_URL + 'img/stars/three-quarter.png', "3/4 Star")
PATH_TO_HALF_STAR = IMG_TEMPLATE % (settings.MEDIA_URL + 'img/stars/half.png', "1/2 Star")
PATH_TO_QUARTER_STAR = IMG_TEMPLATE % (settings.MEDIA_URL + 'img/stars/quarter.png', "1/4 Star")
PATH_TO_BLANK_STAR = IMG_TEMPLATE % (settings.MEDIA_URL + 'img/stars/blank.png', "Empty Star")
 
class ShowStarsNode(Node):
    """ Default rounding is to the whole unit """
    def __init__(self, context_var, total_stars, round_to):
        self.context_var = context_var
        self.total_stars = int(total_stars)
        self.round_to = round_to.lower()
 
    def render(self, context):
        try:
            stars = resolve_variable(self.context_var, context)
        except VariableDoesNotExist:
            return ''
 
        if self.round_to == "half":
            stars = round(stars*2)/2
        elif self.round_to == "quarter":
            stars = round(stars*4)/4
        else:
            stars = round(stars)
 
        fraction, integer = math.modf(stars)
        integer = int(integer)
        output = []
 
        for whole_star in range(integer):
            output.append(PATH_TO_WHOLE_STAR)
        if self.round_to == 'half' and fraction == .5:
            output.append(PATH_TO_HALF_STAR)
        elif self.round_to == 'quarter':
            if fraction == .25:
                output.append(PATH_TO_QUARTER_STAR)
            elif fraction == .5:
                output.append(PATH_TO_HALF_STAR)
            elif fraction == .75:
                output.append(PATH_TO_THREE_QUARTER_STAR)
 
        if fraction:
            integer += 1
 
        blanks = int(self.total_stars - integer)
 
        for blank_star in range(blanks):
            output.append(PATH_TO_BLANK_STAR)
 
        return "".join(output)
 
""" show_stars context_var of 5 round to half """
def do_show_stars(parser, token):
    args = token.contents.split()
    if len(args) != 7:
        raise TemplateSyntaxError('%s tag requires exactly six arguments' % args[0])
    if args[2] != 'of':
        raise TemplateSyntaxError("second argument to '%s' tag must be 'of'" % args[0])
    if args[4] != 'round':
        raise TemplateSyntaxError("fourth argument to '%s' tag must be 'round'" % args[0])
    if args[5] != 'to':
        raise TemplateSyntaxError("fourth argument to '%s' tag must be 'to'" % args[0])
    return ShowStarsNode(args[1], args[3], args[6])   
 
register.tag('show_stars', do_show_stars)

I’ve also wrapped up the star graphics I used. A big ups to the Silk Icons creator. For those of you that don’t care for .zip files, here’s all the star images I used:

To use the tag, it’ll look something like this in your templates:

{% load show_stars %}
     {% show_stars context_var of 5 round to half %}
{% endif %}

Note: You can change the following:

1. context_var is whatever var contains the value you want rounded (it was ‘value’ in the screenshots)
2. 5 can be any integer; this is the maximum number of stars to display. I should probably check to make sure you don’t put a number smaller than context_var, but I’ll leave that up to you.
3. half can be any of the following values: whole, half, or quarter
4. Make sure to edit the paths to the star files in the templatetag file

And since this is on my blog, feel totally free to take/use/steal/distribute/copy/modify any code you see fit. All I ask is if you find any bugs, have any comments, or can think of ways to make my ugly code cleaner - I’d love to hear from you.

Enjoy!