Having been a Capistrano user for a few years, I absolutely love sweet, sweet automation. A few months ago, I wrote a blog about how to use Capistrano to deploy to WebFaction. However, since that post, the winds of change have swept through the Rails and Capistrano community with the release of Rails 2.3 and Capistrano 2.5. WebFaction’s Ruby installation precludes you from taking advantage of these great updates, so I’ve come up with a little update to my previous post.

Learn to Change, Change to Learn

To be able to deploy Rails 2.3 with Capistrano 2.5, we’ll first have to jump through a few hoops. Nice, easy hoops, mind you, but still hoops nonetheless.

Your own Ruby installation

You’ll have to install a custom Ruby installation into your WebFaction home directory. By following Step 1 here, you’ll have a working Ruby 1.8.7 installation and RubyGem 1.3. You’ll need these if you want to deploy a Rails 2.3 application. Also, install a version of Mongrel tied to your Ruby installation.

Note: Typically, I like to install custom installations into ~/opt/local/lib/.

Edit Autostart.cgi

You’ll have to make a few edits to a few files from my previous blog post.

Next, open up your favorite editor of choice (*cough*vi*cough*) and edit the autostart.cgi file. Jump to the end of the file and comment out the following line from my previous blog:

1
2

 
# os.system('/usr/local/bin/mongrel_rails start -c /home/<webfaction_username>/webapps-releases/<webfaction_app_name>/current -d -e production -P /home/<webfaction_username>/webapps/<webfaction_app_name>/log/mongrel.pid -p <port>')

and change it to the following:

1
2

 
 os.system('/home/<webfaction_username>/opt/local/lib/ruby1.8/lib/ruby/gems/1.8/bin/mongrel_rails start -c /home/<webfaction_username>/webapps-releases/<webfaction_app_name>/current -d -e production -P /home/<webfaction_username>/webapps/<webfaction_app_name>/log/mongrel.pid -p <port>')

Notice: We’re using the path to Mongrel tied to our custom Ruby installation! This might be different in your installation!

Edit the Capfile

In my previous blog, we placed a custom deploy.rb into the config directory. With the latest version of Capistrano, you no longer need to do this. Rather, place the following into a file called Capfile in the root of your Rails project.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

 
load 'deploy'
 
set :webfaction_username, "<webfaction_username>"
set :webfaction_db_type, "<webfaction_db_type>"
set :webfaction_db, "<webfaction_db>"
set :webfaction_db_username, "<webfaction_db_username>"
set :webfaction_port, "<webfaction_port (get from autostart.cgi)>"
set :database_yml_template, "database.example.yml"
 
set :application, "test"
set :deploy_to, "/home/#{webfaction_username}/webapps-releases/#{application}"
 
set :scm, :subversion
set :scm_user, "<scm_username>"
set :scm_password, Proc.new { Capistrano::CLI.password_prompt("Subversion password for #{scm_user}: ") }
set :repository, Proc.new { "--username #{scm_user} --password #{scm_password} --no-auth-cache <http://path/to/your/svn/goes/here/>"} 
 
set :user, "#{webfaction_username}"
set :use_sudo, false 
 
set :domain, "<webfaction_domain>"
 
role :app, domain
role :web, domain
role :db,  domain, :primary => true
 
desc "Symlink public to what webfaction expects the webroot to be"
task :after_symlink, :roles => :web do
  run "ln -nfs #{release_path}/public /home/#{webfaction_username}/webapps/#{application}/"
end
 
namespace :deploy do
 
  # Taken from http://jonathan.tron.name/2006/07/15/capistrano-password-prompt-tips 
  # Thanks Jonathan! :)
  desc "Creates the database configuration on the fly"
  task :create_database_configuration, :roles => :app do
    require "yaml"
    set :production_db_password, proc { Capistrano::CLI.password_prompt("Remote production database password: ") }
 
    db_config = YAML::load_file("config/#{database_yml_template}")
    db_config.delete('test')
    db_config.delete('development')
 
    db_config['production']['adapter'] = "#{webfaction_db_type}"
    db_config['production']['database'] = "#{webfaction_db}"
    db_config['production']['username'] = "#{webfaction_db_username}"
    db_config['production']['password'] = production_db_password
    db_config['production']['host'] = "localhost"
 
    put YAML::dump(db_config), "#{release_path}/config/database.yml", :mode => 0664
  end
 
  after "deploy:update_code", "deploy:create_database_configuration"
 
  desc "Redefine deploy:start"
  task :start, :roles => :app do
    invoke_command "/opt/local/lib/ruby1.8/lib/ruby/gems/1.8/bin/mongrel_rails start -c #{deploy_to}/current -d -e production -P /home/#{webfaction_username}/webapps/#{application}/log/mongrel.pid -p #{webfaction_port}", :via => run_method
  end
 
  desc "Redefine deploy:restart"
  task :restart, :roles => :app do
    invoke_command "/opt/local/lib/ruby1.8/lib/ruby/gems/1.8/bin/mongrel_rails restart -c #{deploy_to}/current -P /home/#{webfaction_username}/webapps/#{application}/log/mongrel.pid", :via => run_method
  end
 
  desc "Redefine deploy:stop"
  task :stop, :roles => :app do
    invoke_command "/opt/local/lib/ruby1.8/lib/ruby/gems/1.8/bin/mongrel_rails stop -c #{deploy_to}/current -P /home/#{webfaction_username}/webapps/#{application}/log/mongrel.pid", :via => run_method
  end
end

Now, you should be able to run the standard Capistrano tasks to deploy your application to WebFaction with the latest version of Rails!

Explanation

Basically, the only thing I’ve done is shown you how to update your version of Ruby, RubyGems, Mongrel, and Capistrano to place nicely in WebFaction. This should allow you to run commands like ‘cap deploy:setup’, ‘cap deploy:update’ on your local machine and update your live code on WebFaction’s servers. Nothing too serious going on here! As always, feel free to use, steal, take, and/or copy anything on this blog. Hopefully somewhere, someone on the Interwebs will find these tips handy!

And if you find anything wrong with these scripts, don’t hesitate to leave a comment or email!

Voila! (Enjoy)

Mitch Garnaat from CloudRight has pointed out that you can actually set the policy of the S3 file in the set_contents_from_file call instead of making another roundtrip request into S3 (and saving you some coin). Thanks Mitch!

Btw, I’m using App Engine Patch 1.0 and Boto 1.6a.

Sorry I haven’t updated my blog in a few weeks months, but I’ve been a little busy. With that said, along with Erlang, I’ve been playing around with Google App Engine, App Engine Patch (for Django support), and the Boto library (for Amazon S3 support). After not having touched Python code in a few months, I wanted to document some of my lessons learned to help over developers who may be in a similar boat.

Lessons learned

• If you’re upgrading the App Engine Patch, make sure you don’t have the App Engine library installed in a hidden directory
• Uploading bulk data changed ever so slightly
• If you’re not running off of Boto’s trunk, you’ll need to patch your Boto installation to work with App Engine.

Make sure the App Engine library isn’t installed in a hidden directory

Apparently, Google’s SDK 1.1.9 doesn’t like to rely on files that won’t be uploaded with your application – and hidden directories are no longer uploaded. I was running into the dreaded purple-nurple screen of death. Thank goodness for this AppEngine Google Group post, but I’m still not even sure when this popped up considering Google’s articles still refer to this setup.

Bulk upload

Compared to the previous SDK I was playing around with, bulk uploading changed significantly. I recall having to patch Goog’s bulkupload.py file to get unicode support. However, their new remote api tool has definitely fixed this issue, so +1 for Googs. People are reporting that uploading unicode is still broken, but it’s not. Or at least it wasn’t for me. Second, if you’re like me and don’t read documentation, you’ll find out (the hard way) that the method signature to HandleEntity changed. Instead of accepting a datastore.Entity, it’s now expecting a db.Model object.

Note: When actually running the remote api tool, you’ll also want to make sure your PYTHONPATH includes your current project. (Another one-liner in the documentation. )

Integrating Boto + App Engine

I wasn’t running off of Boto’s trunk and I was getting an obscure type conversion error. Being too lazy to check out the source, I jumped to their issue tracker and found a patch (halfway down the page) by one of the App Engine Patch lead devs. Apply the patch and you’ll be on your way to uploading images/data from App Engine into Amazon S3! If you’re looking for example code, I’ve included a small snippet of what I tested.

1
2
3
4
5
6
7
8
9
10
11

    @staticmethod
    def upload_to_s3(original_filename, photo):
        """ Upload a photo file, storing its original name as metadata in an S3 bucket """
        connection = Connection(settings.AWS_ACCESS_KEY_ID, settings.AWS_SECRET_ACCESS_KEY)
        bucket = connection.get_bucket(settings.AWS_IMAGE_BUCKET_NAME)
        photo_uuid = str(uuid.uuid4())
        new_key = Key(bucket)
        new_key.key = photo_uuid
        new_key.set_metadata('original_filename', original_filename)
        new_key.set_contents_from_file(photo, policy='public-read')
        return photo_uuid

Note: I only tested the code above with small images ~300-500K in size and it seemed to work perfectly fine (with no load! ). As always, feel free to use, steal, take, and/or copy anything on this blog. Hopefully somewhere, someone on the Interwebs will find these tips handy!

Enjoy!

If you like to stay on the edge, check out my latest post describing how to use Capistrano 2.5 to deploy Rails 2.3 to Webfaction!

There’s nothing I love more than sweet automation.

After spending the better part of an hour searching the great Googs, there was only a single blog I could find describing how to use Capistrano to deploy to WebFaction. Unfortunately, Justin was describing a Capistrano 1.4 deployment. I found a few posts on the WebFaction forums, but nothing concrete. So after a few hours fiddling with the technology, here’s how I configured my Rails 2.1.1 project to use Capistrano 2.5 to deploy to WebFaction.

Assumptions

Before getting started, I’m going to assume the following:

• I’m assuming you’ve already used the one-click WebFaction goodness to create a brand new Rails application in ~/webapps/<application_name>. If you don’t know what I’m referring to, make sure to check out the Rails and Typo Demo screencast. Make sure you have a domain, application, and website configured.
• I’m also going to assume that your nifty Rails application is safely stored away in either a Subversion or Git repository and you’ve frozen Rails in your application.
• Finally, I’m going to assume you setup your database via WebFaction’s control panel.

Installing Capistrano

The very first thing you have to do is install Capistrano on your local machine by issuing the following command:

$ gem install -y capistrano

After installing Capistrano, the first thing you have to do is to “capify” your local Rails project. Change into your project’s root directory and issue the following command:

$ capify .

This configures your Rails project to play nicely with Capistrano. Two files should’ve been created; Capfile in the project root and config/deploy.rb. The deploy.rb file contains the Rails project application-specific deployment configuration.

Configuring WebFaction

Jumping back to WebFaction, I followed a few of the steps in Justin’s blog. First thing’s first, ssh into your WebFaction account and create a directory called webapps-releases in your home directory. This directory is where we’re going to deploy the application to.

Since you’ve already configured a Rails application at ~/webapps/<application_name>, change into that directory. You should see a standard Rails project with the exception of an extra file called autostart.cgi. Remove everything in the directory except the autostart.cgi file by issuing the following commands:

$ cd ~/webapps/<application_name>
$ mv autostart.cgi ~/
$ rm -rf *
$ mv ~/autostart.cgi .

Once the directory is clear, create a symlink to the log directory that will be in the webapps-releases directory we created earlier.

$ ln -s ~/webapps-releases/<application_name>/shared/log ~/webapps/<application_name>/log

Note: I’m assuming here that the WebFaction app and the Rails application have identical names.

Next, open up your favorite editor of choice (*cough*Vi*cough*) and edit the autostart.cgi file. Jump to the end of the file and comment out the following line:

1
2

 
# os.system('/usr/local/bin/mongrel_rails start -d -e production -P /home/<webfaction_username>/webapps/<webfaction_app_name>/log/mongrel.pid -p <port>')

and right below it, cut and paste the following:

1
2

 
  os.system('/usr/local/bin/mongrel_rails start -c /home/<webfaction_username>/webapps-releases/<webfaction_app_name>/current -d -e production -P /home/<webfaction_username>/webapps/<webfaction_app_name>/log/mongrel.pid -p <port>')

Creating your custom deploy.rb

After configuring WebFaction, we have to configure the Capistrano application deployment configuration. On your local machine, find the file config/deploy.rb and replace it with the one below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

 
set :webfaction_username, "<webfaction_username>"
set :webfaction_db_type, "<webfaction_db_type>"
set :webfaction_db, "<webfaction_db>"
set :webfaction_db_username, "<webfaction_db_username>"
set :webfaction_port, "<webfaction_port (get from autostart.cgi)>"
set :database_yml_template, "database.example.yml"
 
set :application, "test"
set :deploy_to, "/home/#{webfaction_username}/webapps-releases/#{application}"
 
set :scm, :subversion
set :scm_user, "<scm_username>"
set :scm_password, Proc.new { Capistrano::CLI.password_prompt("Subversion password for #{scm_user}: ") }
set :repository, Proc.new { "--username #{scm_user} --password #{scm_password} --no-auth-cache <http://path/to/your/svn/goes/here/>"} 
 
set :user, "#{webfaction_username}"
set :use_sudo, false 
 
set :domain, "<webfaction_domain>"
 
role :app, domain
role :web, domain
role :db,  domain, :primary => true
 
desc "Symlink public to what webfaction expects the webroot to be"
task :after_symlink, :roles => :web do
  run "ln -nfs #{release_path}/public /home/#{webfaction_username}/webapps/#{application}/"
end
 
namespace :deploy do
 
  # Taken from http://jonathan.tron.name/2006/07/15/capistrano-password-prompt-tips 
  # Thanks Jonathan! :)
  desc "Creates the database configuration on the fly"
  task :create_database_configuration, :roles => :app do
    require "yaml"
    set :production_db_password, proc { Capistrano::CLI.password_prompt("Remote production database password: ") }
 
    db_config = YAML::load_file("config/#{database_yml_template}")
    db_config.delete('test')
    db_config.delete('development')
 
    db_config['production']['adapter'] = "#{webfaction_db_type}"
    db_config['production']['database'] = "#{webfaction_db}"
    db_config['production']['username'] = "#{webfaction_db_username}"
    db_config['production']['password'] = production_db_password
    db_config['production']['host'] = "localhost"
 
    put YAML::dump(db_config), "#{release_path}/config/database.yml", :mode => 0664
  end
 
  after "deploy:update_code", "deploy:create_database_configuration"
 
  desc "Redefine deploy:start"
  task :start, :roles => :app do
    invoke_command "/usr/local/bin/mongrel_rails start -c #{deploy_to}/current -d -e production -P /home/#{webfaction_username}/webapps/#{application}/log/mongrel.pid -p #{webfaction_port}", :via => run_method
  end
 
  desc "Redefine deploy:restart"
  task :restart, :roles => :app do
    invoke_command "/usr/local/bin/mongrel_rails restart -c #{deploy_to}/current -P /home/#{webfaction_username}/webapps/#{application}/log/mongrel.pid", :via => run_method
  end
 
  desc "Redefine deploy:stop"
  task :stop, :roles => :app do
    invoke_command "/usr/local/bin/mongrel_rails stop -c #{deploy_to}/current -P /home/#{webfaction_username}/webapps/#{application}/log/mongrel.pid", :via => run_method
  end
end

Props out to Jonathan for the fantastic Capistrano tips!

After copying the deploy.rb file and editing the appropriate variables, run the following command in your Rails project’s root directory:

$ cap deploy:setup

This command creates the appropriate directory structure for Capistrano on the deployment server based upon values set in your deploy.rb. Next, run the following command to check your dependencies.

$ cap deploy:check

If everything is successful, you should see a message that reads something like…

You appear to have all necessary dependencies installed

Next, push your code out to the server using the following command:

$ cap deploy:update

Finally, to start up your application run the following Capistrano command:

$ cap deploy:start

Now, you should be able to run the standard Capistrano tasks to deploy your application to WebFaction!

Explanation

Most techies like to have an explanation of what’s going on with the Capistrano deploy.rb. I could probably write another blog about it, but I’m lazy (and pressed for time). The :create_database_configuration task basically writes the database.yml production configuration on the fly (courtesy of this blog posting).

The basic gyst of the rest of the script is that WebFaction is proxying a Mongrel instance. The Capistrano deploy.rb override the original deploy:start, deploy:stop, and deploy:restart tasks to run Mongrel commands that WebFaction can understand. Typically, the default Capistrano tasks run script/spin and reaper, but it was easier just to redefine the task. If anyone has any tips/suggestions to improve the script, I’m all ears!

Voila! (Enjoy)

Python goodness

While introducing the concept of automation to a friend of mine, I came across a requirement to archive a series of URL’s on a daily basis. Luckily for me, the URL’s consisted primarily of plain text. Loading up VIM, I concocted this Python script in a few hours – most of which was spent searching Googs <3.

If you're looking for a true web crawler, this won't be for you - though loading up lxml/Beautiful Soup, cssutils, and a Javascript parser to determine what artifacts need to be downloaded shouldn’t be all that difficult…

But, I’ll leave that as an exercise for the reader (That’s you, btw!)

In any case, the following script crawls a URL and sends the page via Googs or Webfaction via SMTP-AUTH or via a plain SMTP server of your choosing. Sorta-kinda like having your own WayBackMachine. In any case, cut and paste the following into a neat file called web2email.py.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174

#! /usr/bin/env python2.5
# -*- coding: utf-8 -*-
#
# Copyright (c) 2008 Ryan Kanno (ryankanno@localkinegrinds.com)
# License: GNU GPLv3
 
import urllib2
 
import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEBase import MIMEBase
from email.MIMEText import MIMEText
from email.Utils import COMMASPACE, formatdate
from email import Encoders
import datetime
 
from optparse import OptionParser
import sys, logging
 
__doc__ = """
 
This script retrieves a URL and sends its contents via email to 
a list of recipients.  Typically, this script is run from a cron
job that sends emails to a Gmail account to archive the contents
of a URL.
 
Mail can be sent via normal or authenticated SMTP.  Tested using 
Gmail SMTP (authenticated), Webfaction SMTP (authenticated), and
localhost (normal).
 
Example:
 
Sends the contents of http://www.espn.com to friend@domain.com using your Gmail settings
 
    python web2email.py -u gmail_username \
                        -p gmail_password \
                        -f gmail_username@gmail.com \
                        -r friend@domain.com http://www.espn.com
 
Sends the contents of http://www.espn.com to friend@domain.com using your Webfaction settings
 
    python web2email.py -u webfaction_username \
                        -p webfaction_password \
                        -f webfaction_account@webfaction_domain.com \
                        -s smtp.webfaction.com \
                        -r friend@domain.com http://www.espn.com
 
Sends the contents of http://www.espn.com to friend@domain.com using your local settings
 
    python web2email.py -f your_email@domain.com \
                        -s localhost \
                        --port 25 \
                        -r friend@domain.com http://www.espn.com
"""
 
__author__  = "ryankanno@localkinegrinds.com"
__url__     = "http://blog.localkinegrinds.com"
__version__ = "0.1"
 
USAGE = "usage: %prog [options] url" 
DESC  = __doc__.split('\n\n')[0]
 
def configure_logging(log_level, format='%(asctime)s %(levelname)s %(message)s'):
    logging.basicConfig(level=log_level, format=format)
 
def _validate_options_and_args(parser, options, args):
    logging.debug("Validating options and arguments.")
    if (len(args) != 1):
        parser.error("Incorrect number of arguments.  Script expects 1 (URL to backup), but received %i." % len(args))
        sys.exit(2) # Command line syntax error
    elif not options.recipients: 
        parser.error("You must include at least one recipient.")
        sys.exit(1) 
    elif (options.username and options.password is None) or (options.username is None and options.password is not None):
        parser.error("You must include both a username and password.")
        sys.exit(1) 
    elif not options.from_email:
        parser.error("You must include a valid from email address.")
        sys.exit(1) 
 
def getPage(url):
    logging.debug("Attempting to retrieve %s" % url)
    try:
        response = urllib2.urlopen(url)
        return response.read()
    except urllib2.HTTPError, e:
        logging.error("HTTPError (%s) occurred retrieving %s" % (e.code, url))
        sys.exit(1)
    except urllib2.URLError, e:
        logging.error("URLError (%s) occurred retrieving %s" % (e.reason, url))
        sys.exit(1)
 
def mail(send_from, send_to, subject, text, content_type, files=[], server='localhost', port=25, username=None, password=None):
 
    def _auth(server, port, username, password):
        logging.debug("Attempting to send email via %s:%i using the following credentials (%s:%s)." % (server, port, username, password))
        smtp = smtplib.SMTP(server, port) 
        smtp.ehlo()
        smtp.starttls()
        smtp.ehlo()
        smtp.login(username, password)
        smtp.sendmail(username, send_to, msg.as_string())
        smtp.close()
 
    def _unauth(server, port):
        logging.debug("Attempting to send email via %s:%i" % (server, port))
        smtp = smtplib.SMTP(server, port)
        smtp.sendmail(send_from, send_to, msg.as_string())
        smtp.close()
 
    assert type(send_to)==list
 
    msg=MIMEMultipart()
    msg['From'] = send_from
    msg['To'] = COMMASPACE.join(send_to)
    msg['Date'] = formatdate(localtime=True)
    msg['Subject'] = subject
 
    text = MIMEText(text)
    text.set_type(content_type)
    text.set_param('charset', 'UTF-8')
 
    msg.attach(text)
 
    for f in files:
        part = MIMEBase('application', "octet-stream")
        part.set_payload(open(file,"rb").read())
        Encoders.encode_base64(part)
        part.add_header('Content-Disposition', 'attachment; filename="%s"' % os.path.basename(f))
        msg.attach(part)
 
    if not username and not password:
        _unauth(server, port)
    else:
        _auth(server, port, username, password) 
 
def main():
    parser = OptionParser(usage=USAGE, description=DESC)
 
    parser.add_option("-u", "--username", dest="username", metavar="USER", help="Username to SMTP server")
    parser.add_option("-p", "--password", dest="password", metavar="PWD", help="Password to SMTP server")
    parser.add_option("-s", "--server", dest="server", metavar="SERVER", help="SMTP server (Defaults to Gmail)")
    parser.add_option("--port", dest="port", metavar="PORT", type="int", help="SMTP server port (Defaults to Gmail)")
    parser.add_option("-f", "--from", dest="from_email", metavar="FROM", help="From address")
    parser.add_option("-r", "--recipient", action="append", dest="recipients", metavar="RCPT", type="string", help="Email recipient")
    parser.add_option('-t', '--test', action="store_true", dest="test", metavar="TEST", help="Run tests")
    parser.add_option('-v', '--verbose', action='store_const', dest='log_level', const=logging.DEBUG, help='Verbose output')
    parser.set_defaults(server="smtp.gmail.com", port=587, test=False, log_level=logging.INFO)
    (options, args) = parser.parse_args()
 
    _validate_options_and_args(parser, options, args)
    configure_logging(options.log_level)
 
    if options.test:
        _test() # Too lazy to write a test for this script.  @TODO - use mocks 
 
    # Retrieve URL and return html
    html = getPage(args[0])
 
    # Send mail with returned html as body 
    mail(options.from_email, options.recipients, 
         '%s @ %s' % (args[0], (datetime.datetime.now().strftime("%A %B %d %I:%M:%S %p %Y"))), 
         html, 'text/html', 
         server=options.server, port=options.port, username=options.username, password=options.password)
 
    # Return with appropriate exit code
    sys.exit(0)
 
def _test():
    import doctest
    doctest.testmod(sys.modules[__name__])
 
if __name__ == '__main__':
    main()

All right stop, cron time! (imagine a 90′s pop song)

As an added bonus, you can install this script to run via cron so you’ll magically end up with webpages archived in your inbox! Neat. You can read my previous post on cron, or you can create the following crontab.

MAILTO=ryankanno@CHANGE_TO_YOUR_EMAIL.com
# minute (0-59),
# |      hour (0-23),
# |      |       day of the month (1-31),
# |      |       |       month of the year (1-12),
# |      |       |       |       day of the week (0-6 with 0=Sunday).
# |      |       |       |       |       commands
  0      0       *       *       *      /usr/bin/python2.5 /PATH/TO/web2email.py -u GMAIL_USER -p GMAIL_PWD -f FROM_USER -r RECIPIENT URL

As a side note, don’t forget double quotes around URL if there’s spaces!

I know, I know. The critics.

The critics will say that your Gmail username and password are in cleartext. I know. They are. So… I’m hoping that since you just need an archive of a publicly available URL on the Internets, the data doesn’t need to be super-duper-Fort-Knox-protected. If it does, this script isn’t for you. Oh, yeah, before I forget… here’s a hint… *cough*create another Google account*cough*. With that said, archive to your heart’s content!

Enjoy!

After searching the great Googs and reading here, here, and here about commenting out regions in your RHTML templates, I couldn’t find a (good) solution aside from the (<... if false ...>) paradigm. Using Rails 1.2.3 (I know, I know… we’re old school), to get block comments in Rails, the following worked for me, but unfortunately, still wasn’t recognized by NetBeans 6.0 (Boo!).

1
2
3
4
5
6
7
8
9
10
11
12
13

<table>
  ...
  <tbody>
    <tr id="<%= "photographer_#{photographer.id}"%>">
      <td><% 
=begin %>
<%= h photographer.first_name unless photographer.blank? %><% 
=end %>
      </td>
    </tr>
  </tbody>
  ...
</table>

Happy Coding!

I guess Googs finally caught on as their 1.1.2 installer works on Win2K! FTW!

Update

After finally getting time to play around with the Google App Engine Django helpers, here’s a few more steps to integrate nicely with the helper suite.

• Move the appengine installation from C:\AppEngine\ to where the Windows installer would have installed it to: C:\Program Files\Google\google_appengine (make sure to clean up your .pyc files)
• Add the following to your PYTHONPATH system variable: %APPENGINE%\;%APPENGINE%\lib;%APPENGINE%\lib\yaml\lib;%APPENGINE%\lib\webob;

After following the instructions, you should be good to go with Django + AppEngine! FTW! Whee.

I sense some pure, unadultered haterade. (j/k)

Since Python is one of those insert_any_synonym_for_fun languages that just works, here’s how to get the Google App Engine SDK working in Win2K.

• Download the Linux/Other platform package and unzip to somewhere neat.
• Add a System Environment variable called ‘APP_ENGINE_HOME’ that points to your App Engine installation. (Notice, I installed mine into C:\AppEngine)

  

• Add the System Environment variable to your System Path so the Windows shell can execute the included Python files.
• Make sure you have .py files associated with the python.exe executable located in your Python installation. (Check file types under folder options)
• Follow the tutorials: here and here, or learn with others – just to name a few.
• Oh, and before I forget, if you develop an application and realize that you can’t kill the development appserver (dev_appserver.py) by pressing Ctrl-C, I found a solution here. Basically, press Ctrl-C, hit the server with your browser one more time and voila, the development application server dies. Thanks Frank!

As an added bonus…

Last but not least…

Big ups to Mr. Fitz for solving all my Google App Engine issues and thanks to Mr. Harper for causing them.

Voila! (Enjoy)

Three things spurred me to release django-yui-layout-templates.

1. I’ve always wanted to see what GoogleCode offered in relation to SourceForge / RubyForge.
2. I’m so caught up in corporate America staring at Java / Ruby code all day, not only haven’t I blogged about anything Django related in quite a while, but it’s nice to get some commentary from the community, i.e. “your code sucks”. (Brings me back to reality)
3. I found myself using the same templates on a variety of projects and figured that I could do my part and help eliminate unncessary cruft/duplication.

So without further adieu, check out the project here. I know, I know – nothing revolutionary here, but I figure since Django is picking up some steam, these templates might help a Djangonaut get a head start on their next million dollar idea.

Voila! Enjoy!

First, my 2 year old “Subversion + Dreamhost + Post-Commit” blog still gets quite a number of hits. Second, after the latest Dreamhost outage move, I’m beginning to feel a little more vigilant about backing up my data.

As a standard disclaimer, if you’re not familiar with the Unix shell, I highly suggest you not try this unless under the supervision of someone who reads Perl books for fun. By accessing your Dreamhost shell, you can seriously f-up your account and I will not fix it for you. You have been warned. (Don’t you just love smileys?)

Setup

There are a few prerequisites to being able to back up your SVN repository.

1. First and foremost, you must have already installed a SVN repository into your Dreamhost account via the control panel.
2. Second, you must know how to SSH into your Dreamhost account. As a FYI, you sorta-kinda-need to know what that means in order to follow this tutorial.

Grabbing the backup script

Wait, you didn’t think I was writing my own right? In any case, if you actually installed/compiled Subversion on your own, it would’ve contained this file, hotbackup.py. Fortunately for us, Dreamhost has this file conveniently available at: /usr/bin/svn-hot-backup, but it’s an older version of the backup script. There are some subtle differences like being unable to pass in the number of backups you want the script to manage. Personally, I like to be on the edge, so let’s get the latest version. Execute the following commands from your home directory.

$ cd ~
$ mkdir scripts
$ cd scripts
$ wget http://svn.collab.net/repos/svn/trunk/tools/backup/hot-backup.py.in
$ mv hot-backup.py.in svn-hot-backup.py

The commands issued above created a directory called scripts in your home directory, switched into the directory, downloaded the latest hot-backup.py file from CollabNet, and renamed it to svn-hot-backup.py. Now that you have the file, you’ll need to make a few edits. Personally, I’m accustomed to vi, but pick your poison (pico, nano, text editor of your choice) and find these two values (they should be close to the top of the file in consecutive lines).

# Path to svnlook utility
svnlook = r"@SVN_BINDIR@/svnlook"

# Path to svnadmin utility
svnadmin = r"@SVN_BINDIR@/svnadmin"

and change them to the following:

# Path to svnlook utility
svnlook = r"/usr/bin/svnlook"

# Path to svnadmin utility
svnadmin = r"/usr/bin/svnadmin"

(If you’re wondering, if and when you compile/install Subversion yourself, these two variables would have been automagically filled in for you.)

The python script we downloaded not only performs a hotcopy of your svn directory, but also can archive it and manage a set number of copies. Pretty neat right?

Preparing for the backups

Before you can actually back up your SVN repository, you’ll want to create a directory structure to manage your backups. Execute the following commands from your home directory.

$ cd ~
$ mkdir backup
$ cd backup
$ mkdir svn
$ cd ~/scripts

The commands issued above created a directory called backup in your home directory, switched into the directory, and created another directory called svn within the backup directory. We’ll be using this directory to store all your backups. Finally, we switched back into the scripts directory created in the previous steps. Now that we have the backup script and directory structure to manage the back ups, let’s test it out!

Before you can back up your repository, you’ll have to know the name of the Subversion repository you’re trying to back up. To find the name of your repository, you can either look in the svn directory in your home directory, or you can check out the ID value in your Subversion Goodies control panel. In any case, remember the name of your SVN repository and issue the following commands.

$ cd ~/scripts/
$ python2.4 svn-hot-backup.py --archive-type=zip --num-backups=10 ~/svn/REPOSITORY_NAME_HERE/ ~/backup/svn/

You should see the following in the console:

Beginning hot backup of '/home/USERNAME/svn/lkg/'.
Youngest revision is REVISION_NUMBER
Backing up repository to '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701'...
Done.
Archiving backup to '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701.zip'...
Archive created, removing backup '/home/USERNAME/backup/svn/REPOSITORY_NAME_HERE-701'...

Voila! (But there’s more)

Automating the backups

Now that you actually have the script backing up your SVN repository, let’s automate them! To do so, we’ll use the handy cron daemon. Cron has similarities to the Windows task scheduler in that it provides a service that enables a user to execute commands at a specified date/time or set intervals. To tell cron the tasks you want to execute, you’ll need to load a configuration file called a crontab. You can read more about it here and here. In any case, here’s what my crontab configuration file looks like.

MAILTO=ryankanno@CHANGE_TO_YOUR_EMAIL.com
# minute (0-59),
# |      hour (0-23),
# |      |       day of the month (1-31),
# |      |       |       month of the year (1-12),
# |      |       |       |       day of the week (0-6 with 0=Sunday).
# |      |       |       |       |       commands
  0      0       *       *       *      /usr/bin/python2.4 /home/USERNAME/scripts/svn-hot-backup.py --archive-type=zip --num-backups=10 /home/USERNAME/svn/REPOSITORY_NAME/ /home/USERNAME/backup/svn/

Create a file in your scripts directory called svn_backup_once_a_day.cron and copy the contents above into your file. I’ve setup my crontab to backup my svn repository once a day.

Once you have this file called svn_backup_once_a_day.cron in your scripts directory, load the file into your crontab by issuing the following command:

$ crontab svn_backup_once_a_day.cron

As a FYI, this will replace your old crontab. If you have other items already running on cron, it’s a good idea to list them via the crontab -l command first. If you want to make sure that your cron will run, you can test it out by setting the values in the crontab to the time you want it to run. I’ll leave this as an exercise to the reader.

Storing your backups

Though out of scope of this blog, you’ll still have to store your backups somewhere. Please just don’t leave them in your Dreamhost account. Your best bet is probably to get an Amazon S3 account and store your backups there. Personally, I like to run another script immediately after the hotcopy finishes that pushes the backup to my S3 account. Other options include scp/sftp’ing the backups to your home machine. Here’s a link to read more about that option.

Voila! Enjoy!

Since I rarely have time to watch live television, my Oceanic Time Warner DVR is constantly filled to max capacity. This means I’m always battling my inner demons on what shows I have to erase… Rock of Love, A Shot at Love, Flavor of Love… you know, all the good stuff. To solve my problem, I’ve finally decided to invest the $150 to upgrade my DVR and increase its total number of recording hours.

Luckily for you, I’ll walk you through the steps to upgrade your own DVR!

As a standard disclaimer, if you attempt to upgrade your own DVR and f-it up, I can’t and won’t fix it. So… if technology scares you, please parents, do not try this unless supervised by your technology-oriented youngster. If you don’t understand what SATA, external enclosures, or hard drives mean, do not, and I repeat do not try this at home!

The setup

Before you can upgrade your DVR, you’ll need to make sure that you have the Scientific Atlanta Explorer 8300HD. Just match what your DVR looks like to the one in the picture. It’s not that hard. This is what mine looks like: the front and the back. I do know for a fact that Oceanic has a few versions of their cable boxes out in the wild. I’m pretty sure you can upgrade (some of) the other models as well, but I’ve personally only upgraded the 8300HD. So if you want to be ballsy and upgrade a different cable box, feel totally free – just be warned that this guide won’t apply to you. I’m not even sure if you can still turn in your old cable box because of the demand for HDTV in Hawaii, but calling up Oceanic can’t hurt.

Aside from owning an 8300HD, you’ll need three additional components to make this upgrade work. I’ve included links to where I purchased the following items. Fear not, I don’t make any commissions on these links so feel free to buy these products from anywhere you see fit.

• 1 external SATA enclosure (I used a KingWin from Fry’s for 30 bucks.)
• 1 SATA/IDE harddrive (I used a Maxtor 500 GB for about a 100.)
• 1 SATA to eSATA cable (I used a 6 ft cable from Monoprice.com for 5 bucks.)

Here are a few pictures of the aforementioned items.

The results

First, make sure your 8300HD is turned off. Place the hard drive into the external enclosure. Next, after connecting the external SATA enclosure to the 8300HD (with the SATA to eSATA cable), power the external hard drive before turning the DVR box back on. Note, it’s extremely important that the external SATA enclosure be turned on prior to the cable box being powered on. Once booted, the 8300HD should recognize a new, external data source and prompt you to format the new drive. The following message should appear:

Once formatted, you should see a success message:

Voila! DVR Upgraded!

The benes

There are numerous benefits to increasing your DVR’s total recording time.

• No more having to rush home because you forgot the DVR is full.
• No more making those life-altering decisions about what movies to delete.
• Being able to store almost a year’s worth of reality crap is fun!

Of course, there’s the almost 4X increase in the DVR’s recording time as you can see by the following before and after pictures. Not bad!

The cons

There’s no such thing as a free pass in life… so here are a few of the cons.

• As I wrote earlier, the external hard drive needs to be powered on before your cable box. This means one of two things. Either you always turn the external drive on first or leave it on permanently. Since I know I could never remember to do the former, I’ve decided to leave the device on permanently – meaning a slightly larger electricity bill. As someone trying to get off the grid, that makes me sad.
• You can’t rip the recorded video off the external hard drive. Unfortunately, the data is encrypted. Unless you’re a cryptographic expert, worked on the 8300HD, or have a few Beowulf clusters, deal with it. You won’t be able to share your recordings.
• $150 bucks is a lot to spend on easing one’s mind, but I think it’s money well spent considering the prices here and here.

Some linkage

Of course I couldn’t have upgraded my DVR without the Internet. Here’s a link to the forums and guides I read to assist me along the way. Check them out, some of them are quite interesting.

• Scientific Atlanta 8300HD User Guide
• AVSForum’s SA 8300 HD Tips & Tricks — SARA thread
• Scientific Atlanta 8300HD SARA eSATA Database

Finally, check out my flickr set if you need to see any more pictures!

Enjoy!

If you’re using jQuery 1.2.6, there is an updated version of the footerStickAlt javascript within the comments below. Thanks Luca!

It’s been tested in IE 5.5+ and Firefox 2.

*Note*: In this implementation, I don’t check for the margin on the ‘body’ element since we’ve set all our margins and padding on to 0. See the demos for details.

Requirements

• Yahoo! UI Grids [download]
• JQuery 1.2+ [uncompressed] [compressed]
• Place the Javascript below in a file of your choosing! Don’t forget to include it in your page!

$(document).ready(function() {
     if ($(window).height() > $(document).height()) {
          $('#bd').css({
               height: $('#bd').height() +
                         ($(window).height() -
                          $(document).height()) + 'px'
          });
     }
});

Check out the demos here: page with short content, page with long content!

(Notice the pages no longer depend upon jQuery Dimensions! Wheee!)

Enjoy!