Ryan Kanno: The diary of an Enginerd in Hawaii

Update:

Mitch Garnaat from CloudRight has pointed out that you can actually set the policy of the S3 file in the set_contents_from_file call instead of making another roundtrip request into S3 (and saving you some coin). Thanks Mitch!

Btw, I’m using App Engine Patch 1.0 and Boto 1.6a.

Sorry I haven’t updated my blog in a few weeks months, but I’ve been a little busy. With that said, along with Erlang, I’ve been playing around with Google App Engine, App Engine Patch (for Django support), and the Boto library (for Amazon S3 support). After not having touched Python code in a few months, I wanted to document some of my lessons learned to help over developers who may be in a similar boat.

Lessons learned

• If you’re upgrading the App Engine Patch, make sure you don’t have the App Engine library installed in a hidden directory
• Uploading bulk data changed ever so slightly
• If you’re not running off of Boto’s trunk, you’ll need to patch your Boto installation to work with App Engine.

Make sure the App Engine library isn’t installed in a hidden directory

Apparently, Google’s SDK 1.1.9 doesn’t like to rely on files that won’t be uploaded with your application – and hidden directories are no longer uploaded. I was running into the dreaded purple-nurple screen of death. Thank goodness for this AppEngine Google Group post, but I’m still not even sure when this popped up considering Google’s articles still refer to this setup.

Bulk upload

Compared to the previous SDK I was playing around with, bulk uploading changed significantly. I recall having to patch Goog’s bulkupload.py file to get unicode support. However, their new remote api tool has definitely fixed this issue, so +1 for Googs. People are reporting that uploading unicode is still broken, but it’s not. Or at least it wasn’t for me. Second, if you’re like me and don’t read documentation, you’ll find out (the hard way) that the method signature to HandleEntity changed. Instead of accepting a datastore.Entity, it’s now expecting a db.Model object.

Note: When actually running the remote api tool, you’ll also want to make sure your PYTHONPATH includes your current project. (Another one-liner in the documentation. )

Integrating Boto + App Engine

I wasn’t running off of Boto’s trunk and I was getting an obscure type conversion error. Being too lazy to check out the source, I jumped to their issue tracker and found a patch (halfway down the page) by one of the App Engine Patch lead devs. Apply the patch and you’ll be on your way to uploading images/data from App Engine into Amazon S3! If you’re looking for example code, I’ve included a small snippet of what I tested.

1
2
3
4
5
6
7
8
9
10
11

    @staticmethod
    def upload_to_s3(original_filename, photo):
        """ Upload a photo file, storing its original name as metadata in an S3 bucket """
        connection = Connection(settings.AWS_ACCESS_KEY_ID, settings.AWS_SECRET_ACCESS_KEY)
        bucket = connection.get_bucket(settings.AWS_IMAGE_BUCKET_NAME)
        photo_uuid = str(uuid.uuid4())
        new_key = Key(bucket)
        new_key.key = photo_uuid
        new_key.set_metadata('original_filename', original_filename)
        new_key.set_contents_from_file(photo, policy='public-read')
        return photo_uuid

Note: I only tested the code above with small images ~300-500K in size and it seemed to work perfectly fine (with no load! ). As always, feel free to use, steal, take, and/or copy anything on this blog. Hopefully somewhere, someone on the Interwebs will find these tips handy!

Enjoy!

Popularity: 22% [?]

Update : September 1, 2008

I guess Googs finally caught on as their 1.1.2 installer works on Win2K! FTW!

Update

After finally getting time to play around with the Google App Engine Django helpers, here’s a few more steps to integrate nicely with the helper suite.

• Move the appengine installation from C:\AppEngine\ to where the Windows installer would have installed it to: C:\Program Files\Google\google_appengine (make sure to clean up your .pyc files)
• Add the following to your PYTHONPATH system variable: %APPENGINE%\;%APPENGINE%\lib;%APPENGINE%\lib\yaml\lib;%APPENGINE%\lib\webob;

After following the instructions, you should be good to go with Django + AppEngine! FTW! Whee.

I sense some pure, unadultered haterade. (j/k)

Since Python is one of those insert_any_synonym_for_fun languages that just works, here’s how to get the Google App Engine SDK working in Win2K.

• Download the Linux/Other platform package and unzip to somewhere neat.
• Add a System Environment variable called ‘APP_ENGINE_HOME’ that points to your App Engine installation. (Notice, I installed mine into C:\AppEngine)

  

• Add the System Environment variable to your System Path so the Windows shell can execute the included Python files.
• Make sure you have .py files associated with the python.exe executable located in your Python installation. (Check file types under folder options)
• Follow the tutorials: here and here, or learn with others – just to name a few.
• Oh, and before I forget, if you develop an application and realize that you can’t kill the development appserver (dev_appserver.py) by pressing Ctrl-C, I found a solution here. Basically, press Ctrl-C, hit the server with your browser one more time and voila, the development application server dies. Thanks Frank!

As an added bonus…

Last but not least…

Big ups to Mr. Fitz for solving all my Google App Engine issues and thanks to Mr. Harper for causing them.

Voila! (Enjoy)

Popularity: 23% [?]

Morning Tutorial

Technical Management of Software Development with Uber Technical Lead @ Google, Alex Martelli

Mr. Martelli covered a gauntlet of topics pertaining to technical management including empowering your team, optimal team size, time management skills, and test-driven development. Even though his tutorial regurgitated a lot of the obvious, it’s nice to hear it being preached by important people at large corporations. What I found most useful in Mr. Martelli’s presentation was his extensive list of suggested reading material about management. I’ll post them when I get a chance to go over his slides.

Overall grade: A-

Afternoon Tutorial

Introduction to Rails with Glenn Vanderburg

Unlike the Ruby presentation from a day ago, the Introduction to Rails was everything I hoped the Ruby talk was going to be (and more). Even though Mr. Vanderburg only had three or so hours, he did a great job of covering quite a bit of material. Topics he covered included the model, the controller, filters, flash, caching, layouts, views, the various helpers, form handling, migration, routing, and testing (*whew*). Though he didn’t get into great detail on each subject, I felt that he covered enough to pique my interest. I actually went back to the hotel itching to code a Rails app.

Overall grade: A

Stay tuned for more updates from OSCON 2007!

Popularity: 7% [?]

I’ve found that many bloggers with Wordpress installations seem to overlook the simple task of backing up their data. Rather, they put their blogging fate into the hands of their trustworthy web-hosting provider. Not that I don’t trust my provider, but I like to sleep knowing full well that my data is resting in a few safe places – rather than putting my eggs all in one basket. So you can either learn the hard way like here, here, and here, or you can follow the simple backup recipe below.

Backing up your Wordpress blog calls for the installation of two very simple Wordpress plugins, WP-DB Backup and WP-Cron, and a dummy Gmail account. After going through the painstaking task of downloading, installing, and configuring each plugin on my hosting provider, I come to find out that other users have reportedthat these plugins have stopped working for Wordpress 2.0.4, the exact version I have installed on my hosting provider. After searching around a bit (how did people live without Googs?), I found this blog posting that outlined the exact versions of the plugins you’ll need to get backups working with Wordpress 2.0.4. In any case, I’ll walk you through the procedure I followed to get my Wordpress installation backed up.

The setup

If you’ve somehow managed to SSH/telnet into your hosting account, you can issue the wget command to retrieve the two Wordpress plugins. I highly suggest creating a temp directory, then cd’ing into the directory to execute these commands.

wget URL/TO/LATEST/WP-DB-BACKUP.zip
wget URL/TO/LATEST/WP-CRON.zip

Note: I didn’t include the URL’s since they are likely to change depending on when this post is read.

The install

After downloading these two files, you’ll need to move them into your Wordpress plugins directory using the following command:

mv *.zip WP_INSTALL_DIRECTORY/wp-content/plugins

Cd into the directory and extract the two zip files using the following commands:

unzip LATEST_WP-CRON.zip
unzip LATEST_WP-DB-BACKUP.zip

Once these files are unzipped, click here to find the files you need to overwrite. This can be done by cutting and pasting the file contents from the blog over to the same files on your hosted account. You can even delete the files on your hosted site and create a new file with the contents from the aforementioned blog.

You pick your poison.

Once installed, login to your Wordpress admin site and find the Plugins link atop the administration panel. You’ll have to ‘activate‘ your plugins via the administration panel. Once you’ve activated both WP-Cron and WP-DB Backup, click on the Manage link atop the administration panel. In the sub panel, you’ll find a new link that reads ‘Backup’.

The config

I highly, highly recommend that before setting up your backup to execute nightly, immediately test to see if you can backup your current Wordpress installation. This can be done by selecting the ‘Email backup to:’ option (fill in an email address) and pressing the Backup button. Once you know that the backups are working (check your email address)… fill in the Scheduled backup section by selecting Daily and filling in an email address. Make sure to add any extra tables (if there are any) that you would like to backup.

You’re probably wondering what the Gmail account was for. Since Google mail offers a sweet 2 GBs of free storage, create an appropriately named dummy Gmail account such as ‘my-blog-backup@gmail.com’ and send all your backups to this address. Periodically, download the backups to your home machine by accessing Gmail via POP, but also leave a few backups sitting in the actual account. If you ever approach the 2 GB storage limit, log into your account via the web and delete the extraneous backups.

Voila! Blog on!

Now bloggers can rest assured that Googs, your hosting provider, and your local machine will have a copy of your current blog.

Popularity: 11% [?]

I’ve finally decided to switch my mail server over to Google Apps. Since I have all of ‘1‘ user on my domain – I figure it should be now or never. Seeing how my last Google experience was FUBAR’ed, I figured this would be another annoyingly painful process.

Yet, to my amazement, it was quite simple.

After signing up my domain, the first thing I had to do was verify that I was the owner. The verification process consisted of uploading a unique html document to the root of my domain. Contained within the document was a unique key that identified to Googs that I was indeed the domain’s owner. After FTP’ing the html document, I went back to Google Apps and indicated that I was ready for the verification process. Once verified, I was allowed to move my mail servers over.

Since my email was hosted on Dreamhost, in order to prevent an interruption in email service, the first thing I did was to create the exact same accounts I had on Dreamhost over on Google’s servers. That way, when my mail servers were finally switched over, all emails in transit would still be delivered to the right inbox.

To actually change my mail servers, I had to edit my Dreamhost MX record to point to Google App’s mail server. Googs even provided Dreamhost specific instructions to accomplish this task.

Nice.

After changing the MX record, I configured each account to allow POP access. Since I’m a Thunderbird user, I followed these instructions. After configuring Thunderbird, I clicked on ‘Get All Mail‘ and…

Voila!

On a difficulty scale of 1-10, with 10 being the most difficult, I’d rate this process of moving your domain over to Google Apps a 3. Not only that, but there were quite a number of other features like enabling Google Calendar, Google Start Page, and Google Talk for my domain.

Good job Googs!

Popularity: 8% [?]