Ryan Kanno: The diary of an Enginerd in Hawaii

Everything you've ever thought, but never had the balls to say.

My LinkedIn Profile
Follow @ryankanno on Twitter
My Feed

Lessons Learned: Google App Engine + App Engine Patch + Django + Boto

Update:

Mitch Garnaat from CloudRight has pointed out that you can actually set the policy of the S3 file in the set_contents_from_file call instead of making another roundtrip request into S3 (and saving you some coin). Thanks Mitch!

Btw, I’m using App Engine Patch 1.0 and Boto 1.6a.


Sorry I haven’t updated my blog in a few weeks months, but I’ve been a little busy. With that said, along with Erlang, I’ve been playing around with Google App Engine, App Engine Patch (for Django support), and the Boto library (for Amazon S3 support). After not having touched Python code in a few months, I wanted to document some of my lessons learned to help over developers who may be in a similar boat.

Lessons learned

  • If you’re upgrading the App Engine Patch, make sure you don’t have the App Engine library installed in a hidden directory
  • Uploading bulk data changed ever so slightly
  • If you’re not running off of Boto’s trunk, you’ll need to patch your Boto installation to work with App Engine.

Make sure the App Engine library isn’t installed in a hidden directory

Apparently, Google’s SDK 1.1.9 doesn’t like to rely on files that won’t be uploaded with your application – and hidden directories are no longer uploaded. I was running into the dreaded purple-nurple screen of death. Thank goodness for this AppEngine Google Group post, but I’m still not even sure when this popped up considering Google’s articles still refer to this setup.

Bulk upload

Compared to the previous SDK I was playing around with, bulk uploading changed significantly. I recall having to patch Goog’s bulkupload.py file to get unicode support. However, their new remote api tool has definitely fixed this issue, so +1 for Googs. People are reporting that uploading unicode is still broken, but it’s not. Or at least it wasn’t for me. Second, if you’re like me and don’t read documentation, you’ll find out (the hard way) that the method signature to HandleEntity changed. Instead of accepting a datastore.Entity, it’s now expecting a db.Model object.

Note: When actually running the remote api tool, you’ll also want to make sure your PYTHONPATH includes your current project. (Another one-liner in the documentation. :P)

Integrating Boto + App Engine

I wasn’t running off of Boto’s trunk and I was getting an obscure type conversion error. Being too lazy to check out the source, I jumped to their issue tracker and found a patch (halfway down the page) by one of the App Engine Patch lead devs. Apply the patch and you’ll be on your way to uploading images/data from App Engine into Amazon S3! If you’re looking for example code, I’ve included a small snippet of what I tested.

1
2
3
4
5
6
7
8
9
10
11
    @staticmethod
    def upload_to_s3(original_filename, photo):
        """ Upload a photo file, storing its original name as metadata in an S3 bucket """
        connection = Connection(settings.AWS_ACCESS_KEY_ID, settings.AWS_SECRET_ACCESS_KEY)
        bucket = connection.get_bucket(settings.AWS_IMAGE_BUCKET_NAME)
        photo_uuid = str(uuid.uuid4())
        new_key = Key(bucket)
        new_key.key = photo_uuid
        new_key.set_metadata('original_filename', original_filename)
        new_key.set_contents_from_file(photo, policy='public-read')
        return photo_uuid

Note: I only tested the code above with small images ~300-500K in size and it seemed to work perfectly fine (with no load! :P). As always, feel free to use, steal, take, and/or copy anything on this blog. Hopefully somewhere, someone on the Interwebs will find these tips handy!

Enjoy!

Be Sociable, Share!
  1. Nice article. Just wanted to point out one small improvement you could make to your upload_to_s3 function. You can actually set the ACL when you call set_contents_from_file rather than making another round trip to S3 with a separate call. It would look like this:

    new_key.set_contents_from_file(photo, policy=’public-read’)

    Mitch

  2. @Mitch

    Thanks for the tip! I didn’t even look at the set_contents_from_file signature. Post will be updated shortly! :)

  3. Hi Ryan, the email provided on your contact page doesn’t work – error given is 50 550-5.1.1 The email account that you tried to reach does not exist. Ping me back, I’d like to send you a question about Subversion at Dreamhost. Thanks.

  4. @Ben

    Oops, you’re right! Darn migration to Google Apps. In any case, you can reach me at @localkinegrinds.com, or feel free to post your questions here. I also created that missing email address. Ahaha. Thanks for letting me know. :)

  5. Hey Ryan,
    I am pretty new to Amazon S3. Is there a mechanism provided by boto or some ither libraries which allow you to construct policies and signatures on the fly. I dont want to get into the hassle of encoding algorithms.

    Thanks,
    Amey Kanade
    Carnegie Mellon University

  6. @Amey – What type of signatures/policies are you referring to? You can use Boto to set the policies for the S3 buckets/files that you upload, etc.

  7. Yo…I almost read everythin what Boto provides…seems like as of now the boto libraries do not provide a easy way u can create policies an signatures for uploads.
    The way PUT based policies work and POST based policies work are different.(Check Amazon documentation).
    I created my own policy maker for POST based requests which are fairly simple. PUT policy making requires a lot other attributes.

  8. Hi Ryan..
    I wanted to Make policies and signatures. A library which does all the encoding and signing stuff. But anyways I figure it out after reading the stuff on Amazon.
    Amey Kanade
    EyeSee360 Inc
    Thanks.

  9. Hi -

    Boto does provide some methods in the boto.s3.connection module called build_post_args and build_post_policy to help construct the policies and actual form data required for direct POST upload to S3.

    Mitch

Please leave a reply »

Powered by Wordpress. Stalk me.