Mitch Garnaat from CloudRight has pointed out that you can actually set the policy of the S3 file in the set_contents_from_file call instead of making another roundtrip request into S3 (and saving you some coin). Thanks Mitch!
Btw, I’m using App Engine Patch 1.0 and Boto 1.6a.
Sorry I haven’t updated my blog in a few
weeks months, but I’ve been a little busy. With that said, along with Erlang, I’ve been playing around with Google App Engine, App Engine Patch (for Django support), and the Boto library (for Amazon S3 support). After not having touched Python code in a few months, I wanted to document some of my lessons learned to help over developers who may be in a similar boat.
- If you’re upgrading the App Engine Patch, make sure you don’t have the App Engine library installed in a hidden directory
- Uploading bulk data changed ever so slightly
- If you’re not running off of Boto’s trunk, you’ll need to patch your Boto installation to work with App Engine.
Make sure the App Engine library isn’t installed in a hidden directory
Apparently, Google’s SDK 1.1.9 doesn’t like to rely on files that won’t be uploaded with your application – and hidden directories are no longer uploaded. I was running into the dreaded purple-nurple screen of death. Thank goodness for this AppEngine Google Group post, but I’m still not even sure when this popped up considering Google’s articles still refer to this setup.
Compared to the previous SDK I was playing around with, bulk uploading changed significantly. I recall having to patch Goog’s bulkupload.py file to get unicode support. However, their new remote api tool has definitely fixed this issue, so +1 for Googs. People are reporting that uploading unicode is still broken, but it’s not. Or at least it wasn’t for me. Second, if you’re like me and don’t read documentation, you’ll find out (the hard way) that the method signature to HandleEntity changed. Instead of accepting a datastore.Entity, it’s now expecting a db.Model object.
Note: When actually running the remote api tool, you’ll also want to make sure your PYTHONPATH includes your current project. (Another one-liner in the documentation. )
Integrating Boto + App Engine
I wasn’t running off of Boto’s trunk and I was getting an obscure type conversion error. Being too lazy to check out the source, I jumped to their issue tracker and found a patch (halfway down the page) by one of the App Engine Patch lead devs. Apply the patch and you’ll be on your way to uploading images/data from App Engine into Amazon S3! If you’re looking for example code, I’ve included a small snippet of what I tested.
1 2 3 4 5 6 7 8 9 10 11
@staticmethod def upload_to_s3(original_filename, photo): """ Upload a photo file, storing its original name as metadata in an S3 bucket """ connection = Connection(settings.AWS_ACCESS_KEY_ID, settings.AWS_SECRET_ACCESS_KEY) bucket = connection.get_bucket(settings.AWS_IMAGE_BUCKET_NAME) photo_uuid = str(uuid.uuid4()) new_key = Key(bucket) new_key.key = photo_uuid new_key.set_metadata('original_filename', original_filename) new_key.set_contents_from_file(photo, policy='public-read') return photo_uuid
Note: I only tested the code above with small images ~300-500K in size and it seemed to work perfectly fine (with no load! ). As always, feel free to use, steal, take, and/or copy anything on this blog. Hopefully somewhere, someone on the Interwebs will find these tips handy!