s3 Multipart chunks upload with Jquery and Rails

Three weeks ago a new feature was needed in the flow of our web app.  Because of our customer profile, we needed a way to improve the speed and reliability of uploaded files processed by our app. Originally we had been using Paperclip with S3; uploading files, pushing straight to our web server and then creating the object in S3.

This wasn’t working very well, especially because some customers have slow connections and were having trouble uploading their files.  Because of this and more, we realize we needed a different approach. We started researching available options, hoping for a short timeframe to support this feature.

We started with https://github.com/blueimp/jQuery-File-Upload, which is a very well-known JS plugin that helps support different non-standard features for file upload. After testing we found that this would have resulted in more work from our backend ratedbrides.com or the need for an enhancement directly to the JS plugin to make this multi-chunk functionality work with AWS S3, Rails, and our existing code.

Next, we looked at another JS library, https://github.com/TTLabs/EvaporateJS.  This seemed like a good choice in many ways, except that our AWS key would have been exposed to the front-end user when loading the page. So this wouldn’t have been a wise change either.

Finally, we moved to the https://github.com/maxgillett/s3_multipart gem that seemed to be more integrated with Rails recommended by @mvidaurre.  Once we bundle installed it we found the first problem, the example in the README didn’t work.  After reading the ‘issues’ from the repo I realized there should be a way to make it work.  I also had to take into account the fact that we had our own form and bootstrap framework integrated and this would have been a problem if the file input wrapper was not customizable.  I realized that it will be a good opportunity to tweak the code a little bit to make it skinnable.  Another thing I realized from the beginning was that was that the gem didn’t let you decide where would you be saving the files uploaded to S3 other than the bucket itself.  In the end,  we had to customize the path where we wanted the files to be temporary stored in S3, and then move them to the correct path since we were using Paperclip for our models to use attachments.

Thankfully after a couple of days of work and customizing the gem we ended up with a custom version of the gem that served our purpose of delegating the uploading of the files to S3 in multiple chunks.

The result of this customization is here:

https://github.com/joseluistorres/s3_multipart

Thanks to https://github.com/maxgillett for a great gem.

JoseLuis Torres
@joseluis_torres