Firebelly Design

March 02, 2010 by Brandon Weiss

The biggest hurdle in my quest to get our site running on Heroku turned out to be asset uploading. Because of the way Heroku is architected, each application instance is read-only. You cannot just upload files and write them to disk, and even if you could, they would only be available to the instance you uploaded them to; none of the others would be able to see them.

Heroku is cloud-based hosting, so the solution is cloud-based storage. Any service will do, but I’d recommend Amazon S3. If you aren’t using it already, now is as good a time as any to switch. The benefits of having a fast, scalable storage system in the cloud are almost too many to count.

Unfortunately our site wasn’t using S3 yet, so the first step was to migrate all the uploaded assets to it. Luckily, if you’re using a file attachment library like Paperclip (which we are), that part is actually pretty easy.

Paperclip filesystem to Paperclip S3

To start, create an s3.yml and put it in config/. Here’s what’s in mine:

access_key_id: xxxxxxxxxxxxxxxxxxxx
secret_access_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
bucket: firebelly

If you like, you can also specify different accounts/buckets for development and production like you do for databases.

Then in your model change your attachment method from something like this:

has_attached_file :photo

To this:

has_attached_file :photo,
  :storage => :s3,
  :s3_credentials => "#{RAILS_ROOT}/config/s3.yml",
  :path => ":attachment/:id/:style.:extension"

Then migrate the files themselves over to S3, and either change the structure to match your :path, or change the :path to match your directory structure. Be sure to set the access permissions correctly; you want Read set for all users, or the files won’t be viewable. Future file uploads will have permissions automatically set by Paperclip, but when you upload them manually the permissions must be set manually as well.

The Plot Thickens

I thought I was done, but then I saw it, lurking in the corner: attachment_fu. Some of our models were sadly still using attachment_fu, and while there’s nothing necessarily wrong with attachment_fu, Paperclip is just more modern and has become the de facto standard for file attachments. Plus, we definitely shouldn’t be using two different libraries in the same app; that’s just ugly. So I have to migrate all the models using attachment_fu to use Paperclip instead.

I had a feeling this was going to be really, really painful, so I hit up Google first to see if maybe someone else had written about it and I could save myself some trouble. But surprisingly, even though many people must have done this before me, I could only find one halfway decent guide for migrating from attachment_fu to Paperclip written by James Stewart. And just my luck, it didn’t work for me. I’m sure it must be something related to my particular setup, because it seemed to work for everyone in the comments, but I just got a bunch of errors that I couldn’t figure out. So instead I hacked James’ script to get around the errors. Here’s what I did:

Attachment_fu filesystem to Paperclip S3

First, grab a copy of the Paperclip migration script that I forked and modified (yay, GitHub) and put it in lib/.

Now make a new migration:

script/generate migration convert_project_slides

And edit it:

class ConvertProjectSlides < ActiveRecord::Migration
  include PaperclipMigrations
  
  def self.up
    add_paperclip_fields :project_slides, :photo
    
    ProjectSlide.reset_column_information
    
    ProjectSlide.all.each do |project_slide|
      populate_paperclip_from_attachment_fu(project_slide, project_slide, 'photo', '/project_slides/')
    end
  end
  
  def self.down
    remove_paperclip_fields :project_slides, :photo
  end
end

Let me break down what’s happening in this migration.

1. You have to include the paperclip_migrations.rb script in your migration.
2. Then the columns Paperclip uses need to be added to the project_slides table.
3. The arguments that have to be passed to populate_paperclip_from_attachment_fu is where I got a little confused. The first one is the model you’re sending to, the second is the attachment you’re sending from. In this case they’re the same thing. The third one is the prefix you specified in add_paperclip_fields, and the fourth is the directory where the attachment_fu files are.

The original script didn’t have that fourth argument because it can be gotten from attachment_fu, but inexplicably that wouldn’t work for me, so I had to disable attachment_fu and specify it manually.

But before you run the migration, you have to take out attachment_fu’s has_attachment declaration in your model. Now would also be a good time to replace it with Paperclip’s declaration. So just like before, something like this:

has_attachment :content_type => :image, 
  :storage => :file_system,
  :path_prefix => 'public/project_slides',
  :resize_to => '345x285>',
  :thumbnails => { :thumb => '135x135!' }

Becomes something like this:

has_attached_file :photo,
  :styles => { :thumb => '135x135!' },
  :storage => :s3,
  :s3_credentials => "#{RAILS_ROOT}/config/s3.yml",
  :path => ":attachment/:id/:style.:extension"

One more thing before you run the migration. When attachment_fu creates thumbnails I guess the way it links them back to the original is by creating a new record in the table, and referencing the original record in a parent_id column, creating a tree structure. But I only want the migration to run on the original attachments, not thumbnails, so I deleted them from the database with something like:

DELETE FROM project_slides WHERE parent_id IS NOT NULL

That SQL query will delete any rows that have a value in the parent_id field (which means they aren’t originals).

OK, all set, now you can run the migration:

rake db:migrate

Assuming everything worked, you might want to drop attachment_fu’s various columns from your table. The ones I had were:

size
content_type
filename
height
width
parent_id
thumbnail

Lastly, just upload the photos to S3 like before, set the permissions and you’re done.

Comments (0)

February 19, 2010 by Brandon Weiss

I talk about this all the time, but it bears repeating: servers are expensive. They might seem so cheap up front, but once you factor in average maintenance time the actual cost could easily be a thousand dollars a month, if not more, a fact I lamented in a breakdown of the different types of hosting here a few months ago.

Unfortunately, my breakdown didn’t really have any hard numbers because I don’t keep track of task types when I freelance, and I don’t keep track of time at all on my personal projects. But at Firebelly we do, and now that I’ve been here long enough to get a good sample, I decided to run the numbers.

In the past 5 months I’ve spent 42 hours on server maintenance. That is, frankly, insane. That’s thousands and thousands of dollars worth of time. And that doesn’t even factor in back-end development work that had to be done as a result of having physical servers, like having to convert our asset uploads to use Amazon S3 because the server ran out of space (something I’ll get into in a later post). So 42 hours could easily be a conservative estimate. I wouldn’t be too surprised to see the true cost of our servers over 5 months exceed $10,000. Which is an obscene amount of money and the reason I’m so interested in cloud hosting.

In the breakdown I mentioned that Heroku might be the holy grail for Ruby hosting, but I didn’t have any direct experience with it, so it was really all just theoretical. But that was three months ago. Since then I’ve migrated all my personal projects to use Heroku.

Transitioning between hosts is a normally excruciatingly painful process, but moving to Heroku was actually fun. The whole process only took about a half hour for each project and went off virtually without a hitch. The only hiccup was a few differences in syntax between MySQL and PostgreSQL (the database Heroku uses), but that took less than a few minutes to fix, and after that it was all smooth sailing.

Now that I’ve experienced Heroku first-hand, the next step is to move Firebelly projects over, starting with our main site. I’ll recap in a few months after it’s done to report on how it went and how much time it’s saved us.

Comments (0)