Firebelly Design

March 02, 2010 by Brandon Weiss

The biggest hurdle in my quest to get our site running on Heroku turned out to be asset uploading. Because of the way Heroku is architected, each application instance is read-only. You cannot just upload files and write them to disk, and even if you could, they would only be available to the instance you uploaded them to; none of the others would be able to see them.

Heroku is cloud-based hosting, so the solution is cloud-based storage. Any service will do, but I’d recommend Amazon S3. If you aren’t using it already, now is as good a time as any to switch. The benefits of having a fast, scalable storage system in the cloud are almost too many to count.

Unfortunately our site wasn’t using S3 yet, so the first step was to migrate all the uploaded assets to it. Luckily, if you’re using a file attachment library like Paperclip (which we are), that part is actually pretty easy.

Paperclip filesystem to Paperclip S3

To start, create an s3.yml and put it in config/. Here’s what’s in mine:

access_key_id: xxxxxxxxxxxxxxxxxxxx
secret_access_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
bucket: firebelly

If you like, you can also specify different accounts/buckets for development and production like you do for databases.

Then in your model change your attachment method from something like this:

has_attached_file :photo

To this:

has_attached_file :photo,
  :storage => :s3,
  :s3_credentials => "#{RAILS_ROOT}/config/s3.yml",
  :path => ":attachment/:id/:style.:extension"

Then migrate the files themselves over to S3, and either change the structure to match your :path, or change the :path to match your directory structure. Be sure to set the access permissions correctly; you want Read set for all users, or the files won’t be viewable. Future file uploads will have permissions automatically set by Paperclip, but when you upload them manually the permissions must be set manually as well.

The Plot Thickens

I thought I was done, but then I saw it, lurking in the corner: attachment_fu. Some of our models were sadly still using attachment_fu, and while there’s nothing necessarily wrong with attachment_fu, Paperclip is just more modern and has become the de facto standard for file attachments. Plus, we definitely shouldn’t be using two different libraries in the same app; that’s just ugly. So I have to migrate all the models using attachment_fu to use Paperclip instead.

I had a feeling this was going to be really, really painful, so I hit up Google first to see if maybe someone else had written about it and I could save myself some trouble. But surprisingly, even though many people must have done this before me, I could only find one halfway decent guide for migrating from attachment_fu to Paperclip written by James Stewart. And just my luck, it didn’t work for me. I’m sure it must be something related to my particular setup, because it seemed to work for everyone in the comments, but I just got a bunch of errors that I couldn’t figure out. So instead I hacked James’ script to get around the errors. Here’s what I did:

Attachment_fu filesystem to Paperclip S3

First, grab a copy of the Paperclip migration script that I forked and modified (yay, GitHub) and put it in lib/.

Now make a new migration:

script/generate migration convert_project_slides

And edit it:

class ConvertProjectSlides < ActiveRecord::Migration
  include PaperclipMigrations
  
  def self.up
    add_paperclip_fields :project_slides, :photo
    
    ProjectSlide.reset_column_information
    
    ProjectSlide.all.each do |project_slide|
      populate_paperclip_from_attachment_fu(project_slide, project_slide, 'photo', '/project_slides/')
    end
  end
  
  def self.down
    remove_paperclip_fields :project_slides, :photo
  end
end

Let me break down what’s happening in this migration.

1. You have to include the paperclip_migrations.rb script in your migration.
2. Then the columns Paperclip uses need to be added to the project_slides table.
3. The arguments that have to be passed to populate_paperclip_from_attachment_fu is where I got a little confused. The first one is the model you’re sending to, the second is the attachment you’re sending from. In this case they’re the same thing. The third one is the prefix you specified in add_paperclip_fields, and the fourth is the directory where the attachment_fu files are.

The original script didn’t have that fourth argument because it can be gotten from attachment_fu, but inexplicably that wouldn’t work for me, so I had to disable attachment_fu and specify it manually.

But before you run the migration, you have to take out attachment_fu’s has_attachment declaration in your model. Now would also be a good time to replace it with Paperclip’s declaration. So just like before, something like this:

has_attachment :content_type => :image, 
  :storage => :file_system,
  :path_prefix => 'public/project_slides',
  :resize_to => '345x285>',
  :thumbnails => { :thumb => '135x135!' }

Becomes something like this:

has_attached_file :photo,
  :styles => { :thumb => '135x135!' },
  :storage => :s3,
  :s3_credentials => "#{RAILS_ROOT}/config/s3.yml",
  :path => ":attachment/:id/:style.:extension"

One more thing before you run the migration. When attachment_fu creates thumbnails I guess the way it links them back to the original is by creating a new record in the table, and referencing the original record in a parent_id column, creating a tree structure. But I only want the migration to run on the original attachments, not thumbnails, so I deleted them from the database with something like:

DELETE FROM project_slides WHERE parent_id IS NOT NULL

That SQL query will delete any rows that have a value in the parent_id field (which means they aren’t originals).

OK, all set, now you can run the migration:

rake db:migrate

Assuming everything worked, you might want to drop attachment_fu’s various columns from your table. The ones I had were:

size
content_type
filename
height
width
parent_id
thumbnail

Lastly, just upload the photos to S3 like before, set the permissions and you’re done.

Comments (0)

October 05, 2009 by Brandon Weiss

I come from a strictly web development background. I’m not a computer science major, nor have I ever really used any software development languages besides in the requisite C++/Java class that every developer seems to take in high school or college. Since I started on the web rather than in software, my first programming language was PHP, so I hadn’t even heard the term ‘stack trace’ until I started using Ruby on Rails.

For those not in the know, a stack trace is really just an error report. You run your application, something goes wrong and it spits out a stack trace so you can figure out what it is. The reason it’s called a ‘trace’ is because rather than just showing the line and file the error occurred in, it actually traces the path through the program that was taken to get to the error, which can be useful in debugging.

Reading a stack trace isn’t hard; it’s one of those things that once you know how seems extremely obvious, but at first is a little unclear, especially if you come from a PHP-based web development background like myself. Here’s an example:

dev:fb brandon$ script/server
=> Booting WEBrick
=> Rails 2.3.4 application starting on http://0.0.0.0:3000
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- acts_as_ferret (MissingSourceFile)
  from /usr/.../custom_require.rb:31:in `require'
  from /usr/.../dependencies.rb:156:in `require'
  from /usr/.../dependencies.rb:521:in `new_constants_in'
  from /usr/.../dependencies.rb:156:in `require'
  from /Users/brandon/Sites/fb/config/environment.rb:64
  from /usr/.../custom_require.rb:31:in `gem_original_require'
  from /usr/.../custom_require.rb:31:in `require'
  from /usr/.../dependencies.rb:156:in `require'
  from /usr/.../dependencies.rb:521:in `new_constants_in'
  from /usr/.../dependencies.rb:156:in `require'
  from /usr/.../commands/server.rb:84
  from /usr/.../custom_require.rb:31:in `gem_original_require'
  from /usr/.../custom_require.rb:31:in `require'
  from script/server:3

Woah, what the hell is all that? It looks intimidating, but it’s really not. You can ignore the ellipses in the paths; they aren’t normally there, I just cut the paths short for brevity’s sake.

So let’s start at the top. script/server is a script that starts up a very simple Ruby server called WEBrick (used for development only). You can see WEBrick starting to boot up and then a whole bunch of FAIL. The stack trace is ordered in reverse chronology, so it starts at the bottom and ends at the top. If you look at the bottom you can see the script I initially called. Then lots of stuff happens. These aren’t errors, it’s just the path that’s being taken through the program. After the line about the Rails application starting you’ll see a summary:

/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- acts_as_ferret (MissingSourceFile)

This was the most confusing part for me, because in PHP, an error generally looks something like this:

Warning: Invalid argument supplied for foreach() in /www/yourserver/html/index.php on line 42

This is fairly unambiguous. On line 42 in index.php the argument given to the foreach() is bad. But the stack trace summary isn’t as simple as that. If you look at the path, that’s not even a file in my application; it’s a file that’s part of RubyGems, the plugin system for Ruby. Obviously there’s not an error in RubyGems (well I suppose there could be, but let’s assume there isn’t). At the end of the summary, you’ll see the error itself:

no such file to load -- acts_as_ferret

A quick Google search shows that acts_as_ferret is a Ruby gem (plugin) for implementing full text search using Ferret. OK, so clearly the acts_as_ferret gem is missing. You could just sudo install gem acts_as_ferret and the next time you run script/server the error would most likely be gone. But something is amiss. Rails has this awesome dependency architecture where you can specify gems that your app requires, and if they aren’t there, Rails will nicely let you know that; it shouldn’t be throwing an error like this.

To find the location of the problem, you actually need to look for the last line (chronologically) in the stack trace that has a path referencing a file in your app. In my case there’s only one line, and it is:

from /Users/brandon/Sites/fb/config/environment.rb:64

If I go to line 64 in environment.rb I see this:

require 'acts_as_ferret'

And there’s the problem. The gem was being referenced incorrectly. In your environment.rb file the syntax for indicating a dependency would be something like:

config.gem 'acts_as_ferret'

Awesome, now when I run script/server I get:

dev:fb brandon$ script/server
=> Booting WEBrick
=> Rails 2.3.4 application starting on http://0.0.0.0:3000
Missing these required gems:
  acts_as_ferret  
  campaign_monitor  
  flickraw  

You're running:
  ruby 1.8.7.174 at /usr/local/bin/ruby
  rubygems 1.3.5 at /Users/brandon/.gem/ruby/1.8, /usr/local/lib/ruby/gems/1.8

Run `rake gems:install` to install the missing gems.

Comments (0)