Controlling Web-Bots with CurbIt

Whether you have a content-heavy site or a very application-centric website, bots and harvesters can wreak havoc by eating up CPU cycles, memory, and system resources.  These ubiquitous pests will gladly retrieve all of your site's content with little regard for copyright laws or your terms of service.

One helpful tool for limiting many types of harmful bots and crawlers is a Ruby gem or plugin called CurbIt.  CurbIt adds application level rate limiting to your Rails app.  I recently had the pleasure of utilizing CurbIt on the Paper Rater website to limit the number of submissions.  This helps us to ensure that humans are submitting documents, but without bothering our users with a CAPTCHA.

Example usage from the CurbIt github page:

class InvitesController < ApplicationController
    def invite
      # invite logic...
    end

    rate_limit :invite, :max_calls => 2, :time_limit => 30.seconds, :wait_time => 1.minute
  end