Rails 3, Ruby 1.9.2, and Character Encoding Nightmares

Ruby (and Rails) can't take all the blame for this mess.  No, it's been present as long as humanity has grasped the social innovation of language.  Even if your goals are not quite so high as the Tower of Babel (perhaps just writing a cool web app), it's understandable to feel some sort of divine opposition when dealing with frustrating character set issues.  Now that Ruby 1.9 and Rails 3 have become wise to character encodings, the reality of dealing with this messy subject is playing out in web applications and wreaking havoc on many.  Having dealt with these problems in many web applications from Java to Ruby over the years, it's easy for me to think that immunity is certain, even deserved, but think again.

The Problem Described
Just recently, a very popular website that I consult on was occasionally throwing the following error that is not uncommon after upgrading to Rails 3 and Ruby 1.9.x:

ActionView::Template::Error: incompatible character encodings: ASCII-8BIT and UTF-8

I've received this type of error (incompatible character encodings) before, but not in a Rails view.  Nevertheless, I began by inspecting the usual suspects.

MySQL:   SHOW VARIABLES LIKE '%char%';
Ok, all clear there...

application.rb includes config.encoding="UTF-8"  --  YES
using 'mysql2' gem --  YES
setting # encoding: utf-8 at the top of any ruby files with UTF-8 string literals  --  YES

running 'locale' on my linux system...UTF-8...  -- YES
firing up the Rails console and printing out Encoding.default_internal and Encoding.default_external... -- YES

If you reach this point and you are still receiving the problem, then it should be clear that this is probably not going to be solved easily.  Fortunately, the stack trace gave me the line in question from my erb file and Airbrake (formerly Hoptoad), provided the specific input that was causing the problem.  Using this, I was able to duplicate the error with a Rails functional test and begin looking for solutions.

A string was being output in the view such that changing it to
<%= mystring.force_encoding("ASCII-8BIT") %>
solved the problem, but not really.  This caused the error to go away, but only by asking that Rails treat a UTF-8 string as an ASCII string, which could lead to more serious problems.  And why would setting a string to ASCII fix the problem in a UTF-8 Rails app?  Shouldn't that cause the problem?

After further tinkering, I'll describe my current understanding of what happened; and if you're reading along, hopefully this will be helpful in allowing you to solve a similar problem that you might be having.  As I will get to later, a string that was output higher up in the erb file had caused Erb to switch its encoding from UTF-8 to ASCII-8BIT.  Perhaps this functionality was added to allow Erb to gracefully handle any character encoding.  The problem in this case arose when 2 strings with different character encodings were being output in the same erb file.  So, it was simply a matter of tracking down which other string caused the problem.

As it turns out, it was a string populated from an HTTP request using Net::HTTP, which sets the encoding to "ASCII-8BIT", even when the actual encoding is "UTF-8".  Since the string was a UTF-8 string mis-labelled as ASCII, the solution was to call "http_string.force_encoding("UTF-8")", which correctly labelled the string.  After this change was made, all tests passed.  I hope you have the same luck with your related problem.  :-)