Wednesday, August 17, 2011

Rails 3, Ruby 1.9.2, and Character Encoding Nightmares

Ruby (and Rails) can't take all the blame for this mess.  No, it's been present as long as humanity has grasped the social innovation of language.  Even if your goals are not quite so high as the Tower of Babel (perhaps just writing a cool web app), it's understandable to feel some sort of divine opposition when dealing with frustrating character set issues.  Now that Ruby 1.9 and Rails 3 have become wise to character encodings, the reality of dealing with this messy subject is playing out in web applications and wreaking havoc on many.  Having dealt with these problems in many web applications from Java to Ruby over the years, it's easy for me to think that immunity is certain, even deserved, but think again.

The Problem Described
Just recently, a very popular website that I consult on was occasionally throwing the following error that is not uncommon after upgrading to Rails 3 and Ruby 1.9.x:

ActionView::Template::Error: incompatible character encodings: ASCII-8BIT and UTF-8

I've received this type of error (incompatible character encodings) before, but not in a Rails view.  Nevertheless, I began by inspecting the usual suspects.

Ok, all clear there...

application.rb includes config.encoding="UTF-8"  --  YES
using 'mysql2' gem --  YES
setting # encoding: utf-8 at the top of any ruby files with UTF-8 string literals  --  YES

running 'locale' on my linux system...UTF-8...  -- YES
firing up the Rails console and printing out Encoding.default_internal and Encoding.default_external... -- YES

If you reach this point and you are still receiving the problem, then it should be clear that this is probably not going to be solved easily.  Fortunately, the stack trace gave me the line in question from my erb file and Airbrake (formerly Hoptoad), provided the specific input that was causing the problem.  Using this, I was able to duplicate the error with a Rails functional test and begin looking for solutions.

A string was being output in the view such that changing it to
<%= mystring.force_encoding("ASCII-8BIT") %>
solved the problem, but not really.  This caused the error to go away, but only by asking that Rails treat a UTF-8 string as an ASCII string, which could lead to more serious problems.  And why would setting a string to ASCII fix the problem in a UTF-8 Rails app?  Shouldn't that cause the problem?

After further tinkering, I'll describe my current understanding of what happened; and if you're reading along, hopefully this will be helpful in allowing you to solve a similar problem that you might be having.  As I will get to later, a string that was output higher up in the erb file had caused Erb to switch its encoding from UTF-8 to ASCII-8BIT.  Perhaps this functionality was added to allow Erb to gracefully handle any character encoding.  The problem in this case arose when 2 strings with different character encodings were being output in the same erb file.  So, it was simply a matter of tracking down which other string caused the problem.

As it turns out, it was a string populated from an HTTP request using Net::HTTP, which sets the encoding to "ASCII-8BIT", even when the actual encoding is "UTF-8".  Since the string was a UTF-8 string mis-labelled as ASCII, the solution was to call "http_string.force_encoding("UTF-8")", which correctly labelled the string.  After this change was made, all tests passed.  I hope you have the same luck with your related problem.  :-)

Monday, August 8, 2011

FeedTools + Ruby 1.9.2 + Rails 3

During an upgrade of a RoR app, the following was encountered:

ruby -c /usr/local/rvm/gems/ruby-1.9.2-p0/gems/feedtools-0.2.29/lib/feed_tools/helpers/uri_helper.rb
/usr/local/rvm/gems/ruby-1.9.2-p0/gems/feedtools-0.2.29/lib/feed_tools/helpers/uri_helper.rb:43: invalid multibyte char (US-ASCII)
/usr/local/rvm/gems/ruby-1.9.2-p0/gems/feedtools-0.2.29/lib/feed_tools/helpers/uri_helper.rb:43: invalid multibyte char (US-ASCII)
/usr/local/rvm/gems/ruby-1.9.2-p0/gems/feedtools-0.2.29/lib/feed_tools/helpers/uri_helper.rb:43: syntax error, unexpected $end, expecting ')'
          if IDN::Idna.toASCII('http://www.詹姆斯.com/') ==
To fix, simply add the following to the first line of uri_helper.rb in the feed_tools gem directory:

  # encoding: utf-8

This allows the Ruby interpreter to correctly interpret the file as a UTF-8 file.

Tuesday, April 26, 2011

Using Mysqldump Without Downtime

Learning MySQL
Mysqldump is a great utility for creating easy to use, easy to restore mysql database backups, but it can cause downtime if certain precautions are not taken.  If you manage MySQL across multiple Linux flavors, various default configurations can cause mysqldump to act in unpredictable ways.  Let's look at a couple common problem areas related to mysqldump.

Using the Right Storage Engine
If you're not certain which MySQL storage engine to use, then consider using InnoDB (instead of MyISAM) if you are not already doing so.  Using mysqldump to backup a large MyISAM table can cause the entire table to lock until the backup is complete.  Even though the long-running read done by mysqldump does not block other reads of the same table, what happens is that an update query issued will cause all subsequent reads to be queued and therefore blocked.  So, switching to InnoDB will solve this type of problem due to its usage of row-level locking.

Using --single-transaction
On most of the MySQL installations that I've managed, simply using InnoDB will allow for backups to be created w/o tables locking and queries/updates blocking.  However, a recent Ubuntu installation continued to block our Rails application from using the database while mysqldump was running and it was necessary to use the parameter "--single-transaction" like so:

mysqldump -u myuser -pmypass --single-transaction db_name > output.sql

If you have any tips to share, I welcome them in the comments.