Taking a Random Walk with Processing 1.5

If you've ever dabbled with simulations you have probably come across Processing, the open source environment for animation, interaction, and much more.  The Java-ish scripting language allows for quick prototyping and the library of examples allow even beginners (me) to get moving quickly.

To the right is an image I created from a very simple simulation of a "random walk" along the y-axis as time moved along the x-axis.  Many have noted the similarity between this process and the peaks and valleys of markets, mountain ranges, and other stochastic processes.  If you haven't already, give it a try.

Source for Random Walk:

int x, y, r, middle, randOffset, previousX, previousY;

 void setup() {
   size(900, 900);
   stroke(0);
   background(192, 64, 0);
   x = 0;
   r = 2;
   y = 200;
 } 

 void draw() {
   previousX = x;
   previousY = y;
   x++;
   // y does random up or down of 10 units
   randOffset = 10 - (int)random(21);
   y = y + randOffset;
   //line(150, 25, mouseX, mouseY);
   stroke(0);
   line(previousX, previousY, x, y);
   if(x < 800) save("random_walk.tif");
 }

Rails 3, Ruby 1.9.2, and Character Encoding Nightmares

Ruby (and Rails) can't take all the blame for this mess.  No, it's been present as long as humanity has grasped the social innovation of language.  Even if your goals are not quite so high as the Tower of Babel (perhaps just writing a cool web app), it's understandable to feel some sort of divine opposition when dealing with frustrating character set issues.  Now that Ruby 1.9 and Rails 3 have become wise to character encodings, the reality of dealing with this messy subject is playing out in web applications and wreaking havoc on many.  Having dealt with these problems in many web applications from Java to Ruby over the years, it's easy for me to think that immunity is certain, even deserved, but think again.

The Problem Described
Just recently, a very popular website that I consult on was occasionally throwing the following error that is not uncommon after upgrading to Rails 3 and Ruby 1.9.x:

ActionView::Template::Error: incompatible character encodings: ASCII-8BIT and UTF-8

I've received this type of error (incompatible character encodings) before, but not in a Rails view.  Nevertheless, I began by inspecting the usual suspects.

MySQL:   SHOW VARIABLES LIKE '%char%';
Ok, all clear there...

application.rb includes config.encoding="UTF-8"  --  YES
using 'mysql2' gem --  YES
setting # encoding: utf-8 at the top of any ruby files with UTF-8 string literals  --  YES

running 'locale' on my linux system...UTF-8...  -- YES
firing up the Rails console and printing out Encoding.default_internal and Encoding.default_external... -- YES

If you reach this point and you are still receiving the problem, then it should be clear that this is probably not going to be solved easily.  Fortunately, the stack trace gave me the line in question from my erb file and Airbrake (formerly Hoptoad), provided the specific input that was causing the problem.  Using this, I was able to duplicate the error with a Rails functional test and begin looking for solutions.

A string was being output in the view such that changing it to
<%= mystring.force_encoding("ASCII-8BIT") %>
solved the problem, but not really.  This caused the error to go away, but only by asking that Rails treat a UTF-8 string as an ASCII string, which could lead to more serious problems.  And why would setting a string to ASCII fix the problem in a UTF-8 Rails app?  Shouldn't that cause the problem?

After further tinkering, I'll describe my current understanding of what happened; and if you're reading along, hopefully this will be helpful in allowing you to solve a similar problem that you might be having.  As I will get to later, a string that was output higher up in the erb file had caused Erb to switch its encoding from UTF-8 to ASCII-8BIT.  Perhaps this functionality was added to allow Erb to gracefully handle any character encoding.  The problem in this case arose when 2 strings with different character encodings were being output in the same erb file.  So, it was simply a matter of tracking down which other string caused the problem.

As it turns out, it was a string populated from an HTTP request using Net::HTTP, which sets the encoding to "ASCII-8BIT", even when the actual encoding is "UTF-8".  Since the string was a UTF-8 string mis-labelled as ASCII, the solution was to call "http_string.force_encoding("UTF-8")", which correctly labelled the string.  After this change was made, all tests passed.  I hope you have the same luck with your related problem.  :-)

FeedTools + Ruby 1.9.2 + Rails 3

During an upgrade of a RoR app, the following was encountered:

ruby -c /usr/local/rvm/gems/ruby-1.9.2-p0/gems/feedtools-0.2.29/lib/feed_tools/helpers/uri_helper.rb
/usr/local/rvm/gems/ruby-1.9.2-p0/gems/feedtools-0.2.29/lib/feed_tools/helpers/uri_helper.rb:43: invalid multibyte char (US-ASCII)
/usr/local/rvm/gems/ruby-1.9.2-p0/gems/feedtools-0.2.29/lib/feed_tools/helpers/uri_helper.rb:43: invalid multibyte char (US-ASCII)
/usr/local/rvm/gems/ruby-1.9.2-p0/gems/feedtools-0.2.29/lib/feed_tools/helpers/uri_helper.rb:43: syntax error, unexpected $end, expecting ')'
          if IDN::Idna.toASCII('http://www.詹姆斯.com/') ==
                                             ^

To fix, simply add the following to the first line of uri_helper.rb in the feed_tools gem directory:

  # encoding: utf-8

This allows the Ruby interpreter to correctly interpret the file as a UTF-8 file.

Using Mysqldump Without Downtime

Mysqldump is a great utility for creating easy to use, easy to restore mysql database backups, but it can cause downtime if certain precautions are not taken.  If you manage MySQL across multiple Linux flavors, various default configurations can cause mysqldump to act in unpredictable ways.  Let's look at a couple common problem areas related to mysqldump.

Using the Right Storage Engine
If you're not certain which MySQL storage engine to use, then consider using InnoDB (instead of MyISAM) if you are not already doing so.  Using mysqldump to backup a large MyISAM table can cause the entire table to lock until the backup is complete.  Even though the long-running read done by mysqldump does not block other reads of the same table, what happens is that an update query issued will cause all subsequent reads to be queued and therefore blocked.  So, switching to InnoDB will solve this type of problem due to its usage of row-level locking.

Using --single-transaction
On most of the MySQL installations that I've managed, simply using InnoDB will allow for backups to be created w/o tables locking and queries/updates blocking.  However, a recent Ubuntu installation continued to block our Rails application from using the database while mysqldump was running and it was necessary to use the parameter "--single-transaction" like so:

mysqldump -u myuser -pmypass --single-transaction db_name > output.sql

If you have any tips to share, I welcome them in the comments.

To Freemium or Not to Freemium?

This is a topic that comes up a lot when dealing with Internet businesses.  I would say that the topic is interesting, but I actually find it somewhat nauseating as I reflect on the number of online quarrels I've encountered when reading about the Freemium debate.  This is why I found it refreshing to come across this nice article from the folks at MailChimp.  A beautifully laid out case study on some of the pros and cons of Freemium, and an encouragement to startups that have been around a little too long to be considered a true startup.  Personally, the idea of Freemium appeals to me as it gives users a chance to taste your product for free, and gives you (the business) an opportunity to learn from your users and grow your user base.  So, "To Freemium or Not to Freemium?" -- you decide.

PayPal Charset IPN Issue

This post is for anyone experiencing problems with the PayPal verification step in the IPN process.  After pulling my hair out trying to figure this out, I found the solution on another blog.

The Problem:
PayPal by default posts its IPNs using the Windows-1252 character encoding (why would they do that?).  If you are like any sensible UTF-8-loving developer, then you will find that your postbacks to PayPal are not receiving the VERIFIED status that ensures they are a legitimate IPN.  It seems non-ASCII characters in the IPN parameters are not being interpreted correctly by PayPal (because they are expecting Windows-1252 and you are sending UTF-8).

The Fix:
A simple setting in your business PayPal account under "Language Encodings".  You will need to select "More Options" to find the screen that allows you to select UTF-8 from the dropdown.

Beware of YSlow and GZip

Much time can be wasted trying to configure mod_deflate IF it is already properly configured.  Although I had previously enabled compression using this simple and powerful Apache module, YSlow was giving me an F for not compressing the page text as well as js and css files.  To make a short story even shorter I will simply say this: 

YSlow does not accurately detect gzip compression.  If you are uncertain, check the response headers for "Vary: Accept Encoding" or use Port80's Compression Check.