Will a Robot Take My Job?

With many jobs lost and the economy teetering on recovery, this question couldn't be more relevant. The man vs machine debate has a long history and conjures up images of John Henry racing to his death against a steam hammer. The reality is that machines already do the work of people and will continue to usurp greater and greater responsibilities. If your job is not one of the ones lost, then it is quite easy to see that this expansion should be welcome as it allows for greater societal productivity. After all, do you miss doing dishes by hand? Wish you could wash your laundry in a tub and hang it out to dry? Want to pay greater prices for hand made products? The key is understand which jobs are likely to be filled by machines. The following 5 questions should help.

  

Is your work highly repetitive?  
Specialized, repetitive tasks are easier to automate.  It is the human ability to generalize that makes our human "wetware" different than robotic hardware. Maybe you execute the same movements as part of a manual labor position, or maybe you spend the day downloading the same data and plugging it into the same spreadsheet.  Whatever the case may be, if you find yourself doing the same tasks over and over, then your job is more likely to be taken by a machine.


Are there many people in your company in the same role as you?
If so, you are a great target for automation.  Developing algorithms and/or hardware to replace one person is often not cost effective, but it may be a no-brainer for a company to invest in machines that replace 100 people.


What collar do you wear?  White or blue?
Previous generations of machines have found great success replacing manual labor in factories, but the next generation of machines will be more likely to replace office workers -- white collar folks.  The reason for this is that massive data and computing power have reached a critical mass allowing difficult "thinking" tasks to be conquered by machines.  We now have machines that can dominate Jeopardy, diagnose cancer from a breast scan, and grade essays.  This trend will only increase as more data and computing power become available in the years ahead. 

On the other hand, most jobs that involve service work or manual labor will be difficult for robots to replace.  The reality is that complex analytical thinking will be easier to duplicate than our nerves and muscles.  We take for granted how easily we handle sophisticated movements, but it will be many years before robots are equipped with the sensors and actuators comparable to a human's.  Until then, don't expect the moving company to send in robots to pack up your house.


Do you interact with people?
Many jobs require a uniquely human touch, something that machines do not offer -- something that machines may never offer.  If meaningful interaction with people is an important part of your job, then you are unlikely to be replaced by a machine any time soon.  Don't expect to be seeing a salesperson or a robotic shrink in the next century.




The bottom line is that the use of machines will continue to allow society to be more productive, and productivity often means one person doing the work of many.  We should embrace this new potential, but also be cognizant of the skill sets most needed in the 21st century.

Quick Solutions: Rails 3, send_data, and garbled PDF output

This is just a quick blog entry to help anyone experiencing garbled PDF output (in Safari) when using Rails 3's send_data to output dynamically-generated PDF files. 

If you are following the documentation, then you are probably outputting something like this:

    send_data data, :filename => "myfile.pdf",
                          :type => 'application/pdf'

I don't know if this is specific to Rails 3, but the issue is that the Content-Type header is not being set to 'application/pdf', so setting it explicitly in the response should fix this issue:

    response.headers["Content-Type"]='application/pdf'
    send_data data, :filename => "myfile.pdf",
                          :type => 'application/pdf'

Machine Learning: Naive Bayes Classification with Ruby

Maybe you've wondered, "Where are all the Ruby libraries for Machine Learning and NLP?".  Despite Ruby's growing user base and ability to quickly manipulate data and text, there seems to be a dearth of tools for NLP and Machine Learning.  One statistical tool that finds itself in the intersection of NLP and Machine Learning is Naive Bayes.  [For those already familiar with Naive Bayes, you may wish to skip ahead to the Quickstart section below (although I promise not to be long-winded in my intro).]    

This peculiarly-named approach to classification tasks is based on the well-known Bayes' Rule and is used to calculate the probability an instance belongs in a particular class (or category) based on the components of the instance.   It is called "Naive" Bayes because of the manner in which it calculates the probabilities.  It treats each component as if it is independent from other components, even though this is usually not the case.  Suprisingly, Naive Bayes does quite well and is optimal in the case in which the components of an instance actually are independent. 

Enough generalities...Naive Bayes is used widely for text classification problems and the "components" that I referenced above are actually tokens -- typically words.  Even if you have no desire to understand the probabilistic engine beneath the hood, Naive Bayes is easy to use, high performance, and accurate relative to other classifiers.  It requires a 2 step process:

1) Train the classifier by providing it with sets of tokens (e.g., words) accompanied by a class (e.g., 'SPAM')
2) Run the trained classifier on un-classified (e.g., unlabelled) tokens and it will predict a class

Quickstart

gem install nbayes

After that, it's time to begin training the classifier:

# create new classifier instance
nbayes = NBayes::Base.new
# train it - notice split method used to tokenize text (more on that below)
nbayes.train( "You need to buy some Viagra".split(/\s+/), 'SPAM' )
nbayes.train( "This is not spam, just a letter to Bob.".split(/\s+/), 'HAM' )
nbayes.train( "Hey Oasic, Do you offer consulting?".split(/\s+/), 'HAM' )
nbayes.train( "You should buy this stock".split(/\s+/), 'SPAM' )

Finally, let's use it to classify a document:

# tokenize message
tokens = "Now is the time to buy Viagra cheaply and discreetly".split(/\s+/)
result = @nbayes.classify(tokens)
# print likely class (SPAM or HAM)
p result.max_class
# print probability of message being SPAM
p result['SPAM']
# print probability of message being HAM
p result['HAM']

But that's not all!  I'm claiming that this is a full-featured Naive Bayes implementation, so I better back that up with information about all the goodies.  Here we go:

Features

• Works with all types of tokens, not just text.  Of course, because of this, we leave tokenization up to you.
• Disk based persistence
• Allows prior distribution on classes to be assumed uniform (optional)
  
• Outputs probabilities, instead of just class w/max probability
  
• Customizable constant value for Laplacian smoothing
  
• Optional and customizable purging of low-frequency tokens (for performance)
  
• Optional binarized mode to reduce the impact of repeated words
  
• Uses log probabilities to avoid underflow
  

I hope to post examples of these features in action in a future post.  Until then, view nbayes_spec.rb for usage.

Solutions to invalid byte sequence in UTF-8 (ArgumentError)

Let's keep this short and simple. You're reading this because you too are tired of all the new character set issues in Ruby 1.9. This page lists some possible solutions and links to other pages with helpful info on solving "invalid byte sequence in UTF-8 (ArgumentError)".  

The Cause
This problem occurs when your application is expecting one character encoding and gets a different one.  


Possible Solutions:

1) Explicitly specify the encodings that you expect in the top-level Ruby Encoding class. Put this in a file that will be loaded (e.g., boot.rb for Rails):

Encoding.default_external = 'UTF-8'
Encoding.default_internal = 'UTF-8'

2) Convert the character encoding on a lower level within your application

ascii_str.encode("UTF-8")

More info: http://blog.grayproductions.net/articles/ruby_19s_string


3) Finally, if the text you are working with has a mix of character encodings (occasionally this happens when data is aggregated improperly), then you may need to decide what is the most common character set and convert from that to the new char set (e.g., UTF-8), while ignoring any other unrecognizable characters.

You can do this using iconv from the command line or from within Ruby:
within Ruby:

ic = Iconv.new('UTF-8//IGNORE', 'UTF-8')
valid_string = ic.iconv(untrusted_string)

(source: http://smyck.net/2011/05/13/files-with-mixed-and-invalid-encodings-in-ruby/)

 command line:

iconv -f WINDOWS-1252 -t "UTF-8//IGNORE" some_text.txt > some_text.utf8.txt

4) Catch the error and handle it more gracefully:

begin
  # your code
rescue ArgumentError => e   
  print "error: #{e}"
end

Taking a Random Walk with Processing 1.5

If you've ever dabbled with simulations you have probably come across Processing, the open source environment for animation, interaction, and much more.  The Java-ish scripting language allows for quick prototyping and the library of examples allow even beginners (me) to get moving quickly.

To the right is an image I created from a very simple simulation of a "random walk" along the y-axis as time moved along the x-axis.  Many have noted the similarity between this process and the peaks and valleys of markets, mountain ranges, and other stochastic processes.  If you haven't already, give it a try.

Source for Random Walk:

int x, y, r, middle, randOffset, previousX, previousY;

 void setup() {
   size(900, 900);
   stroke(0);
   background(192, 64, 0);
   x = 0;
   r = 2;
   y = 200;
 } 

 void draw() {
   previousX = x;
   previousY = y;
   x++;
   // y does random up or down of 10 units
   randOffset = 10 - (int)random(21);
   y = y + randOffset;
   //line(150, 25, mouseX, mouseY);
   stroke(0);
   line(previousX, previousY, x, y);
   if(x < 800) save("random_walk.tif");
 }

Rails 3, Ruby 1.9.2, and Character Encoding Nightmares

Ruby (and Rails) can't take all the blame for this mess.  No, it's been present as long as humanity has grasped the social innovation of language.  Even if your goals are not quite so high as the Tower of Babel (perhaps just writing a cool web app), it's understandable to feel some sort of divine opposition when dealing with frustrating character set issues.  Now that Ruby 1.9 and Rails 3 have become wise to character encodings, the reality of dealing with this messy subject is playing out in web applications and wreaking havoc on many.  Having dealt with these problems in many web applications from Java to Ruby over the years, it's easy for me to think that immunity is certain, even deserved, but think again.

The Problem Described
Just recently, a very popular website that I consult on was occasionally throwing the following error that is not uncommon after upgrading to Rails 3 and Ruby 1.9.x:

ActionView::Template::Error: incompatible character encodings: ASCII-8BIT and UTF-8

I've received this type of error (incompatible character encodings) before, but not in a Rails view.  Nevertheless, I began by inspecting the usual suspects.

MySQL:   SHOW VARIABLES LIKE '%char%';
Ok, all clear there...

application.rb includes config.encoding="UTF-8"  --  YES
using 'mysql2' gem --  YES
setting # encoding: utf-8 at the top of any ruby files with UTF-8 string literals  --  YES

running 'locale' on my linux system...UTF-8...  -- YES
firing up the Rails console and printing out Encoding.default_internal and Encoding.default_external... -- YES

If you reach this point and you are still receiving the problem, then it should be clear that this is probably not going to be solved easily.  Fortunately, the stack trace gave me the line in question from my erb file and Airbrake (formerly Hoptoad), provided the specific input that was causing the problem.  Using this, I was able to duplicate the error with a Rails functional test and begin looking for solutions.

A string was being output in the view such that changing it to
<%= mystring.force_encoding("ASCII-8BIT") %>
solved the problem, but not really.  This caused the error to go away, but only by asking that Rails treat a UTF-8 string as an ASCII string, which could lead to more serious problems.  And why would setting a string to ASCII fix the problem in a UTF-8 Rails app?  Shouldn't that cause the problem?

After further tinkering, I'll describe my current understanding of what happened; and if you're reading along, hopefully this will be helpful in allowing you to solve a similar problem that you might be having.  As I will get to later, a string that was output higher up in the erb file had caused Erb to switch its encoding from UTF-8 to ASCII-8BIT.  Perhaps this functionality was added to allow Erb to gracefully handle any character encoding.  The problem in this case arose when 2 strings with different character encodings were being output in the same erb file.  So, it was simply a matter of tracking down which other string caused the problem.

As it turns out, it was a string populated from an HTTP request using Net::HTTP, which sets the encoding to "ASCII-8BIT", even when the actual encoding is "UTF-8".  Since the string was a UTF-8 string mis-labelled as ASCII, the solution was to call "http_string.force_encoding("UTF-8")", which correctly labelled the string.  After this change was made, all tests passed.  I hope you have the same luck with your related problem.  :-)

FeedTools + Ruby 1.9.2 + Rails 3

During an upgrade of a RoR app, the following was encountered:

ruby -c /usr/local/rvm/gems/ruby-1.9.2-p0/gems/feedtools-0.2.29/lib/feed_tools/helpers/uri_helper.rb
/usr/local/rvm/gems/ruby-1.9.2-p0/gems/feedtools-0.2.29/lib/feed_tools/helpers/uri_helper.rb:43: invalid multibyte char (US-ASCII)
/usr/local/rvm/gems/ruby-1.9.2-p0/gems/feedtools-0.2.29/lib/feed_tools/helpers/uri_helper.rb:43: invalid multibyte char (US-ASCII)
/usr/local/rvm/gems/ruby-1.9.2-p0/gems/feedtools-0.2.29/lib/feed_tools/helpers/uri_helper.rb:43: syntax error, unexpected $end, expecting ')'
          if IDN::Idna.toASCII('http://www.詹姆斯.com/') ==
                                             ^

To fix, simply add the following to the first line of uri_helper.rb in the feed_tools gem directory:

  # encoding: utf-8

This allows the Ruby interpreter to correctly interpret the file as a UTF-8 file.