Given /^I am browsing the site using Safari$/ do
  header "User-Agent" , "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us)"
end

Like me, you may have found out the hard way that this doesn’t work. Webrat does not automagically apply these new HTTP headers to your request – they certainly don’t make it to my controller. What worked for me:

Given /^I am browsing the site using Safari$/ do
  headers["User-Agent"] = "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us)"
end
 
When /^I visit my precious site$/ do
  get '/my/precious/path', my_query_string, headers
end

In the code above, headers is a method call that returns all the HTTP headers for your request. Just tack headers on as the third argument of your request, and you’re good to go.

In that same spirit, I took Lopex’s advice and turned on the –server flag; performance gain went from almost 2-to-1 to almost 3-to-1, with no other changes or optimizations.

Before:

[~/project/jrubytest/primes]> ruby ./primes.rb
FINDING first 100000 PRIMES
1299709
Time: 57.716853s
[~/project/jrubytest/primes]> jruby ./primes.rb
FINDING first 100000 PRIMES
1299709
Time: 33.885273s

After:

[~/project/jrubytest/primes]> ruby primes.rb
FINDING first 100000 PRIMES
1299709
Time: 57.204178s
[~/project/jrubytest/primes]> jruby --server primes.rb
FINDING first 100000 PRIMES
1299709
Time: 21.067204s

Thanks to the JRuby crew for stopping by here, and for being so helpful generally.

On the performance front, I did a quick-and-dirty test with a CPU-expensive script that finds prime numbers – not a real benchmark, but something to give me an idea of how JRuby held up under a steady load of basic math operations. Details follow.

Disclaimer: I know this is not the optimal algorithm for finding prime numbers. The point here is not to find prime numbers, but to tax my CPU predictably. If you post a comment with the general intent of HEY FINDING PRIMES UR DOIN IT WRONG, I’ll just delete it.

The CPU is an Intel Core Duo in an old Mac Mini. The Ruby and JRuby versions under consideration were:

[~/project/jrubytest/primes]> ruby -v ; jruby -v
ruby 1.8.6 (2007-03-13 patchlevel 0) [i686-darwin8.9.1]
jruby 1.1.6 (ruby 1.8.6 patchlevel 114) (2008-12-17 rev 8388) [i386-java]

The code for the test was:

#!/usr/bin/env ruby
 
limit = (ARGV[0] || 100000).to_i
puts "FINDING first #{limit} PRIMES"
 
t0 = Time.now
primes = [2, 3]
counter = 2 # starting with 2 and 3
 
currnum = primes.last
while counter < limit
  currnum += 2
  primes.each do |p|
    break if (0 == (currnum % p)) and (p <= Math.sqrt(currnum.to_f))
    if p >= Math.sqrt(currnum.to_f)
      primes << currnum
      counter += 1
      break
    end
  end
end
 
puts primes.last
puts "Time: #{Time.now - t0}s"

The results showed just shy of a factor of 2 gain using JRuby (which is consistent with other tests I’ve run since writing this up):

[~/project/jrubytest/primes]> ruby ./primes.rb
FINDING first 100000 PRIMES
1299709
Time: 57.716853s
[~/project/jrubytest/primes]> jruby ./primes.rb
FINDING first 100000 PRIMES
1299709
Time: 33.885273s

These results were gotten after I’d made sure I had plenty of free RAM so I wasn’t I/O bound; the difference you’re seeing is all CPU. Note also that the timers are inside the Ruby code – they don’t account for JVM startup time.

Just for laughs, let’s see what the difference is with JVM overhead taken into account:

[~/project/jrubytest/primes]> date ; ruby ./primes.rb ; date ; jruby ./primes.rb ; date
Sun Feb 15 22:26:36 EST 2009
FINDING first 100000 PRIMES
1299709
Time: 54.065758s
Sun Feb 15 22:27:30 EST 2009
FINDING first 100000 PRIMES
1299709
Time: 34.427694s
Sun Feb 15 22:28:05 EST 2009

So startup overhead is about a second for JRuby. Not too shabby.

Part 2 of this post is going to cover some of the hairier tasks involved in:

• Subclassing Java classes and implementing Java interfaces.
• Packaging JRuby code in JARs with Rawr
• Calling code in those JARs from Java

More to come…

Hadoop is a framework (written in Java) that supports distributed computing – specifically Google’s MapReduce algorithm. It also comprises HDFS (the Hadoop Distrubuted File System), which allows you to redundantly store large quantities of data across multiple disconnected disks as if they were a single storage unit. I’ve used Hadoop at two jobs and at home, and it rocks.

It comes with a problem, though, which you may spot in the first paragraph: It’s written in Java. Now, don’t get me wrong – Java’s a great language. But developing software in Java with the most commonly used tools (Eclipse, Ant/Maven, &c) is a monumental pain (and you’re hearing this from an old Visual C++ hand). I’m the first to admit that I should shore up my skills with the Java tools, but even if I were better at it, getting a non-trivial Java project from zero to first runnable build is still about as complex as the invasion of Normandy, and the proliferation of XML config files is just inhumane.

Still, the performance and solidity of Java make it the right choice for a production Hadoop project. But what if you just want to kick around an idea or test an algorithm? Wouldn’t it be nice if you could do that in 2 hours instead of a day and a half? Wouldn’t it be nicer still if you could do it in your language of choice?

Hadoop Streaming has made me a very happy man. Any language that can take data from stdin and give it to stdout can be used to make Hadoop MapReduce jobs.

So of course, I’m doing mine in Ruby.

As an example, I’ll transliterate the trivial canonical Hadoop word counting example into Ruby. This example involves taking a large text, and counting the number of instances of each word it contains. The mapper takes in rows of text, and emits key-value pairs where the key is a word and the value is the number of times that word has occurred in a given row of text. It would look something like this:

word_count_mapper.rb:

#!/usr/bin/env ruby
 
STDIN.each_line do |line|
  word_count = {}
  line.split.each do |word|
    word_count[word] ||= 0
    word_count[word] += 1
  end
 
  word_count.each do |k,v|
    puts "#{k}\t#{v}"
  end
end

Already, we’re at our first non-trivial design decision, and it’s due to a significant design difference between Hadoop Streaming and regular Hadoop jobs: When you’re streaming, a single instance of your script will handle many pieces of input – it’s a bit like a filter-style Unix command (e.g., grep). A regular Hadoop mapping job is more firmly seated in the functional paradigm – each call to your mapper gets one line of data, with no knowledge of others. Because of this, some tutorials will suggest shortcuts such as keeping the hash-based accumulator outside the STDIN loop, and emitting all your rows of [word, count] pairs at the end of processing. There are advantages to this, but I’m going to stick with my code above, for two reasons:

1. If you maintain global state in your mapper, you’ll incur significant rework if you try to port the code to a more purely functional Java Hadoop mapper.
2. This could get damagingly memory-intensive on large data sets. Going functional and streaming everything straight to HDFS will add some storage cost, but for large jobs I’m much more worried about RAM than disk.

The purpose of the reduce step is to bring together all the per-line counts for each word, and reduce them to a single, global count per word. This is facilitated by the fact that Hadoop orders all the records by key before handing it to the reducer (where by default the key is everything before the first tab character – in this case, the word being counted.) My version of the reducer looks like this:

word_count_reducer.rb:

#!/usr/bin/env ruby
 
current_word = nil
current_count = 0
STDIN.each_line do |line|
  word, count = line.strip.split
  if word != current_word
    puts "#{current_word}\t#{current_count}" unless current_word.nil?
    current_word = word
    current_count = 0
  end
 
  current_count += count.to_i
end
 
puts "#{current_word}\t#{current_count}" unless current_word.nil?

The whole business with current_count is necessitated by another Hadoop Streaming-vs.-Hadoop quirk: In a regular Hadoop reducer, one call to the reducing function gets a collection of all rows associated with a particular key. When you’re streaming, you’re getting only one row at a time, but you’re guaranteed that the rows will be ordered on keys, and that the rows for a particular key will not be split up across reducer instances. Again, the temptation might be to place a single accumulator outside the main STDIN loop, but I’m going to stick with my strategy of lean runtime footprint and going straight to storage. The price for that is keeping track of which key you’re on.

I have placed the mapper and reducer in a folder called scripts in my $HADOOP_HOME folder. To run them against an existing file in HDFS, I do:

bin/hadoop jar contrib/streaming/hadoop-0.19.0-streaming.jar \
  -mapper scripts/word_count_mapper.rb -reducer scripts/word_count_reducer.rb \
  -file `pwd`/scripts/word_count_mapper.rb \
  -file `pwd`/scripts/word_count_reducer.rb \
  -input texts/my_text -output word_counts

…and results may be extracted from the word_counts folder in HDFS.

This is all there is to the trivial example. There’s more, of course – I’m currently using this technology to prototype a much more ambitious project that I will eventually port to Java (more on that later). While I’m finding that there is (at least subjectively) some performance overhead associated with streaming that I don’t see otherwise, it’s proving incredibly useful and speedy for testing out algorithms and designs.

Are you working with Hadoop Streaming? Do you have questions? Drop a line in the comments – I’d love to swap learnings.

The solution was quick & dirty (as most solutions are during Rails Rumble), but the results worked, and allowed us to qualify for judging and reach a respectable 28th place finish in the “Completeness” category.

Check out the details after the fold.

Design Constraints

Great Minds is a word game played by 2 or more players. The game is played in a virtual chat Room, containing one or more Players. Players can take various actions, such as submitting a word to the game, chatting, or changing their handle/username. The results of these actions must be transmitted to other players in the room.

Other constraints on the design included:

• No New Technologies: There are tools like Juggernaut that make messaging pretty easy, but we had no hands-on experience with them, and learning a new tool on a 48-hour schedule was deemed a risk to completing the project & qualifying for Rumble judging.
• Broadcast: We needed something more than a simple message queue like Beanstalk; the consumer of a message couldn’t remove it from the queue, as there might be other consumers needing the same message.
• Room State Reconstruction: Someone coming to the room for the first time (or leaving and returning) should be able to see the current room state, including all participants, recently played words, and recent chat.
• Replay: Though we didn’t get around to implementing it, we wanted to offer the ability to review a particularly funny game.

The first two constraints affected our design and technology choices; the second two demanded that the messages be persisted. (Alternatively, we could have saved the room state in some other way, but as long as we had the messages anyway, it seemed simpler to just keep them and reconstruct the state.)

Client Implementation

In the absence of a viable push solution like Juggernaut, we opted for a polling solution: Clients would periodically send a signal that:

• let the server know they’re still alive and connected,
• sent any new messages to the server, and
• requested any pending messages from the server.

On the client side, this meant that every message got packaged as JSON, stuck in a queue, and periodically collected and sent to the server as the optional payload of a keep-alive signal. With some help from Prototype, this was as simple as:

  function SyncMessages() {
          var data = Object.toJSON(MESSAGES_QUEUE);
          var url = '/room/sync_messages';
          var params = {
                  authenticity_token: AUTH_TOKEN,
                  messages: '[]',
                  //messages: Object.toJSON(MESSAGES_QUEUE),
                  last_message_id: LAST_MESSAGE_ID };

          new Ajax.Request(url, {
            method: 'post',
                  parameters: params,
            onSuccess: HandleMessages });
  }

  SyncMessages(); //start first on load
  var message_syncer = new PeriodicalExecuter(SyncMessages, 3);

HandleMessages() updates the DOM based on messages coming back.

The other feature to note is the LAST_MESSAGE_ID token above. Each client is responsible for keeping track of the last message ID it got. On the server side, message IDs are assigned sequentially, so all new messages are guaranteed to have a higher ID.

We could have kept track of each client’s last message on the server side, but this would have been much more complex – we would have needed to store data either in the session (which already held a fair amount of stuff) or in another table in order to keep track of who had received what.

Server Implementation

On the server side, messages were persisted in a table. This table would have relatively frequent inserts, somewhat more frequent reads, and no updates or deletions. For this reason, we went with the default MyISAM table architecture. (We had considered a design that would have updated the table with delivery status information and would have required InnoDB’s row-level locking, but in the end we opted for the simpler design.) The table looks like:

  class CreateMessages < ActiveRecord::Migration
    def self.up
      create_table :messages do |t|
        t.integer :message_type_id, :room_id, :game_id, :user_id
        t.string  :data, :user_handle
        t.timestamps
      end
    end

    def self.down
      drop_table :messages
    end
  end

It also has an index:

  class AddRoomIndexToMessages < ActiveRecord::Migration
    def self.up
      add_index :messages, [:room_id, :id]
    end

    def self.down
      remove_index :messages, [:room_id, :id]
    end
  end

Recall that the client keeps track of the last message it got, and periodically requests all messages since. This index speeds that request - which is just msgs = Message.find :all, :conditions => ['room_id = ? AND id > ?', session['room_id'], last_message_id], rder => 'id'. (Yes, I know that should be a named scope. I hadn't gotten around to learning those yet at the time of the Rumble.)

When the a keep-alive signal hits the server:

• any new messages attached to the signal are added to the messages table,
• any game mechanics related to those messages are executed (which may result in more messages being added to the table), and
• all messages for the caller's room with an ID greater than the caller's LAST_MESSAGE_ID are packaged as JSON and sent back down in the HTTP response.

A note: If you're in a room and you enter (for example) a chat message, there's actually a server round-trip involved (pushing the message to the server and getting the same message back as a response) before that chat message is shown in your browser. We worried about this, but not for long - the interactivity delay turned out to be minimal, and it was much simpler to keep it this way (every message is treated the same for all clients) than to create an exception (you update your DOM with your chat messages immediately, but your own chat messages are somehow filtered from your message stream).

What's Missing?

Obviously, this was a quick & dirty solution, tailored to the immediate needs of a Rails Rumble project. Here's what I'd do differently (and indeed, will do differently, as we have plans for the future of Great Minds - stay tuned):

• Security: We do use the Rails session authenticity_token, but there's still no protection from a server flood from someone who has a token, either by cranking down the polling interval or by fiddling with the LAST_MESSAGE_ID.
• Server Push: Polling is wasteful - requests are made whether or not there are any messages to send or receive. We plan to move to Juggernaut soon.
• Decoupling from Game Mechanics: In the first iteration, message receipt and delivery were too intimately tied to game mechanics, with Message objects being explictly constructed and followed by Message.save! calls. I've already started the process of teasing them apart in preparation for the move to Juggernaut, factoring this into a deliver_message() method.

Questions? Thoughts? Add a comment! I promise to get back to you.