Comments on: Hadoop Streaming for Rapid Prototyping of Distributed Algorithms http://www.kickasslabs.com/2009/01/04/hadoop-streaming-for-rapid-prototyping-of-distributed-algorithms/ We ♥ code. Thu, 26 Jan 2012 05:50:23 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Brad http://www.kickasslabs.com/2009/01/04/hadoop-streaming-for-rapid-prototyping-of-distributed-algorithms/comment-page-1/#comment-670 Brad Thu, 26 Jan 2012 05:50:23 +0000 http://www.kickasslabs.com/?p=132#comment-670 Zahide: You're correct about the reducer. One reducer instance can receive many keys, and you must store some state to detect when your reducer gets a new key. However, all values for a single key will pass through a single reducer instance. (It wouldn't be MapReduce if that weren't the case.) The times I've tried it, Streaming was slower. I was also using a 1.8 series Ruby, which is not the fastest of interpreted languages. JRuby or another more performant scripting language will give you better results; I happened to use Ruby because I wanted to prototype an algorithm quickly. The fact that Hadoop Streaming might not be as fast as writing your own Hadoop framework .jar files doesn't make it a bad thing - if you're doing something that doesn't require extreme performance, it might be appropriate to write MR jobs in the language of your choice, either to re-use working code or just to save on developer time (which tends to be more expensive than cluster time, for small and medium clusters). Zahide: You’re correct about the reducer. One reducer instance can receive many keys, and you must store some state to detect when your reducer gets a new key. However, all values for a single key will pass through a single reducer instance. (It wouldn’t be MapReduce if that weren’t the case.)

The times I’ve tried it, Streaming was slower. I was also using a 1.8 series Ruby, which is not the fastest of interpreted languages. JRuby or another more performant scripting language will give you better results; I happened to use Ruby because I wanted to prototype an algorithm quickly.

The fact that Hadoop Streaming might not be as fast as writing your own Hadoop framework .jar files doesn’t make it a bad thing – if you’re doing something that doesn’t require extreme performance, it might be appropriate to write MR jobs in the language of your choice, either to re-use working code or just to save on developer time (which tends to be more expensive than cluster time, for small and medium clusters).

]]> By: Zahide http://www.kickasslabs.com/2009/01/04/hadoop-streaming-for-rapid-prototyping-of-distributed-algorithms/comment-page-1/#comment-669 Zahide Thu, 26 Jan 2012 03:56:48 +0000 http://www.kickasslabs.com/?p=132#comment-669 Thanks for the useful suggestions Brad. I have two questions about streaming. First, so NOT like the Java Reducer (in which every Reducer gets values associated with a single key), in streaming one reducer might get values from multiple keys. That's something we need to take care of when we write the reducer code. Is it possible that values for single key might spread over different reducers ? Second, are you saying that in most cases, streaming is NOT as efficient as MapReduce written in java ? Thanks Thanks for the useful suggestions Brad.
I have two questions about streaming.
First, so NOT like the Java Reducer (in which every Reducer gets values associated with a single key), in streaming one reducer might get values from multiple keys. That’s something we need to take care of when we write the reducer code. Is it possible that values for single key might spread over different reducers ?

Second, are you saying that in most cases, streaming is NOT as efficient as MapReduce written in java ?

Thanks

]]> By: Brad http://www.kickasslabs.com/2009/01/04/hadoop-streaming-for-rapid-prototyping-of-distributed-algorithms/comment-page-1/#comment-454 Brad Thu, 08 Sep 2011 19:24:23 +0000 http://www.kickasslabs.com/?p=132#comment-454 Pig/Hive is one option, but for raw performance, it's tough to beat a custom-written Hadoop job in Java. This entails writing Java classes that inherit from Hadoop's Mapper and Reducer classes, and packaging them as a .jar that gets distributed to your cluster. Naturally, speed is not the only concern - and you can always buy/rent more machines to get more speed. A lot depends on your team's skills and what you're paying for developer time vs. machine time. Pig/Hive is one option, but for raw performance, it’s tough to beat a custom-written Hadoop job in Java. This entails writing Java classes that inherit from Hadoop’s Mapper and Reducer classes, and packaging them as a .jar that gets distributed to your cluster.

Naturally, speed is not the only concern – and you can always buy/rent more machines to get more speed. A lot depends on your team’s skills and what you’re paying for developer time vs. machine time.

]]> By: jo http://www.kickasslabs.com/2009/01/04/hadoop-streaming-for-rapid-prototyping-of-distributed-algorithms/comment-page-1/#comment-447 jo Tue, 06 Sep 2011 14:30:05 +0000 http://www.kickasslabs.com/?p=132#comment-447 Thanks Brad ; will check up on JRuby. Reg #2, we have a Hadoop cluster with enough machines, so shouldn't be a big problem. In pt #3-Switch languages, you mention "regular Hadoop jobs" -- hope you meant using stuff like Pig/Hive. Thanks Brad ; will check up on JRuby. Reg #2, we have a Hadoop cluster with enough machines, so shouldn’t be a big problem. In pt #3-Switch languages, you mention “regular Hadoop jobs” — hope you meant using stuff like Pig/Hive.

]]> By: Brad http://www.kickasslabs.com/2009/01/04/hadoop-streaming-for-rapid-prototyping-of-distributed-algorithms/comment-page-1/#comment-438 Brad Sun, 04 Sep 2011 14:00:41 +0000 http://www.kickasslabs.com/?p=132#comment-438 Jo - There was a significant performance hit; part of it was Ruby, part of it was just the fact of leaving the JVM and pushing all this data across processes. There are three things you might do about it: 1) If this is performing as well as you need it to, don't sweat it. (But do keep an eye on your expected data throughput growth.) 2) Add more machines. That's what the horizontal scalability is for, after all. 3) Switch languages. You could move up to Ruby 1.9, or JRuby, or Java or C/C++ (though if you're going to do the latter, might as well go with regular Hadoop jobs). If you're going to do this, do some profiling to determine that you're going after the correct bottleneck. If your job chain is long - that is, if you have many scripts that would need replacing - pick one, profile it, swap it with an equivalent in the language you think will do better and profile that. If you do wind up switching languages, I'd recommend taking a deep look at JRuby. The last time I played with JRuby in a Hadoop context was a couple of years ago, but I was seeing order-of-magnitude improvements in speed (when I put the -server flag in my #! line). Good luck! Jo -

There was a significant performance hit; part of it was Ruby, part of it was just the fact of leaving the JVM and pushing all this data across processes.

There are three things you might do about it:

1) If this is performing as well as you need it to, don’t sweat it. (But do keep an eye on your expected data throughput growth.)

2) Add more machines. That’s what the horizontal scalability is for, after all.

3) Switch languages. You could move up to Ruby 1.9, or JRuby, or Java or C/C++ (though if you’re going to do the latter, might as well go with regular Hadoop jobs). If you’re going to do this, do some profiling to determine that you’re going after the correct bottleneck. If your job chain is long – that is, if you have many scripts that would need replacing – pick one, profile it, swap it with an equivalent in the language you think will do better and profile that.

If you do wind up switching languages, I’d recommend taking a deep look at JRuby. The last time I played with JRuby in a Hadoop context was a couple of years ago, but I was seeing order-of-magnitude improvements in speed (when I put the -server flag in my #! line).

Good luck!

]]> By: jo http://www.kickasslabs.com/2009/01/04/hadoop-streaming-for-rapid-prototyping-of-distributed-algorithms/comment-page-1/#comment-434 jo Sun, 04 Sep 2011 03:33:33 +0000 http://www.kickasslabs.com/?p=132#comment-434 Hi Brad, Is there a real performance hit when using Hadoop Streaming? . We're planning to use this in a production env and wanted to know if the whole processing would be slower if MR is done in Ruby vs Java. Thanks, jo Hi Brad,
Is there a real performance hit when using Hadoop Streaming? . We’re planning to use this in a production env and wanted to know if the whole processing would be slower if MR is done in Ruby vs Java.
Thanks,
jo

]]> By: Brad http://www.kickasslabs.com/2009/01/04/hadoop-streaming-for-rapid-prototyping-of-distributed-algorithms/comment-page-1/#comment-354 Brad Tue, 22 Mar 2011 14:39:16 +0000 http://www.kickasslabs.com/?p=132#comment-354 Vic - I'm happy I could help. Vic – I’m happy I could help.

]]> By: vic http://www.kickasslabs.com/2009/01/04/hadoop-streaming-for-rapid-prototyping-of-distributed-algorithms/comment-page-1/#comment-353 vic Tue, 22 Mar 2011 08:19:40 +0000 http://www.kickasslabs.com/?p=132#comment-353 Thanks a lot, you just saved me many hours of hadoop reading :) Thanks a lot, you just saved me many hours of hadoop reading :)

]]> By: omiplirlChorn http://www.kickasslabs.com/2009/01/04/hadoop-streaming-for-rapid-prototyping-of-distributed-algorithms/comment-page-1/#comment-347 omiplirlChorn Mon, 30 Aug 2010 19:05:55 +0000 http://www.kickasslabs.com/?p=132#comment-347 why not... why not…

]]> By: Carmelo http://www.kickasslabs.com/2009/01/04/hadoop-streaming-for-rapid-prototyping-of-distributed-algorithms/comment-page-1/#comment-201 Carmelo Wed, 10 Mar 2010 15:15:45 +0000 http://www.kickasslabs.com/?p=132#comment-201 This is the mainn reason I read www.kickasslabs.com. Fascinating posts. This is the mainn reason I read http://www.kickasslabs.com. Fascinating posts.

]]>