01:13 <rtyler> HOLY LOL, I submitted a Spark job written in JRuby

01:21 <rtyler> oh interesting, there's some object serialization that has to happen to distribute some things across the cluster

01:22 <rtyler> which of course, means RubyBasicObject is going to be fun

02:16 <headius[m]> Ahh yeah that gets trickier

04:24 <rtyler> I think colin had some work for redstorm which I might be able to re-use here. I don't recall exactly how much wacky object serialization hacks we needed for Storm

04:24 <rtyler> hah

04:25 <rtyler> https://github.com/jruby-gradle/redstorm/blob/master/src/main/redstorm/storm/jruby/JRubyBolt.java#L22-L24

04:25 <rtyler> yes I have seen this movie before

04:31 <rtyler> headius[m]: I doubt you're still awake, but if you have any pointers or examples of creating a native Java lambda object from JRuby, that might help me along

15:52 <headius[m]> rtyler well lambdas are all just implementations of a single method interface, which we do automatically for procs and blocks passed out to Java

16:06 <rtyler> ah

16:11 <rtyler> headius[m]: watching a presentation of yours, I've now seen you speak enough to be able to tell what you've practiced versus not :P

16:11 <rtyler> the practiced ones are very good lectures :)

16:12 <headius[m]> Ha yeah, sometimes I do a little too much editing the day before

16:14 <rtyler> so with redstorm it's passing classes around and working with classes, with spark it _seems_ like it's doing something similar

16:14 <rtyler> digging to figure out where objects are serialized and sent along to spark workers in the cluster,

16:20 <headius[m]> It may be possible to set up some custom serialization that can negotiate Ruby runtime etc

16:26 <rtyler> much of what Spark seems to rely on are interfaces which implement java.io.Serializable

16:27 <rtyler> the key difference, as far as I can tell, between how their cluster operations work compared to Storm is that Storm would send whole classes to nodes to do work (spouts, bolts, etc)

16:28 <rtyler> whereas with spark, it appears they're serializing some runtime state and sending it along

16:52 <headius[m]> I just had a weird idea

16:53 <headius[m]> If we just serializes Ruby objects as a marshal dumped byte[] then it gets across fine and we just unmarshal from Ruby on the other side

16:54 <headius[m]> Depends how smart the serialized object is supposed to be

16:54 <rtyler> that does not seem unreasonable to me

16:54 <headius[m]> If it's just a data object it would hook up on other side with something that can understand it

16:55 <headius[m]> Or hell something like msgpack

16:55 <rtyler> now you're cookin' with gas

16:55 <headius[m]> I don't know how spark hooks up serialization thiugh

16:56 <rtyler> I sent a message off to the dev list because I'm not finding it right now either

16:57 <rtyler> I _think_ this might be it: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/java/JavaRDD.scala#L214 but I'm not sure

16:57 <headius[m]> Seems like it should be possible in any case to plug in a serializer which would just be Marshal

16:58 <headius[m]> Yay implicits

16:59 <headius[m]> Doesn't look like this cares what the data is

16:59 <headius[m]> Gathers up chunks of bytes and then hands it off?

17:01 <rtyler> headius[m]: what are you doing online, aren't you supposed to be trying beers or something on weekends? :)

17:13 <rtyler> headius[m]: I actually think this might be even easier: https://github.com/apache/spark/tree/master/core/src/main/scala/org/apache/spark/serializer

17:13 <rtyler> when creating the SparkContext, the serializer can be specified by tthe user

17:23 <headius[m]> That sounds promising

17:24 <rtyler> I <3 jruby-gradle, it's rare that I still enjoy using wacky things I've build many years on

17:24 <rtyler> making testing and building this super easy

17:27 <rtyler> muharharharhar, test failure in RSpec, now we're movin'!