<GitHub122>
[rubinius] YorickPeterse pushed 2 new commits to master: http://git.io/g5aOnA
goyox86 has joined #rubinius
benlovell has quit [Read error: Connection reset by peer]
benlovel1 has joined #rubinius
<yorickpeterse>
Hm, throughput rate of 860 jobs per minute atm
<yorickpeterse>
notbad
<yorickpeterse>
New Relic does some weird grouping/evening out when zooming out to an entire week, so it's still a bit hard to compare what we had with MIR
<yorickpeterse>
* MRI
<goyox86>
yorickpeterse: With rbx?
<yorickpeterse>
Yes
<yorickpeterse>
Hm, seems a past batch on MRI had a peak throughput of 505 jobs/minute
<yorickpeterse>
With average timings being higher this, at least so far, shows how easy it is to scale when you have proper threads
<yorickpeterse>
that is until you hit database conneciton limits and the likes
<goyox86>
yorickpeterse: Are you running sidekiq?
houhoulis has quit [Remote host closed the connection]
<yorickpeterse>
Nope
<yorickpeterse>
We use a custom queueing setup using AWS SQS
<GitHub171>
rubinius/master 0cec02e David Boot: fix typo in authors
benlovel1 has quit [Ping timeout: 256 seconds]
goyox86 has quit [Ping timeout: 264 seconds]
<yorickpeterse>
mpapis: why does RVM fuck up shebangs, even when rubygems-bundler is not installed? That is, why does it add the rvm executable hooks to my env shebangs?
<yorickpeterse>
Because this breaks _every_ update I've done so far of Ruby. Every update I made sure rubygems-bundler is _not_ installed and every update I have to re-generate shebangs because RVM fucks up the paths
<yorickpeterse>
oh for fucks sake, it's another gem
<yorickpeterse>
argh
meh` has joined #rubinius
<yorickpeterse>
Fuck this, I'm going to nuke RVM out of my CI this week
<yorickpeterse>
I've had it
<yorickpeterse>
I've wasted far too much time on this already
benlovell has joined #rubinius
goyox86 has joined #rubinius
goyox86 has quit [Ping timeout: 245 seconds]
flavio has quit [Ping timeout: 265 seconds]
flavio has joined #rubinius
heroux has quit [Ping timeout: 265 seconds]
DanielVartanov has quit [Ping timeout: 265 seconds]
JohnBat26 has quit [Ping timeout: 265 seconds]
DanielVartanov has joined #rubinius
JohnBat26 has joined #rubinius
heroux has joined #rubinius
<mpapis>
yorickpeterse, well you could use chruby, or if you decide to still use rvm you can tell rvm not to install rubygems-bundler / executable-hooks, and well rubygems rejected a PR to provide this simple functionality
<yorickpeterse>
The latter I've already done to make sure it doesn't also install executable-hooks and bundler-unload
<yorickpeterse>
euh what the hell, this one daemon is now leaking memory like crazy on rbx
<yorickpeterse>
Right, time to dump those metrics somewhere so I can actually somewhat see what's going on
DanielVartanov has quit [Remote host closed the connection]
<headius>
brixen: nice post
<|jemc|>
brixen: speaking of software and development infrastructure imitating biological systems, you may find http://hintjens.com/blog:74 interesting
<headius>
|jemc|: funny how a colon with a number in a URL and I immediately think it's a port...my first thought was "why is that blog on port 74?"
<|jemc|>
headius: heh
<headius>
yorickpeterse: I've wondered that too, but they don't take up *that* much space compared to a nontrivial app
<|jemc|>
headiis: it's to sweep under the NSA radar, of course
<headius>
|jemc|: yeah there ya go...terrorists are well known for only using port 80
<yorickpeterse>
I thought they only used port 22
<yorickpeterse>
errr 25
<bennyklotz>
brixen: very nice post :)
<yorickpeterse>
brixen: I take it the auto updated doesn't bother you in non interactive environments?
<yorickpeterse>
also I hope that can be disabled, not too fond of my processes phoning home in production
<headius>
well, the example had a (N)ever option right?
<headius>
we had a gem to update JRuby for a while, similar to rubygems...not sure if people used it though
<yorickpeterse>
headius: that still requires an initial response
<headius>
indeed
<brixen>
|jemc|: very cool
<brixen>
|jemc|: got a twitter handle so I can attribute you? :)
<brixen>
yorickpeterse: of course, in a noninteractive env it will just block forever, staring blankly at the wall
<brixen>
bennyklotz: thanks :)
<yorickpeterse>
brixen: Ah nice
<|jemc|>
brixen: twitter handle is @JoeEliMac but I never use it - the blog owner's twitter handle is @hintjens
<|jemc|>
also, was just re-reading through the hintjens blog post - I love the part where he compares meetings and group consensus to shared mutable state :P
<yorickpeterse>
heh
<|jemc|>
wrapped in a big slow mutex of course
<yorickpeterse>
hm, this daemons leaks much faster the more threads are running
<yorickpeterse>
|jemc|: no no, the knowledge is shared in parallel
<yorickpeterse>
it's just stored in MongoDB, so it vanishes rather quickly
<brixen>
meetings, much like shared mutable state, are incredibly powerful
<brixen>
also, much like shared mutable state, we fuck them up really badly
<yorickpeterse>
Hm, heap dump shows a lot of various things, but nothing stands out
<dreinull>
first step to help people contributing to rbx is to move away from jekyll. Dependencies are a joke.
<yorickpeterse>
e.g. 20MB of Rubinius::Tuple instances, but not sure how that's going to help
<brixen>
yorickpeterse: tons of things are Tuples, so that makes it hard
<yorickpeterse>
yeah, second entry is String...which is pretty much always there
<brixen>
dreinull: which dependencies? and what would you switch to?
<yorickpeterse>
I'll generate another dump and compare
<yorickpeterse>
I have to say though, regardless of the pain I can finally see light at the end of the tunnel
<brixen>
yorickpeterse: one thing the new heap tools will do is understand composite structures much better
<brixen>
yorickpeterse: yay!
<yorickpeterse>
brixen: we need to fix evan's histo tool too though
<yorickpeterse>
rails girls students didn't really do that :/
<dreinull>
brixen, oh, just found a gemfile. I used jekyll and kramdown which resulted in installing x gems on a slow system
<brixen>
yorickpeterse: that was a huge task for them
<dreinull>
brixen, I'll fix the docs and a typo in your last post. Good writeup though.
josh-k has joined #rubinius
<brixen>
yorickpeterse: it was awesome they wanted to work on it, but they were 12m under water from the start
<yorickpeterse>
brixen: correct, pretty sure they learned a lot, but I can't help but feel the end result was a bit underwhelming
<brixen>
dreinull: ah ok
<dreinull>
brixen so how do i generate html these days? jekyll build?
<brixen>
dreinull: if you bundled, jekyll serve will give you the site
<yorickpeterse>
dreinull: if you don't want to use Jekyll, try nanoc
<brixen>
dreinull: github builds it for us, no need for checking in html
<yorickpeterse>
unless we're talking about the rbx website?
josh-k has quit [Read error: Connection reset by peer]
<dreinull>
brixen no more building these days? didn't know that.
<yorickpeterse>
dang it! I actually could've used my dtrace allocation hooks here
<brixen>
dreinull: I'm also moving to Travis CI building so I can move the site to https
<yorickpeterse>
curses, I need to clean that up
josh-k has joined #rubinius
<yorickpeterse>
argh, I set my threshold to 400MB for another dump, of course _now_ it leaks slowly
<yorickpeterse>
though it leaked about 150MB already in a few minutes time
<yorickpeterse>
90MB in 40 seconds, so roughly 2,25MB per second
<yorickpeterse>
I swear if this is my own code leaking...
<yorickpeterse>
ah well, it's going to be an interesting few months
<yorickpeterse>
rbx in production, oga in production, hopefully getting oga to be used by the aws-sdk too, probably bunch of crashes
<yorickpeterse>
not a lot of useful info, though the AWS objects are interesting
<yorickpeterse>
note that this was taken _while_ polling SQS (= dump in a separate thread), so some of that data might actually just be used (and not kept around by accident)
<yorickpeterse>
meh, I'll run this tomorrow using my dtrace allocation hooks, see if that finds anything
* yorickpeterse
had dtrace fever
josh-k has quit [Remote host closed the connection]
josh-k has joined #rubinius
jnh has joined #rubinius
josh-k has quit [Ping timeout: 256 seconds]
<yorickpeterse>
brixen: regarding our current JIT, do we have any docs on what it can/can't optimize, and maybe a bit on how that works? Or is it all just hidden in C++?
max96at is now known as max96at|off
max96at|off is now known as max96at
dberlinger has joined #rubinius
<bennyklotz>
ls
<bennyklotz>
ups :D
DanielVartanov has joined #rubinius
DanielVartanov_ has joined #rubinius
<brixen>
bennyklotz: ENOTAFILESYSTEM :)
<brixen>
that's what Pry should return, too
<brixen>
but I digress
<brixen>
yorickpeterse: no, not really
DanielVartanov has quit [Remote host closed the connection]
DanielVartanov_ has quit [Remote host closed the connection]
<yorickpeterse>
dang it
<yorickpeterse>
so I had this idea
<yorickpeterse>
"I had a dream" to quote a certain man
<yorickpeterse>
and I think this is basically what we already discussed in the past
<yorickpeterse>
But basically I see Rbx not just providing a JIT, but a generic native code compiler (with the cool abbrev NCC). This toolkit would be fully exposed to Ruby as well as being used by Rbx itself for the JIT, and possibly an AOT compiler
<yorickpeterse>
basically Ruby comes in, magic happens, native code comes out
<brixen>
with drums
<brixen>
a drum roll even
<brixen>
and magic
<brixen>
yes, magic happens with drums
<brixen>
:)
<enebo>
tides go in and tides go out...
<yorickpeterse>
In all seriousness though, I was thinking of 80-something % Ruby, 20% C++ (depending on how much wrapper crap LLVM will need)
<brixen>
yorickpeterse: yeah, you're thinking in the right direction
<yorickpeterse>
for a change :P
<yorickpeterse>
so hence the question what our JIT currently does, trying to sort of figure out what it would have to do exactly
<yorickpeterse>
besides "make things webscale"
<yorickpeterse>
I was thinking of just compiling Ruby "1+1" to LLVM, but that alone will probably be a few weekends worth of work
<yorickpeterse>
(as a start)
<brixen>
yorickpeterse: well, the biggest thing our JIT does now is painstakingly generate asm IR with verbose C++ code while walking over bytecode instructions
<brixen>
yorickpeterse: the question of how much C++ we need to *interface* with LLVM may reduce to zero
<yorickpeterse>
that's like saying "All our rocket does is fly to the moon"
<brixen>
we're probably not going to implement register allocation and instruction selection & scheduling in the near future
<yorickpeterse>
I have no clue what that specifically means, besides the terms themselves
<brixen>
our rocket is a little short of the moon at the moment but mars is the objective, or jupiter
<brixen>
hah
<yorickpeterse>
well my evil plan was to replace our rocket
dberlinger has quit [Quit: Leaving...]
<brixen>
yorickpeterse: it may make more sense after thursday
<yorickpeterse>
heh
<yorickpeterse>
keeping me in suspsense here
<brixen>
not intentionally, you get a preview :p
<yorickpeterse>
yeah like 1 hour ahead :P
<yorickpeterse>
when I'm on a train
<brixen>
hah
<yorickpeterse>
with wifi that's worse than when you'd plug a router in your butt
<yorickpeterse>
either way, I have to see where I start
<brixen>
my wifi is pretty good but as soon as it hits godamn comcast, it's shit
<yorickpeterse>
I feel "Starting simple" isn't really possible with LLVM
<brixen>
yorickpeterse: it's very steep, yes
<brixen>
we can improve that, though
<yorickpeterse>
heh
<yorickpeterse>
"Dear Santa, for Christmas I'd like humane LLVM documentation"
<yorickpeterse>
ah what the hell, time to read those docs
<brixen>
yorickpeterse: one thing that would help a ton is to write the questions you have down
<|jemc|>
headius: or some tool that uses that?
<brixen>
bennyklotz: heh
<headius>
it does generate itself, though, so it may be possible to emit ffi code for llvm proper too
<yorickpeterse>
brixen: oh yeah, I typically write down _a lot_ in my notebook when going through new things
<yorickpeterse>
I was going to put all that on Github a while ago but haven't gotten to it yet
<headius>
|jemc|: ffi-gen was an earlier attempt...the ffi-clang stuff is a lot more robust, and generates itself using itself
<brixen>
yorickpeterse: in a format like gist you can share with me
<yorickpeterse>
brixen: sure
<brixen>
yorickpeterse: cool
<yorickpeterse>
Typically what I do on paper is write the literal questions, e.g. "Why does X not do Y." then followed by a whole bunch of arrows, ideas, etc
<yorickpeterse>
I found that to be quite a nice way of learning things while also keeping a commit log basically
<brixen>
yorickpeterse: totally
<yorickpeterse>
no more "what the heck was I thinking back then"
<brixen>
feel free to just scan a page
<yorickpeterse>
Nah typing it in is not really a problem
<brixen>
if only it were easier to share images
<yorickpeterse>
actually, it's easier than getting my scanner set upp
<yorickpeterse>
* up
<brixen>
you have a phone that takes pics, no? :p
<yorickpeterse>
well, sort of
<brixen>
who uses a scanner anymore?
<yorickpeterse>
It runs Android on a firefox phone, the camera is total garbage
<brixen>
oh, that sucks
<brixen>
borrow someones? :)
<yorickpeterse>
I can use my DSLR, but that's no better than my scanner
<yorickpeterse>
either way, I'll type it in
<brixen>
man magic : best man command or bestest man command
<yorickpeterse>
I like "man man" as it (or "man man man") is a thing you can say in Dutch as a form of disapproval
<yorickpeterse>
So something like "man man man dat was slecht" basically translates to "Oh man that was bad"
<|jemc|>
I'm sick of this patriarchal society - a woman can do anything a man can do - even print instructions for using programs
<|jemc|>
`sudo ln -s /usr/bin/man /usr/bin/woman`
<yorickpeterse>
|jemc|: people make woman
<yorickpeterse>
as in, the command
<yorickpeterse>
I believe it was as badly executed as you'd expect
<yorickpeterse>
also was written in Node I think
<yorickpeterse>
ok searching for that is nigh impossible
<|jemc|>
yeah, I didn't even try for that very reason :P
<yorickpeterse>
now I must find it
<|jemc|>
so my gut instinct made me think of symbolhound.com, then quickly realized that wouldn't help either in this case
<yorickpeterse>
So I know it was somewhere on Reddit
<yorickpeterse>
but searching this on there is even worse than HackerNews
jfredett-w has quit [Read error: Connection reset by peer]
<yorickpeterse>
ha
<yorickpeterse>
npm.org is a church website
<yorickpeterse>
Ah yes, it's npmjs.org
<yorickpeterse>
doh
<yorickpeterse>
oh hey, Groupon backed off on their Gnome claim
<yorickpeterse>
Yeah, it kinda comes down to "But we have open source too!!"
<brixen>
not an apology
<|jemc|>
some of my best friends are open
<yorickpeterse>
like, you literally search "Gnome" on Google and the Gnome.org website is the first result
<yorickpeterse>
either they need to fire all their lawyers, or stop being asshats
<brixen>
|jemc|: hah, you captured it
<brixen>
that's very much one part of it
<brixen>
yeah, I was disappointed to read that post, but I'm not sure how to call it out without writing a post that touches on things like harassment in tech
<brixen>
the overlap is huge
<brixen>
I suppose I could queue a post
<brixen>
anyway, I chose to defer a snarky tweet for once :)
<brixen>
I like the "cranky; irritable" definition for snarky
<yorickpeterse>
"We big corporate make software, we take trademarks. Together we make big moneys"
<yorickpeterse>
On the other hand, this shows that you're not always on your own
<yorickpeterse>
That is, if you shout loud enough
<brixen>
nobody likes a bully
goyox86_ has joined #rubinius
<yorickpeterse>
True, the scary thing is it makes you wonder how often this happens to those unknown enough that they can't raise their voice like this
<yorickpeterse>
and thus basically just have to take the shit
<yorickpeterse>
e.g. if some company were now to claim "Oga" as a name I'd literally be fucked, nothing I could do even if I had a trademark on the name
<|jemc|>
yorickpeterse: literally?
* |jemc|
runs
<yorickpeterse>
|jemc|: depends on the company
<yorickpeterse>
if it's the Italian mafia, maybe
<yorickpeterse>
Then again I don't really worry about the big evil corporations coming after my stuff
<yorickpeterse>
If anything they'll probably come after me because things don't work :D
lbianc_ has joined #rubinius
lbianc has quit [Ping timeout: 245 seconds]
lbianc_ is now known as lbianc
lbianc_ has joined #rubinius
lbianc has quit [Ping timeout: 245 seconds]
lbianc_ is now known as lbianc
JohnBat26 has quit [Ping timeout: 264 seconds]
lbianc has quit [Read error: Connection reset by peer]
lbianc has joined #rubinius
goyox86_ has quit [Ping timeout: 264 seconds]
<yorickpeterse>
Hm I wonder, did anybody ever build some tool to visualize what Ruby code is running along with what data?
<yorickpeterse>
wait that's basically a debugger
<brixen>
it's usually called a debugger, but that's a great question
havenwood has quit [Remote host closed the connection]
<yorickpeterse>
well, I mean something that shows code in one column, the values in the other
<yorickpeterse>
so you don't have to step through every frame, dump all locals, etc
* yorickpeterse
is quickly hacking that atm
<headius>
GUI debuggers usually do that
<headius>
oughta be possible at console with curses or something