<GitHub120>
jruby/master 4d11849 Thomas E. Enebo: HEH. reenable new specialized instr. Our persistence is depending on these specialized operation enums :(
<GitHub136>
jruby/master 444c2d6 Thomas E. Enebo: Double fault. Fixed infinite loop if fixnum call instr but canot be emited by JIT specially as fixnum call
<headius>
I'd wager that Crystal on JVM would smoke their LLVM version
<lopex>
headius: but that's just semantics changes
<lopex>
the whole thing is just an appeal to syntax
<headius>
well, I'm more talking about how they have a crappy GC and no parallelism right now and they say "we'll add that later"
<lopex>
or I'am wrong
<headius>
history has shown it's pretty damn hard to "just add" parallelism to a runtime after the fact
<headius>
lopex: you're not wrong
<lopex>
and pretty hard to bootstrap when you only have a semantics
<headius>
it's basically Mirah but with full type inference
<lopex>
concurrency is hard in mutable world
<lopex>
you need to make hard choices
<chrisseaton>
I should be trying Crystal on Sulong
<chrisseaton>
Did someone write a Ruby C extension in Crystal - am I remembering that correctly?
<lopex>
any llvm on sulong
<havenwood>
chrisseaton: Yeah, they have an example on in the crystal repo. I embedded mruby in a crystal extension just for fun, to take the inception one deeper.
<cremes>
enebo: ha. not sure it would even register on a profiler now… the DB part of the run was so fast…
<enebo>
cremes: how big is that box?
<enebo>
cremes: you must have 8 cores at least?
<cremes>
8 cores and 46GB RAM allocated to this VM. all IO is to SSDs also.
<enebo>
cremes: I guess I am just guessing based on how big removing a lock did
<enebo>
ok so proping Array literals is quite a bit more difficult than I had hoped
<enebo>
when we build something like lasgn we return the value
<enebo>
and not the variable
<enebo>
but if I prop array then the lvar returns the array itself and not a temp var
<enebo>
so we get this situation where a call may just use the array as an argument
<enebo>
so we end up duplicated the same literal array
<enebo>
To making fixing this simple I think I would need to propagate whether something was involved in assignments higher up in the AST chain and then use a tempvar + copy in those cases
pitr-ch has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<headius>
cremes: oh man, that's excellent
<headius>
JRuby wins again!
<headius>
so 7x faster than MRI
<cremes>
headius: yep
<headius>
well that makes my day :-)
<cremes>
it freaked me out. I thought something went wrong!
<cremes>
but yes, it makes my day too
<headius>
hopefully we can start to whittle away the heap size with some of these improvements
<headius>
hoping to land my small object specializations for 9.1.3
<headius>
cremes: well keep your issues coming...you're a good use case for us to make improvements :-)
<cremes>
headius: will do. sometimes I’m surprised that I run into these issues and no one else does. Honestly I think that most folks just don’t open issues. Oh well.
<cremes>
I’ll find another juicy one for you soon.
<headius>
that's probably true
<headius>
or they're running mostly-isolated threads that never see each other
<headius>
like for a webapp
<cremes>
maybe
hobodave_ has quit [Quit: Computer has gone to sleep.]
<cremes>
I probably have another juicy perf issue in this same use-case. I’ll just have to dig it out.
<enebo>
who uses sqlite3 in production though?
<headius>
might be interesting to throw some profiling at it now and see what is the top item
<cremes>
there was a slow part (phase 2) that took about 50m.
<headius>
--sample is probably the simplest one
<enebo>
cremes: your use case is less common I think in that regard
pitr-ch has joined #jruby
<cremes>
enebo: that’s probably true too
<enebo>
cremes: and so I remember…you use it because you make lots of separate dbs?
<headius>
you're using sequel...it probably wouldn't be hard to switch to one of the embedded JVM databases
<headius>
they're darn quick and don't require any native libs
<headius>
and they're ACID, if you care about that
<headius>
I'd be surprised if sqlite3 is does any of those
<headius>
well...efficiently anyway
<cremes>
enebo: correct, lots of separate DBs. I don’t want every little write/update to trigger a backup of a single 300GB file. I’d prefer 100 3GB files (or whatever).
<cremes>
headius: I haven’t switched DBs because I don’t want to learn Yet Another DB’s quirks.
<cremes>
i’m comfortable with the sqlite3 repl.
<cremes>
I did play with H2 a long time ago though.
<enebo>
cremes: thanks…that’s what I remembered
<cremes>
regardless, I want to thank headius for acting like a dog with a bone and staying on top of that issue all of these months. It’s fixed and I’m happy!
<jeremyevans>
cremes: One thing to keep in mind is jdbc-sqlite doesn't even half-support date/time type conversion, date/time column values are always returned as strings
cprice404 has quit [Remote host closed the connection]
<cremes>
jeremyevans: yep, I am well aware. That’s okay for my use-case.
<cremes>
jeremyevans: while you are here, let me thank you for a great project in sequel.
<cremes>
I remember interacting with the original author way back when… was his/her name Sharon?
<cremes>
anyway, great project. my “go to” for all DB handling.
<headius>
cremes: I'm glad it turned out to be so simple
<headius>
now we just need to get them to release it
<cremes>
haha, yeah. i updated that ticket so hopefully that will light a fire.
<headius>
great, thanks
<cremes>
headius: you haven’t posted to your blog in a while. It might be useful to bring everyone up to speed on the 9k update, new performance stuff in the pipeline, etc. I don’t have nearly the time to lurk in IRC like I used to so I know I miss out on a lot of the scuttlebutt.
<headius>
yeah I'm going to do some posts soon about the work from this week
<enebo>
I ran into an issue with Array literal copy reduction
<enebo>
I think builder will need to change substantially to address it
<enebo>
individual built nodes need more knowledge of their parents to fix it
<subbu>
worth figuring out what it will give you before doing a lot of changes.
<enebo>
subbu: well atm it will only give us copy removal of Array
<enebo>
subbu: probably also for Hash
<enebo>
in my case of gem list I remove 11k array copys of 81k instrs total
<subbu>
but, i mean ... hotspot can handle those kind of simple global copy proagation .. the qn. is how many instrs. will be removed to push the scope below the hotspot inlining/opt. threshold.
<enebo>
so it is not a small number in that particular use case
<enebo>
subbu: well I guess that depends on the program right?
<headius>
lopex: Array could pack begin or size into flags if it were long
<subbu>
ya
<enebo>
subbu: some programs will get pushed below and threshold and some won't
<headius>
or both with a flag to say whether to look there
<headius>
e,g. if they're both under X bytes
<enebo>
subbu: I am not saying all copy’s are bad and we should trade-off impl complexity vs reward
<enebo>
subbu: but I think the smaller our programs are the better startup time and memory should get
<enebo>
subbu: or pining data in temp vars
<enebo>
subbu: or hotspot threshold checks
<subbu>
right.
<enebo>
with all that said I do not have a concrete answer on what is worth it
<lopex>
headius: but now it's in mri thinking territory
<enebo>
I just realized why one spec failed quite recently
<headius>
I'd like to get more explicit instructions for using frame/scope too so we might be able to have those optimize away rather than all these nasty flags
<headius>
i.e. we might have a CallWithFrame that explicitly consumes frame, and then don't have to look at what it's calling to know if frame can DCE away
<headius>
lopex: well, the old tricks are the best tricks
<headius>
travis needs a better way to set up IRC notifications
<headius>
I hate the idea of people having to patch their fork for that
<headius>
asarih: do you know of any way to limit IRC notifications to just pushes on jruby/jruby?
<subbu>
headius, but, you cannot add those instrs. at the call site without knowing the call target.
<subbu>
the target scope has explicit push/pop of scope & frame in the entry bb right now.
<lopex>
headius: if it could pack begin + size that would be something
<headius>
yeah
<subbu>
i mean push/pop in entry/exits
<headius>
arrays > 16 bits of size have to be pretty damn rare
<lopex>
but you need a check for that then
<headius>
subbu: right, but I mean at build time we'd say "ok, this is a frame-consuming call" and make a CallWithFrame
yfeldblum has joined #jruby
<subbu>
but, how do you know that?
<enebo>
headius: you mean native calls?
<headius>
we only do it based on name now anyway
<headius>
but it's done during flags calculation
<headius>
enebo: yeah native calls that need frame
<headius>
I'm not saying we do any more than name-based guessing about frame, but make it explicit in the instruction that the frame is used
<enebo>
so if we are in a method scope and we call block_given?
<headius>
yeah, stuff like that
hobodave has joined #jruby
e_dub has quit [Read error: Connection reset by peer]
<headius>
block_given would depend on frame, so frame creation would not DCE
johnsonch is now known as johnsonch_afk
e_dub has joined #jruby
<headius>
dunno, maybe it's just moving the deck chairs around to let DCE remove frame versus never adding it
<enebo>
headius: but this would be for a very small number of methods right?
<enebo>
headius: like 12 or so
<headius>
yes
<enebo>
if that
<lopex>
headius: hmm, so as with that flags you'll need a framework for dealing with thresholds
<headius>
this would also help us eliminate useless loads of module and staticScope
<headius>
because we'd say which instructions explicitly need them
<enebo>
headius: do we add a flag to IRScope for that?
<headius>
we wouldn't need it to be in flags
<headius>
ACP would just add both frame and scope, but it would DCE away if nothing used them
<enebo>
headius: well yeah but adding more instrs is not super desirable
<headius>
hmm, sounds wasteful maybe?
<headius>
or there would be NO ACP at all
<enebo>
headius: well for stuff like persistence it is a pain
<headius>
we just assume frame is needed, and when it's not it goes away
<enebo>
headius: addvisitors and crud like that
<headius>
enebo: so how about another operand to Call then?
<headius>
for frame
<enebo>
headius: I guess if we did do this and wanted instrs we could maybe add as RuntimeHelper and not really calls
<enebo>
headius: after all we are not treating as calls really at that point anyways
<headius>
this would also set the stage for passing frame through call stack and never pushing it
<enebo>
well I have liked that idea for a long time :)
<headius>
which would eliminate GEB
<enebo>
I just cower at the commit :)
<headius>
I think this is a good idea :-)
<enebo>
but pushing frame through eliminates this need altogether too
<headius>
well it requires this
<enebo>
I do not think it would eliminate GEB
<headius>
there has to be an instr for getting the frame into the call
<headius>
sure it would...no push/pop means no finally
<headius>
the geb is mostly for protocol, right?
<enebo>
we still need to have an edge to that code and n ensures need to execute in the same scope
<headius>
ensures don't run in GEB
<headius>
they get their own blocks and exception-handling
<headius>
GEB is for whole-method protocol
<enebo>
oh yeah they have a weird name don’t they
<headius>
so if there's no protocol, there's no GEB
hobodave has quit [Quit: Computer has gone to sleep.]
<enebo>
but we do not need to have frame if we just pass it everywhere in IR
<enebo>
err to all calls
<headius>
if we pass it everywhere, we need to have it :-)
<enebo>
not as an IR concept
<headius>
yeah but what are you going to pass?
<headius>
in a frameless method
<enebo>
null?
<enebo>
:)
<headius>
you're lame
<headius>
stand in the corner
<headius>
we'd still need intelligence whether to pass null
<headius>
I think my idea could be done without any new instructions, or just a minimal set
<headius>
Call has an optional operand Frame
<headius>
if frame is consumed, the protocol lives through GEB
<headius>
if frame is not consumed, protocol and GEB go away
<headius>
and then inlining gets simple too because frames know which calls they go to and might just disappear
<headius>
simpler I mean...wrt framing
<enebo>
but if we detect the methods at build time we do not need this in the call do we?
<headius>
we do if we want the IR to see data flow of frame
<enebo>
we can know whether frame will exist in scope
<headius>
which it needs to be able to eliminate it
<headius>
this also could be used to defer frame creation until it's actually needed
<enebo>
I am not following you mean if we have a provably dead path we can kill frame?
<headius>
we do it manually, statically right now
<headius>
I want IR to do it
<headius>
simplifies other things
<headius>
and if we get to the point of knowing exactly *what* gets consumed out of the frame, we can start doing partial frames
<enebo>
so we mark named calls with something to indicate they require a frame
<headius>
like block_given? would only need a frame to hold block
<headius>
right
<enebo>
then passes will determine whether it is really needed
<headius>
I think it would be worth it for the potental to eliminate GEB alone
<headius>
that adds a ton of code for even simple methods that need frame
<headius>
I should correct myself...it would still be a static optimization, but within the realm of IR
<enebo>
I always wish that frame was not a thing in IR personally
<headius>
we wouldn't have manual code all over to check if a frame is needed
<enebo>
I wished we could splinter that apart to what we really need to check
<enebo>
like visibility
<headius>
sure, that was my second point
<enebo>
did you write the second point above?
<headius>
visibility would be an operand for e.g. a call to "public"
<enebo>
I think I missed it
<headius>
"and if we get to the point of knowing exactly *what* gets consumed out of the frame, we can start doing partial frames"
<enebo>
oh that’s what you meant by partial
<headius>
we actually do have that information already
<enebo>
ok partial made me wonder if you just meant part of the method would stand up a frame only if reached or something
<headius>
I don't just blanket native methods with "frame = true"...they all say exactly what they consume and whether they read or write it
<headius>
ahh that was lazy framing
<headius>
that's there too
<headius>
this is all possible but I'm pretty sure it would be easier if IR just did it
<headius>
and it doesn't require adding more than a few instrs
<enebo>
It might not require adding any more instrs
<headius>
might not
<enebo>
maybe more operands
<enebo>
“special” ones like %current_module
<headius>
yeah
<enebo>
I guess if all of this is truly static from build time you can do this in two ways via flags on scope and via extra info on instrs
<headius>
all of this combined could mean we eliminate frame altogether even for cases that need it, by having some other thread-local mechanism for individual frame fields
<headius>
back to JRuby 0.8.3 framing
<enebo>
the former is more of a pain during generation but you can say I have this method and it requires visibilty
<enebo>
yeah I would love to blow apart frames
<enebo>
back to sphagetti stacks
<enebo>
if IR could help simplify managing that then it would be a win for sure
<headius>
it matches MRI semantics better too
<enebo>
I wish GEB was not called GEB
<subbu>
headius, enebo i haven't followed your conversation ... but, ACP adds frames / scopes if it determines the scope needs it. so, it is already conditional. if you have more info to add to that flag computation, you can update it.
<subbu>
but, i am not following this new instr. and passing frame into call business.
* subbu
was reading election news
<headius>
subbu: I'm thinking that if frame were just another operand to be consumed, the call protocol would just DCE away without us having to calculate those flags
<headius>
and we could start splitting up the frame and only using what we need
<headius>
which would reduce cost of populating and clearing it
<subbu>
but, you need to know what calls in the scope are going to need it.
<subbu>
are you saying there is information available about calls and their targets and what need them and what don't?
<subbu>
if so, that information can be added to the flag computation and not emit them in the first place.
<headius>
yes
<subbu>
why add them and run dce?
<headius>
I suppose the instructions that prepare and pop frame could just aggregate better information
<headius>
like we know block_given? only reads block from frame, so if that's the only thing that needs frame we only populate block
<subbu>
in the past we have talked about splitting up frame into into its constituents ..
<headius>
right
<subbu>
so that we know what part of the frame is required ... but, we couldn't determine how much it might buy us.
<subbu>
but, that is still interesting.
<headius>
is that better to do by having frame instr be smart or by having instrs for frame fields?
<subbu>
so, worth revisiting .. i agree.
<headius>
yeah
<enebo>
yeah I am all for splitting up frame
<headius>
nothing I'm going to do today
<enebo>
Adding instrs or operands is less clear to me
<headius>
hell, I want the opposite of enebo...but I'm probably leaning toward having a JVM bytecode IR I can emit rather than directly emitting bytecode
<headius>
have thought about using that Soot project
<headius>
bytecode API with an IR and optimization passes
<enebo>
headius: that is an interesting idea but completely different idea :)
<headius>
I'm full of it
<headius>
ideas are cheap...I should make a project that just comes up with ideas and never implements anything
lanceball is now known as lance|afk
<enebo>
our instr set at least is supposed to give us ruby semantics we can leverage…although I am unsure how well that has worked out in practive
<enebo>
block_given? is an instr now alrteady
<subbu>
sorry .. was distracted IRL :)
<enebo>
headius: we should just sub block_given? with the instr and not emit it as a call at all
<enebo>
got a motor
enebo has quit [Quit: enebo]
<subbu>
oh, enebo left.
<headius>
yup, party's over
<subbu>
:)
<subbu>
ok.
<subbu>
he is off like clockwork around 5ish.
Aethenelle has quit [Quit: Aethenelle]
<subbu>
anyway ... we can revisit the frame splitting and see what makes most sense.
<headius>
oh yeah, Tom's a 9-5er
<headius>
I'm more of a whenever to whenever guy
<subbu>
i think i am in between you two .. :) not quite 9-5 but not quite whenever-to-whenever either.