<headius[m]>
hardcoded logic to dump compiled scripts to ~/.ir
<headius[m]>
enebo: I did run into some scope filename issues I'm working around for now
<headius[m]>
also realized we still need to be able to set scopes into compiled class in the case of jit, where the existing scope is already live and may have things like refinements on it
<headius[m]>
so the from-bytecode scope creation is now only done when building from script body
<enebo[m]>
headius: yeah I do have a patch which will get rid of actual errors but has a couple of dangling issues. I am incrementally applying pieces to try and get a cleaner solution with no errors
<enebo[m]>
headius: worst case end of day the Fs will just end up being issues to solve
<headius[m]>
is spec:compile green for you?
<headius[m]>
compiler
<enebo[m]>
with my patch? or in general
<headius[m]>
you said you'd gotten persist to pass
<enebo[m]>
I guess the answer is yes to both
<headius[m]>
ok
<enebo[m]>
yeah persist is fixed
<headius[m]>
may be seeing failures because of a file change I made
<enebo[m]>
I told you I had it all pushed back to runtimehelpers but I did not know how to do that for indy
<headius[m]>
ok
<headius[m]>
I'll finish that now then
<enebo[m]>
so it is literally just pushing those few things in that helper method through
<headius[m]>
but hey we got through a few scripts :-D
<enebo[m]>
well with that down it will only be File/FILE require_relative and probably the dynamic scope thing you are bypassing
<enebo[m]>
and two of those three will be fixed soon...
<enebo[m]>
I think it is possible there are IRScope related access in unexpected places yet but it will not be via method/block/jit so maybe it won't matter
<enebo[m]>
getIRScope itself is not inherently evil...it is how all interpreted stuff works so it is mostly making sure we never need it in JIT/AOT
<headius[m]>
it is EVIL
<headius[m]>
ok what do I need to do for metaclass?
<headius[m]>
I need to review that conversation
<headius[m]>
o I see
<headius[m]>
o i c
<headius[m]>
so from there down it's using StaticScope but JIT still tries to pass in IRScope
<headius[m]>
gotcha
<headius[m]>
hmm line number should be in static scope?
<headius[m]>
ick
<headius[m]>
metaclass body doesn't have protocol?
<enebo[m]>
well that could be an opt but today those are passed as constants
<headius[m]>
is this the only one?
<enebo[m]>
the only what?
<enebo[m]>
File and line can eventually be gotten from StaticScope once we can guarantee it is correct but I am only fixing file today
<enebo[m]>
once file is fixed I will remove it from all class/methods and defer back to staticscope
<enebo[m]>
same for line but I have not tried to verify line at all on staticScope
<enebo[m]>
but metaclass only needs staticscope and maybeusingRefinements
<enebo[m]>
your no protocol observation is similar to me noticing inliner may have only be using boxed call path
<headius[m]>
ok
<headius[m]>
CompileIRNoProtocolMethod wants a line number
<enebo[m]>
Yeah once file is done I should probably remove from CompiledIR*
<enebo[m]>
yes
<enebo[m]>
file/line/staticscope for that
<headius[m]>
I have no line!
<headius[m]>
NO LINE!
<headius[m]>
staticscope has no line
<enebo[m]>
you pass it from JVM visitor to indy
<enebo[m]>
and indy to the helper
<headius[m]>
that's fine
<enebo[m]>
heh so is there a question still?
<enebo[m]>
more or less any reference to IRMetaBlerrggh in that IRRuntimeHelper needs to be passed from JVMVisitor through indy (or gotten from something in bootstrap)
<enebo[m]>
I believe my confusion with that one was the extra handle transform
<enebo[m]>
prepareBlock did not do that extra stuff
<headius[m]>
nah it's ok I can embed all the way from JVMVisitor through to indy
<headius[m]>
we'll need to decide on the best way to verify cached IR and class
<headius[m]>
I'm not sure e.g. python does anything more than checking last modified
<enebo[m]>
yeah it is a good question
<headius[m]>
aha I may have a tricky one now
<headius[m]>
eval needs parent IRScope to be around?
<headius[m]>
seems like that's what's blowing up now
<enebo[m]>
oh! uh oh
<enebo[m]>
honestly I do not think it is much more that variable access
<headius[m]>
yeah I wouldn't expect so but this is killing our internal enumerator.rb
<headius[m]>
because we eval some method defs
<enebo[m]>
It is a wrinkle which needs to be examined
<headius[m]>
probably a good justification for removing eval in kernel though
<enebo[m]>
yeah although the odds of rubygems using it are probably fairly high
<enebo[m]>
but removing evals would not be a bad thing if we can internally anyways since it kills a parse
<enebo[m]>
I am too in-thought on this file stuff to examine what it will take for eval not use a parent one
<enebo[m]>
I do feel that is a bit of a rabbit hole since it could be multiple nested closures and see down more than one scope
<enebo[m]>
with that said an eval present in a file could just force full serialization of the file
<enebo[m]>
that would likely be a quicker path to making this run more stuff
<headius[m]>
I'm sure it would blow up on more though
<enebo[m]>
yeah it probably will at least look at local vars but that one is not very important
<enebo[m]>
it could be any number so long as it is not the same number as another closure at that level
<enebo[m]>
and I am not even sure that is true
<enebo[m]>
actually I doubt that is true
<enebo[m]>
It is nice to see CLOSURE_4 and know it is the first lexically seen closure
<enebo[m]>
but not important for running
knu has quit [Quit: Reboot...]
ur5us has joined #jruby
<headius[m]>
I'm replacing these with block evals and define_method for now
<headius[m]>
I'm not pressing on eval for the moment because it goes on to use the parent scope for other stuff
<enebo[m]>
yeah more than variables?
<headius[m]>
well I don't know... IRClosure passes parent scope to parent constructor which sets lexicalParent and that's accessed dozens of places
<headius[m]>
I mean it does have to create a scope for eval, so...I'm not sure how to resolve that
<headius[m]>
hopefully it's possible to compile and execute an eval without a parent IRScope :-)
<enebo[m]>
yeah I am not fully sure but weirdly I just ran into that getNextClosureId() problem
<enebo[m]>
It is in eval for jit
<headius[m]>
until this is fixed probably will have to turn scope setting back on
<enebo[m]>
I think it is fine if eval is present to just do old school full serialize
<enebo[m]>
or I should say it is better than nothing
<headius[m]>
yeah but I can't
<enebo[m]>
we treat it same way as too big method in a file
<headius[m]>
unless you mean non-jit IR serialize
<enebo[m]>
I mean how AOT worked before we started
<headius[m]>
it would basically invalidate the whole file from bytecode jit right now
<enebo[m]>
yes it will
<headius[m]>
ok
<enebo[m]>
I think we will have some slowness from that but overall hopefully most ruby files do not have eval
<headius[m]>
well hopefully it's a temporary limitation
<headius[m]>
I'd say worst case we have to stand up skeleton IRScope
<enebo[m]>
I am a little confused though..I am doing spec:ruby:fast:jit and so we could for JIT still put the live IRScope in there
<enebo[m]>
if AOT it is different obviously
<enebo[m]>
"in there"...meaning some static field with a live IRScope
<headius[m]>
JIT is fine because it always sets live scopes because it has to (and it has them)
<headius[m]>
AOT is standing up StaticScope only
<headius[m]>
there's no IRScope anymore after AOT
<enebo[m]>
ok then I am confused why this is broken since I am not doing AOT
<enebo[m]>
I am going to end up having to rediff and then reapply my changes since my stash pop ended in a tiny conflict (which then tried to commit a zillion files)
<headius[m]>
what are you doing?
<headius[m]>
this breaks any main script with eval in it too
<headius[m]>
but I don't think you're doing that
<enebo[m]>
no I was just trying to get my file changes working but when I pulled that last commit it detected a conflict when I did a stash pop
<enebo[m]>
so now I have this dirty tree which is a combination of your last commit and my changes
<enebo[m]>
but your last commit is why I am seeing these errors
<headius[m]>
is this the IRClosure line?
<enebo[m]>
I may actually go back one commit and be green then apply my changes
<enebo[m]>
yeah from IREvalScripty -> IRClosure
<headius[m]>
hey almost everything at boot AOTed ok
<enebo[m]>
spec:ruby:fast:jit it shows up
<headius[m]>
one of them is has a giant hash though
<headius[m]>
uri/common.rb
<headius[m]>
everything else in kernel and rubygems compiled
<enebo[m]>
but your last push broke some non AOT stuff
<headius[m]>
yes I'm not surprised
<enebo[m]>
yeah I am going to work off previous commit for now to get file working
<headius[m]>
you are probably seeing top-level scripts fail because they try to AOT?
<headius[m]>
I'm not sure what you're running
<headius[m]>
oh spec:ruby:fast
<headius[m]>
yeah anything that spawns would blow up
<enebo[m]>
ruby:fast:jit
<headius[m]>
probably
<headius[m]>
I'll check it
<enebo[m]>
yeah so likely you are correct...let me look at trace
<enebo[m]>
you know non-capture evals should be fixable without a parent scope by changing logic a bit
<headius[m]>
might be a bug in block_given?
<enebo[m]>
Actually maybe not so simple...we look down the scope until we find the variable but evals will just create for the special slot if it cannot fins it
<headius[m]>
yeah I don't know how many evals that will cover though
<headius[m]>
it wouldn't have helped any of those in enumerator.rb
<enebo[m]>
well all method ones should be fine
<enebo[m]>
captures are not uncommon but not all of them
<enebo[m]>
I will shut up on that idea for now though. I feel like I need to study this a bit to figure it otu
<enebo[m]>
ok with my changes pushed test:mri only failed with lazy enum
<headius[m]>
hmm strange...setting scopes should have eliminated these eval fails
<headius[m]>
method ones definitely capture
<headius[m]>
for the name of the method at the very least
<headius[m]>
oh wait no you're right
<headius[m]>
they are interpolated into the string
<headius[m]>
ahh I see... my run unconditionally uses the blank staticscope which wipes out that top level
<headius[m]>
enebo: did you see spec fast jit running green?
<headius[m]>
I see some failures still but they don't seem related to my work
<enebo[m]>
headius: hmm I just ran non-jit a few minutes ago and the lazy enum stuff was broken
<enebo[m]>
[115/122] TestLazyEnumerator#test_zip_nested = 0.00 s
<enebo[m]>
22) Error:
<enebo[m]>
If you pushed for that then I should repull
<headius[m]>
I mean before this recent stuff
<headius[m]>
was it ever green
<headius[m]>
yeah I have tons of stuff that doesn't pass but seems unrelated
<enebo[m]>
oh spec:ruby:fast:jit had like 15-16 EF
<enebo[m]>
they were not very important problems but not completely green no
<enebo[m]>
have you ever seen massive retests on test:mri:core:jit from the parallel testing stuff
<headius[m]>
oh ok
<headius[m]>
good
<headius[m]>
not massive retests but always some
<headius[m]>
so I got 18F8E now
<enebo[m]>
This run is redoing 2673
<headius[m]>
there are some lazy failures still from my change
<enebo[m]>
well the good news is I do not see any file issues now
<enebo[m]>
and I just remembered I have println + printStackTrace in there still
<headius[m]>
yeah ok
<enebo[m]>
pushed with skipci
<headius[m]>
ok I have more fixes locally to push
<headius[m]>
I will push with CI once I do a quick sanity check
<enebo[m]>
So personally I think for evals we will need to be able to materialize IRScopes as simplest solution
<headius[m]>
eval should work again for target script with this push
<headius[m]>
hmm ok
<headius[m]>
that's a tricky one then
<headius[m]>
we need to be able to restore them individually, rather than as a tree
<enebo[m]>
well it is but we have a longer term plan to bring back IRScopes like how you did StaticScope with handles
<headius[m]>
yeah
<enebo[m]>
but for that to work we need to change IRpersist a lot
<enebo[m]>
which is ok
<enebo[m]>
but it is not for the next week
<headius[m]>
right...need to be able to just persist what we can't get out of StaticScope
<enebo[m]>
I think we accept this limitation of evals though
<headius[m]>
once we can do that we can lazily restore IRScope as needed
<enebo[m]>
yeah
<headius[m]>
and if we can do that then we only need to restore IRScope up to frame scope
<headius[m]>
for an eval
<enebo[m]>
and honestly IRScope is for building/compliling so this is a reasonable limitation
<headius[m]>
it won't need to restore anything but structural info
<enebo[m]>
and when we thinking about AOT loaded code if we ever want that to be able to speculatively optimize in IR we need to be able to this for any scope
<headius[m]>
yes
<headius[m]>
ok lazy is green so I'm pushing
<headius[m]>
bbiab
<enebo[m]>
yeah having worked on that yesterday I can say I have a lot more knowledge about persist paged into my brain
<enebo[m]>
I will work until at most 5pm today and not tomorrow
<enebo[m]>
Monday I think regardless of where we are we make a draft of talk
<enebo[m]>
I am fairly bullish on all this right now. We did a ton of runtime decoupling and it is nearly totally done. The only significant goal we wno't get done is incremental parent up loading of IRScopes on demand
<enebo[m]>
but that is ok. We will just drop back to 9.2.9 persistent loads for those files which contain evals. Not the end of the world
<headius[m]>
Yeah, I need to unify the logic for loading from class to support both
<headius[m]>
The pure bytecode logic is really shoehorned in at the moment
<enebo[m]>
ah yeah and you plan on working Sunday
<headius[m]>
But hey, it appears to be working
<headius[m]>
Eval will bite us for gem list since it probably will need to evaluate specs
<enebo[m]>
yeah but it was always going to be a cost
<headius[m]>
my brute force elimination of evaled defs
<headius[m]>
pushed that fix
<headius[m]>
MRI should be ok except for the END thing you're looking at
<headius[m]>
looks like END fix would fix -Ptest too
<enebo[m]>
well that is the only thing I think it fixes
<headius[m]>
aha and I was wondering if I broke load wrapping
<headius[m]>
it appears I did
<enebo[m]>
the third one I think is about load + wrap
<headius[m]>
yeah I was being lazy and didn't do wrapping in AOT script
<enebo[m]>
I do not see how that END commit broke anything
<headius[m]>
most recent END commit just fixes parent... but previously I cloned scope
<headius[m]>
I would suspect the earlier change more
<enebo[m]>
cloned end scope itself?
<enebo[m]>
just the parent
<headius[m]>
end was getting parent's static scope
<headius[m]>
so I changed it to get a clone of parent's scope because it can't share a scope for AOT
<enebo[m]>
static scope should not be important at all for this ... I don't think so anyways
<headius[m]>
maybe not that stuff then
<headius[m]>
ok
<enebo[m]>
I mean it is fairly clear if it shows up as getClosures() something which adds to that is adding it when it should go oh isEnd() and say nope
<enebo[m]>
10ec7bef9fb0a910c14b8583e6bdd25ed521919d looks suspect but then why would this have passed up til now
<enebo[m]>
and it does not really look suspect
<enebo[m]>
you duplicate the staticscope so it ends up being needed
<enebo[m]>
ok I have a fix and it is doing what I said it should be doing (but seemingly never was before)
<headius[m]>
ok
<headius[m]>
I am fixing load wrap
<enebo[m]>
your fix for lazy did not seem to work or it changed the result to make another test fail?
<headius[m]>
odd that this didn't fail any MRI tests
<headius[m]>
hmm really?
<headius[m]>
that should have been it
<headius[m]>
I'll have a look while this sanity-checks
<headius[m]>
bleh maybe I missed two pastes
<headius[m]>
this is just an inspect difference
<headius[m]>
yup
<enebo[m]>
10ec7bef9fb0a910c14b8583e6bdd25ed521919d did actually cause the END issue in -Ptest but I really don't see how. I guess probably something magical with not setting ScoeType before
<headius[m]>
yeah probably never saw the END scope at all before
<headius[m]>
now it sees it as a closure
<enebo[m]>
I mean yeah that is the exact issue
<enebo[m]>
but I only see staticscope changes
<enebo[m]>
IRScope analysis does not look at staticscope
<enebo[m]>
looks like a few wrap errors still exist
<headius[m]>
ok wrap is broken for AOT scripts with or without this branch
<headius[m]>
it doesn't pass the proper self through to execution
<headius[m]>
man these entry paths from Ruby to code need a total overhaul
<headius[m]>
run and parse and load variants have a ton of duplicate code and several paths are not even needed
<headius[m]>
well that will fix some AOT failures somewhere
<enebo[m]>
HAHAHA OMG
<headius[m]>
wrapped AOT load is working now and wrapped interpreted load is repaired
<headius[m]>
this should fix remaining spec failures
<enebo[m]>
I still don't know why this fails BUT in both before and after commit they both fail to actually full compile on that pass
<headius[m]>
aha
<headius[m]>
well that's a clue
<headius[m]>
that END scope screws everything up
<enebo[m]>
well it does or it doesn't but the assert failing changes something because of that staticscope change
<headius[m]>
enebo: everything runs fine from precompiled up until specification.rb uses eval
<headius[m]>
so that's another big leap forward
<headius[m]>
oh and that uri file with a giant hash still doesn't load
<headius[m]>
giant hashes I can probably figure out a way around...right now all hashes are created by pushing EVERYTHING onto JVM stack, which is why I had to cap it at like 128
<headius[m]>
we need chunking
<headius[m]>
we need chung king
<enebo[m]>
pushed a fix for END
<headius[m]>
we need chongqing
<enebo[m]>
Presence of END is already a massive deopt so I am not performing in the presence of an END. The pass was not really running on it anyways
<enebo[m]>
So it is not like I eliminated an optimization...I am pretty perplexed though. I really don't see how adding a duplicated staticscope would change anything here
<headius[m]>
maybe we should actually try to compile it as an ensure after the trip
<headius[m]>
yeah I dunno
<enebo[m]>
yeah I also am somewhat game on removing FOR via similar simulation
<enebo[m]>
END is easier I think
<headius[m]>
uri/mailto.rb has an eval too
<headius[m]>
used by gem -v
<enebo[m]>
so you plan on working tomorrow and making fallback AOT work when new AOT is either too big or contains and eval?
<headius[m]>
yeah that would be a good item for tomorrow
<headius[m]>
or just don't even try to AOT
<enebo[m]>
I am not working tomorrow
<headius[m]>
we will still need source on filesystem for load searches anyway
<headius[m]>
is there a way to determine if a file has an eval in it other than walking everything?
<headius[m]>
I could disable AOT for those files right now
<enebo[m]>
search of all scopes for flags == USES_EVAL is simple enough to systemize it
<enebo[m]>
which would be a little different than how too big methods would not AOT but it is just a scope search for that flag
<headius[m]>
yeah disabling is the quick fix
<headius[m]>
unification is better
<headius[m]>
but unification just saves parse+compile at a cost of deserializing
<enebo[m]>
I am mostly thinking about this for time and not what we should do