<headius>
enebo: you build with 8 for day to day work?
<headius>
I'm having some weird effect where I think I'm building with 8 but it's picking up those ByteBuffer.clear methods added in 9
<headius>
ugh wtf
<headius>
it's using 9 for maven even though 'java' is pointing at 8
<enebo>
I am using 8 yeah
<enebo>
I almost feel like I need some java version randomizer so I am constantly using a different one
<headius>
ugh, it's passing
<headius>
oh wait not on branch
sgeorge has joined #jruby
<headius>
enebo: we may have to accept the reality of release builds requiring both an 8 JDK and a 9+ JDK
<headius>
the API changes they make are really difficult to avoid without bootstrapping off 8
<enebo>
headius: so long as we can regularly build off of 8 and use it daily then the only requirements is releasing with 9 right?
<enebo>
I am happy largely using 8 for day to day over 9+ for a loooong time
<headius>
yes, only requirement would be for release
<lopex>
next lts is 11 right ?
<headius>
yes
<headius>
that's why I want to get back to JPMS work
<lopex>
and that cds thing ?
<lopex>
how does it apply to jruby ?
<lopex>
enebo: I'm still not seeing the artifacts deployed, though I have seen the uploads completed successfully for staging
<enebo>
lopex: did you close them from sonatype then release them?
<enebo>
lopex: or you just mean for a snapshot?
<lopex>
enebo: did what ?
<enebo>
lopex: you need to log into oss.sonatype and in the web ui close the artifacts you want to release. If they successfully close then you can use the ui to release them
<enebo>
lopex: this is a safety net for not doing things properly like signing since once the artifact is really published you cannot change it
<headius>
lopex: and CDS/AOT but that's a longer term thing too
<headius>
how does it apply? CDS?
jmalves has joined #jruby
jmalves has quit [Read error: Connection reset by peer]
jmalves has joined #jruby
<kares>
enebo: not really an option for a full Rails app, maybe some parts but those parts also changed in Rails
<kares>
hopefully I can easily profile out just the rendering parts I want and it gives a clue
jmalves has quit [Ping timeout: 252 seconds]
<lopex>
enebo: jeeze this thing is slow
sgeorge has quit [Remote host closed the connection]
<lopex>
enebo: oh, so I close multiple stating artifacts at once
sgeorge has joined #jruby
<lopex>
enebo: ok, signature thing
<lopex>
lol how clueless I was about it
sgeorge has quit [Read error: Connection reset by peer]
sgeorge has joined #jruby
jmalves has joined #jruby
<headius>
enebo: I think I was right about there being some binary incompat in the new stuff
<headius>
appears that most of these errors are due to jnr-enxio now using an "invalid argument" for some native read operation
<headius>
I'm going to try to isolate what change it is that caused the breakage and undo it
<headius>
I thought these would still be binary compat but I forgot some constants might not have been generating before
<enebo>
headius: yeah I wondered if we would fall over to something generic if something was not defined but I am not sure I can still articulate what scenario that would be
<enebo>
kares: RG server is an excellent real world use case for us too. Not a ton run it but everything acknowledges it as a usefule real world app
<headius>
I'd love to see RG.org run JRuby too :-D
<headius>
we have a lot of fragile logic around hiding and unhiding errno results inside IOException
<headius>
JDK does it, we do it to conform to Channel API, but not consistently doing this translation is an issue
sgeorge has quit [Read error: Connection reset by peer]
sgeorge_ has joined #jruby
sgeorge_ has quit [Remote host closed the connection]
xardion has quit [Remote host closed the connection]
xardion has joined #jruby
Eiam has joined #jruby
<lopex>
enebo: so after cliking release it's done ?
<lopex>
headius: hey there was a problem with pool.sks-keyservers.net
<lopex>
I could recv-key from it, but that http layer didn work
<lopex>
and sonatype uses that
<lopex>
I had to send to ubuntu key server
<lopex>
and that fixed the problem
<lopex>
thye query a bunch of them
<lopex>
*they
sgeorge has joined #jruby
<headius>
enebo: this jnr-constants update is becoming a major headache
<headius>
most of this stuff had not been regenerated in years and now there's all these weird issues coming up because the enums were manually changed before
<enebo>
headius: can you see actual errors in generation?
<headius>
errors?
<headius>
I'm not positive but I believe these errno issues are a knock-on effect from some enums changing
<headius>
and I discovered a moment ago that jnr-posix was using the Windows "LastError" constants directly
<enebo>
headius: yeah I mean if the enums changed values that is what I mean by errors since the manually entered ones were working?
<headius>
I'm trying to patch up the issues
<headius>
I have no evidence that changed values are the problem yet
<headius>
so far it seems more likely that it's binary incompat due to enums that got reordered
<enebo>
oh you mean the enum of 2 is something else from last compilation
<enebo>
so anything using it would need to recompile against it?
<enebo>
you have said binary incompat a couple of times but I am unsure what you specifically mean
<headius>
Java enums are binary-compatible if you only ever add values to the end of the list
<enebo>
headius: yeah I was just making sure you just mean that
<headius>
right
<headius>
I thought we fixed most of these so they didn't reorder but still having problem
<headius>
so the error coming out if jnr-enxio is an EINVAL during the read, which according to man page means the fd is negative
<enebo>
headius: So was the sort one of the bigger issues?
<headius>
jnr-enxio throws IOException with the platform's description for EINVAL and we're not handling that in our IOException-to-Errno conversion in JRuby
<headius>
that's why it ends up a weird SystemCallError
<enebo>
ah
<headius>
so jnr-enxio needs to actually throw something we can pull an Errno enum out of first of all
<headius>
and second of all why is this raising EINVAL suddenly
<enebo>
heh this is a bit of a mess if we need to depend on same order of enums
<enebo>
I guess we could diff the main commits per platform and rearrange them so they are in same order with new entries at the end? Seems like a lot of work for what I feel is a design flaw in the library
<headius>
well, that's how enums work...I don't know how we would get around it
<headius>
it would be even worse if they were static final int
<headius>
may be that platform constants are a poor fit for enums
<headius>
I also wasn't aware there were users (just jnr-posix so far) of the internal enums
<enebo>
yeah but adding them onto the end is a solution right? We just need to look at old values and new values
<headius>
haven't audited JRuby to see if we're using any platform-specific jnr-constants enums
<headius>
adding to end us a solution
<headius>
is
<enebo>
we just need to ... heh
<headius>
that's what I did before pushing this
<headius>
I moved all the things that were new to the end...it was mostly Errno that changed
<enebo>
but things still broke
<headius>
yeah but I don't know why
<enebo>
so maybe something missed or something new interfering
<headius>
the errno for read seems valid but I don't know why it is suddenly einval
<headius>
it is actually the right errno number coming from native for that error
<headius>
so it's not something broken in Errno
<headius>
EINVAL is also used to specify that you passed a bad flag in somewhere, so possibly an open flag that changed value or enum moved or something
<enebo>
errno is right but enum value for that error is right or wrong?
<headius>
it is correct
<headius>
I don't think any Errno values changed
<headius>
only new ones added
<enebo>
Do we have code where we lookup value in some table and assign if unknown
<enebo>
I could see something like that suddenly breaking as we would now have a more precise value and we don't know how to deal with it or something like that
<headius>
hmmm
<headius>
d is attached to an object which is unsuitable for reading; or the file was opened with the O_DIRECT flag, and either the address specified in buf, the value specified in count, or the current file offset is not suitably aligned.
<headius>
oops that's "fd is attached..."
<headius>
from read(2) man page
<enebo>
O_DIRECT
<enebo>
hmm constants to the left of us constants to the right of us
<headius>
unsuitable for reading...so possible an O_RDWR enum got borked, or something's passing O_DIRECT now when it shouldn't
<headius>
we open the fd for jnr-enxio separately, within Jruby
<enebo>
headius: I forgot about the +1 thing we saw the other week
<headius>
jnr-enxio doesn't do the open
<enebo>
where all enums were off by one
<headius>
yeah
<enebo>
that was only for a few things
KeyJoo has joined #jruby
<enebo>
but I felt like one was for some fd READ thi9ng
<headius>
yeah that was for one that wmeissner turned into a bitmask but it's not a bitmask
<enebo>
you could try older release and print out the fd constant value perhaps and compare against now
<enebo>
it sounds like you just figured something out though
<enebo>
probably wrong value on fd to read causing proper error message and we are handling the error ok
<headius>
only Windows actually has an O_BINARY flag at open(2) level
<enebo>
yeah but we have two constants with those two names and I cannot tell how BINMODE is toggled based on code like the above
<headius>
well binmode probably is unix term for it too
<headius>
but "rb" doesn't exist as a mode on unix does it?
<enebo>
I am confused but I thought b will force ascii-8bit AND no newline/cr garbage
<enebo>
headius: in any case if you set it to 0 and nothing fails then perhaps my thoughts don't matter
<headius>
yeah 0 is what MRI uses all over when it's not defined
<headius>
because it's a bitmask
<enebo>
I am mostly just looking at ModeFlags.BINARY and the interchanging of that and BINMODE are not super clear to me
<headius>
BINMODE from where?
<enebo>
yeah that makes sense if it is windows only
<enebo>
OpenFile
<enebo>
getModeFlagsAsIntFrom(fmode)
<enebo>
It is possible it does not matter if O_BINARY is only significant on windows
<enebo>
I just thought it also was ASCII-8BIT which would affect all platforms
<enebo>
Although it may set that encoding right away too
<enebo>
these flags may not be used for determining encoding
<headius>
the binmode concept in MRI does default to BINARY or ASCII-8BIT encoding for the resulting strings
<enebo>
'"b" Binary file mode Suppresses EOL <-> CRLF conversion on Windows. And sets external encoding to ASCII-8BIT unless explicitly specified.'
<headius>
so I guess "BINMODE" is basically that plus O_BINARY on Windows
<headius>
right
<headius>
whew, guess my understanding is correct
<enebo>
I guess my understanding was right too
<headius>
ok I pushed a minor fix to JRuby that appears to work properly with *existing* jnr-constants release
<headius>
there are fixes required in jnr-constants though for other things
<enebo>
I just am looking at our code and it is unclear since one seems to set the other
<headius>
namely we lost constant.value() method in generated code because it changed to intValue in the generator at some point but nobody regenerated after that
<headius>
I just added another .value() to all constant sets that does the same as intValue
<enebo>
if (fmode_p == null) fmode_p = new int[]{0};
<enebo>
I half wonder how many dummy objects we make
<enebo>
although in this case it will potentially set [0] to something
<enebo>
interestinly in this case fmode_p could be removed and we could use a return value
<headius>
can we?
<enebo>
just for that one method where I saw this
<headius>
ah
<enebo>
it is a void static method
<headius>
well that's an easy change if so
<headius>
MRI has a ton of functions with "out" params so that's where these come from
<enebo>
It is a rats maze overall since fmode_p cannot be done that in most places
<headius>
always wanted to do another pass and un-C this code but multiple return values are hard to do another way
<enebo>
yeah I know this was a port
<enebo>
before you ported over the work I did I already had this out var
<headius>
it's funny how much typical C code uses out params and then you realize Java has nothing equivalent *at all* and it feels so strange
<enebo>
the real problem with MRI code is it is not really thought out...you can see how organic these methods are
<headius>
I mean...out values or multi-return, either way it's a gap
<enebo>
yeah and making a box feels really shitty
<enebo>
and using a field when only a few things use it is weird from a modelling perspective
<enebo>
Box == pointer but we probably pay a significant price for it whereas pointer is free (other than locality weirdness)
<enebo>
OO can be a huge pagefault machine but I have wondered how many page faults in C exist just do to weird rando pointer being passed all over
<headius>
yeah I wonder
<lopex>
sometimes it's a pointer to pointer to some struct field
<lopex>
enebo: backing to releasing, so if any of us stage something it will end up in different per user stating repos right ?
<lopex>
*staging
<lopex>
enebo: since these things seem to accumulate in per user repo
Puffball has quit [Remote host closed the connection]
<enebo>
lopex: I am not sure I think there will just be two entries in staging for each?
<enebo>
lopex: It does not happen very often so I don't remember
<lopex>
enebo: when I released both joni and jcodings almost at once, they ended up in same staging repo
<lopex>
enebo: so that's why I'm asking
<lopex>
so if I'd have closed them both they'd be closed together
<lopex>
er, they'd have been ?
<lopex>
wrt grammar
<enebo>
lopex: until you release they are just in staging if I released the same artifact I don't know if it would replace the previous one in staging or ???
<enebo>
but closing is just running verification that it is safe to release
<enebo>
so they are both closed independently
<lopex>
enebo: I talk about different artifacts
<lopex>
in the same staging
<enebo>
lopex: I guess I don't understand what you are asking...all things we push go to a staging server. I doubt it is n staging servers.
<enebo>
lopex: and I don't actually understand what a full release means in a staging server
<lopex>
enebo: so I go orgjruby_somenumber and both joni and jcodings in the content
<enebo>
other than verifying that all the artifact stuff has been done correctly
<lopex>
*got
<lopex>
enebo: so let's recap orgjruby_number is a staging repo right ?
bbrowning is now known as bbrowning_away
<enebo>
I have no idea
<lopex>
enebo: the thing you click to close it
<enebo>
lopex: yeah I don't know technically what it is
<enebo>
lopex: but I have never thought about it so I am confused why you are asking
<enebo>
lopex: is there something bad which can happen if it is not a staging repo? or if it is?
<lopex>
enebo: what I'm saying is that after both performing release for joni and jcodings, I got one staging repo on sonatype side
<enebo>
ok
<lopex>
enebo: with both them in the staging repo content
<enebo>
so if you expand 'content' it shows a tree of two artifacts?
<lopex>
so, one close, both released
<lopex>
yes
<lopex>
yes!
<enebo>
lopex: weirdly I don't think I have even done that the entire time I used it
<lopex>
enebo: so maybe it;s just a time threshold ?
<enebo>
lopex: no I just think I release one thing at a time
<enebo>
lopex: so i have not noticed that. Can you have bi-directional deps in Maven?
<enebo>
lopex: maybe it is so you can release deps dependent on each other?
<lopex>
why is that relevant ?
<lopex>
the versions are upped
<enebo>
lopex: maybe close verifies the artifact you depend on exists already?
<lopex>
hm
<lopex>
so if the deps might have changed ?
<enebo>
lopex: with bi-dependent artifacts you could not ever pass close?
<lopex>
makes sense
<enebo>
lopex: but I am just guessing
<enebo>
lopex: I always just release the base artifact first in a series
<lopex>
anyways, I can show you in next release
<enebo>
then do next one etc...
<lopex>
yeah
<lopex>
but in differen staging repos you mean...
<enebo>
lopex: so what you did is interesting in that I have never done it but I guess knowing that maybe I can release a series at the same time
<lopex>
enebo: and it depends in the timint perhaps ?
<lopex>
*timing
<enebo>
perhaps release at same time is a reason for it? but I am not sure they will neccesarily show up in central mirrors at same time. It would be useful to know all artifacts did successfully closed I guess?
<lopex>
or it is sonatype opt
<enebo>
I can only guess
<enebo>
You know I do this regularly actually
<lopex>
enebo: lol
<enebo>
I just never think about it
<lopex>
enebo: embrassing
<enebo>
jruby-*
<lopex>
what ?
<enebo>
In my mind I am just releasing one thing so I did not think about the fact that I am closing jruby-complete, jruby-stdlib, ... all at once
<enebo>
I think that way because I am just doing a single mvn command though
<lopex>
enebo: but you have to clickit
<enebo>
well yeah there is a single item to select and then say close
<enebo>
but if I expand it there are several artifacts involved in the close
<lopex>
ah, yes, that what I'm talking about
<lopex>
yes yes
<lopex>
doh
<enebo>
yeah so I do it but did not think about that when you brought it up
<lopex>
we came to that eventually
<enebo>
yeah
<enebo>
but I can say I never worried about it either :P
<lopex>
enebo: yeah, my question was would the versions accumulate
<enebo>
I probably realized this when I first started using sonatype and then forgot about it
<lopex>
enebo: talk like to the newbie
<enebo>
lopex: I don't know. I am sure I have never been in that situation
<lopex>
ok
<enebo>
lopex: like I release joni then realize we missed a commit then push another release with a new version?
<lopex>
yeah ?
<enebo>
lopex: I would probably just be inclined to finish the first one if it happened then release after first was out I guess
<enebo>
lopex: but I don't know
<lopex>
you drop the previous stage ?
<enebo>
lopex: maybe it will handle n versions at once
<lopex>
ok
<enebo>
lopex: you can drop. I have done that many times but that is basically realizing the release is boned before you finish
<lopex>
enebo: at least you appreciate it was a legitimate question
<enebo>
lopex: but I have been paranoid about not dropping
<lopex>
why ?
<enebo>
lopex: I assume if you fix it and push a new release it will replace
<enebo>
lopex: but I am not sure since I have never knowingly left a bungled upload without explicitly dropping it
<lopex>
enebo: so I was dropping like child in a fog today
<enebo>
lopex: I have to think it will just replace it if it is the same version though
<lopex>
but only after fail or sucessful close
<lopex>
er, success has that drop checkbox on by default
<lopex>
aka release
<enebo>
the other thing I wonder is if two people up the same release at same time what happens
<lopex>
yeah, like the verification block or something ?
<lopex>
*blocks
<lopex>
er, not
<lopex>
enebo: you mean actually hittinh that relase button
<lopex>
*hitting
<lopex>
enebo: though same applies for earlier stages
<lopex>
or not ?
<enebo>
lopex: I just mean what you see in staging that you can close/drop/release
<lopex>
yes
<lopex>
let's start from that
<lopex>
the I dont know
<lopex>
it's up to a decision like sign time etc
<lopex>
whatever criteria
<lopex>
enebo: is sign time actually a thing ?
<enebo>
lopex: well I am asked during release plugin
<enebo>
lopex: so I am guessing I sign artifact as part of whatever it happening
<lopex>
enebo: but it's up the maven plugin what's actually that right ?
<lopex>
*up to
<enebo>
lopex: yeah it is just a black box to me
<enebo>
lopex: I just trust the plugin is actually signing it and sonatype seems to think so
<lopex>
enebo: I've run -X when I had that sig problem so probably easily traceable
<lopex>
yeah
<lopex>
and then there's issue between gpg2 etc
<lopex>
lots of mismatchs
<lopex>
*es
<lopex>
enebo: anyways, thanks for your assistance, I've never done that before and have learned whole lot of things
<enebo>
lopex: sure. I am glad you can do it now too
<lopex>
like that keyserver no responding with key etc
<lopex>
enebo: and there's was a question how settings.xml will interact with .gitconfig
<lopex>
wrt signing
<lopex>
so much bifuracation
<lopex>
nirvdrum: you here ? how does TR do on sequel and jdbc (via java integration) ?
<lopex>
is it even a legitimate question wrt java integration ?
<nirvdrum>
lopex: No idea. It's probably worth taking a look at.
<lopex>
nirvdrum: it probably depends on usages right ?
<lopex>
er, user base
<nirvdrum>
Well, first off it'd only work when running on a JVM. There no way to dynamically load the adapter code in the native image.
<lopex>
right
<nirvdrum>
I could try to build it with the adapter JAR on the classpath. But doing that for every adapter would be less than ideal.
<lopex>
nirvdrum: on the plus side of things sequel seems to use only rudimentary java integration
<lopex>
so just plain method calls etc
<nirvdrum>
Yeah. For some reason I thought it called into JRuby specific classes.
<nirvdrum>
Maybe it did years ago. Or maybe my memory is just bad.
<lopex>
nowadays jeremyevans seems to be specializing lots of things for pg in exts
<lopex>
nirvdrum: you think it would be easier to support sequel than AR on first sight ?
<nirvdrum>
It's really the adapter gems that we'd need to support.
<lopex>
I know about the elephant rails
<lopex>
nirvdrum: so it;s all about exts ?
<nirvdrum>
I think so.
<nirvdrum>
The rest of it is just Ruby.
<lopex>
ok
<nirvdrum>
I really don't know what the best approach would be.
<lopex>
nirvdrum: but at some point you;ll need jdbc
<nirvdrum>
Using JDBC on Java seems to make a lot of sense. But AR-JDBC is a massive undertaking for you guys and most AR "extensions" don't work with it because they're really all monkey-patches that assume a particular class structure.
<lopex>
yes
<lopex>
I understand the compat matrix for ar-jdbc :P
<lopex>
enebo: ^^
<lopex>
enebo: from your talk anyways
<headius>
yo
<headius>
I'm back
<lopex>
nirvdrum: but yep, lots of monkey patching
<lopex>
and ultra fragile
<headius>
there's almost no patching now
<lopex>
in ar-jdbc ?
<headius>
the structure of AR is more amenable to alternative adapters
<headius>
yes
<lopex>
from when ?
<headius>
from 50.0
<enebo>
nirvdrum: lopex: we copy main adapter file and postgresql has some different method but it should be all compat with AR
<headius>
the reboot removed all legacy support which also killed most of the patching...recent Rails abstracts the adapters better
<enebo>
sqlite3 is only a few dozen lines now
<enebo>
mysql is like 150
<headius>
yeah that's just temporary though...we need to get them to move some class logic to modules and we can just include then
<enebo>
mysql has AbstractMySQLAdapter since they support two mysql variants
<enebo>
so we can just include that one
<nirvdrum>
lopex: I set JRUBY_VERSION in my script. Then I hit a new interop issue.
<nirvdrum>
headius: I mean 3rd party AR extensions. Or has that improved as well?
<lopex>
nirvdrum: I have no opinion on AR
<headius>
We can support them if they aren't C exts
<headius>
and don't depend on the C adapters
<headius>
there are some that were C but have JRuby versions as well
<lopex>
headius: sadly sequel goel ext way for pg
<lopex>
*goes
<nirvdrum>
I tried to do that once with a popular Postgres extension that added in support for arrays, json, and hashes, before AR supported them properly. It was a major headache.
<headius>
sequel should work fine with pg on jdbc
<lopex>
headius: it does
<lopex>
headius: he does some optz
<lopex>
Jeremy
drbobbeaty has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<nirvdrum>
The AR-JDBC class internals differed, so the monkeypatch wouldn't work. I tried to fix that on a branch, but Bundler won't let you have the same gem name listed with different versions or branches for different platforms.
<headius>
AR-JDBC is no longer a "massive" undertaking either...the reboot cleaned up like 90% of the code
<nirvdrum>
Cool.
<headius>
nirvdrum: yes, that's probably much better these days
<lopex>
well, java drivers will stay jdbc 4 forever right ?
<nirvdrum>
That was something I encountered when migrating from MRI to JRuby. At some point I gave up maintaining it and switched to using Sequel. I was considerably happier afterwards.
<headius>
lopex: I don't see why they would have to
<headius>
nirvdrum: yeah, AR took a long time to restructure
<nirvdrum>
Heh, so my knowledge on that might be 6.5 years out of date.
<lopex>
headius: you mean supporting odbc dedicated bridbges ?
<headius>
much of that came along with ARel since it needed to remove the logic of sql generation from AR itself
<lopex>
*bridges
<headius>
lopex: oh you mean level 4
<lopex>
yeah
<headius>
I thought you meant jdbc 4.x
<headius>
yeah probably
<lopex>
right
<headius>
they could be anything though
<headius>
I'm not sure the non-pure drivers have any benefit now though
jmalves has quit [Remote host closed the connection]
<lopex>
hm
jmalves has joined #jruby
<lopex>
I confused all the time about jruby and tr thing wrt dbs
<enebo>
nirvdrum: as of last week we just fixed an issue with initialize of all three adapters to stuff like postgresql-gis works since AR 5+ changed arity
<headius>
enebo: with repaired jnr-constants and some patches to JRuby everything passes again on Travis
<enebo>
well we fixed it over a week ago but kares just revved new points for 5.0-5.2
<headius>
I'll finalize the PR if you can have a look through it
<enebo>
headius: ok
<headius>
jnr-constants changes are just on master
<headius>
so needs a release, and then I can merge the update
<lopex>
nirvdrum: so all the exts for sequel pg you will run right ?
<enebo>
nirvdrum: so I believe we should work with extensions sans expectations of adapter API itself (e.g. pg specific calls)
<enebo>
we do not emulate connection adapters API themselves
<headius>
that's closer to what we had to do before, or else reimpl the entire adapter
<headius>
thank goodness they improved the architecture
<lopex>
headius: so on graal, can jruby hook up to these exts in somewhat ?
<headius>
lopex: I suppose if we wanted to do a similar sulong-based rig...probably could reuse much of what TR does
<nirvdrum>
enebo: Awesome (on the gis thing).
<lopex>
yep
<lopex>
headius: but it seems to be pretty centralized work anyways
<enebo>
nirvdrum: well we thought we were supporting it up until recently so it was a bummer we weren't :)
<nirvdrum>
lopex: I assume at some point all the exts will run. The goal is to be run what MRI can out of the box.
<nirvdrum>
*to be running
jmalves has quit [Ping timeout: 252 seconds]
<lopex>
headius: and api mapping ?
<lopex>
having sulong seems to simplify things long way
<headius>
well my understanding of the C ext stuff in TR is that there's specific upcalls implemented on the Java or Ruby side and they map those to the API
<headius>
so we'd map them to something equivalent on our side
<headius>
that's basically how our JNI C ext support worked...most of the API just ended up being an upcall to a similar-named method in Java
<lopex>
headius: including cleanups and thread issues ?
<headius>
well...that's where it gets nasty and why I don't trust C exts :-)
<lopex>
hah
<lopex>
I just cant imagine how much work gets into these things
<lopex>
and mostly it;s c api on both jvm and mri sides
sgeorge has quit [Remote host closed the connection]
<nirvdrum>
I'm trying to eat dinner and then I need to go drop my car off at the dealership.
<nirvdrum>
If I'm quiet, I'm not trying to ignore anyone :-)
<lopex>
nirvdrum: offence taken :P
<lopex>
headius: but we're on graal then why not just use sulong ?
sgeorge has joined #jruby
<nirvdrum>
lopex: If you don't want to require Graal to be fast, then note that Sulong will run strictly in interpreter mode. That may be prohibitively slow.
<headius>
lopex: JRuby is a JVM language, not a GraalVM language
<nirvdrum>
It's a Truffle language after all.
<nirvdrum>
The call bridge is also Truffle's interop mechanism.
<headius>
if at some point we can use Sulong on other JVMs, then sure
<enebo>
If it is trivial to wire up then it could be fun to play with from jruby but not for supporting MRI cext footprint
sgeorge_ has joined #jruby
<headius>
sure, probably could use it for something
<enebo>
we decided not to take that support effort on years ago and I don't think any of us have had a change in heart
<headius>
though if we're not going all-in on truffle I'm not sure what it would gain us versus FFI
<lopex>
why cant you mix both ?
<enebo>
lopex: both of which?
sgeorge_ has quit [Read error: Connection reset by peer]
<lopex>
java program and truffle
<lopex>
that's the way I see it
<enebo>
lopex: you can but it will only work well with graal
<nirvdrum>
You guys are in a much better position than we are currently with regards to extensions. All the major native ones have JRuby-compatible releases already.
<lopex>
enebo: yes
<nirvdrum>
And I think the incremental on-going maintenance isn't too baad.
<nirvdrum>
*bad
<nirvdrum>
Nokogiri is probably the next most complicated one.
<headius>
most new ext authors at least give some consideration to JRuby, so that's helped a lot
<enebo>
lopex: so we can do that but if we want to support non-Graal (admittedly high 90s of Java world) we are then supporting two things
sgeorge has quit [Ping timeout: 252 seconds]
<lopex>
nirvdrum: hey I want to run TR too
<headius>
there are also fewer exts these days purely for performance...that majority are just wrapping some lib, which we can do with FFI or shimming a Java lib
<lopex>
headius: but in mostly non portable ways
<headius>
non portable?
<enebo>
he means platform compat I think
<headius>
Java libs are obviously portable...you mean FFI?
<lopex>
well, if you mean linux and windows then sure
<lopex>
yeah
<lopex>
I agree
<nirvdrum>
lopex: I wish native extension authors would provide a pure Ruby variant as well.
<lopex>
headius: I somewhat meant as400
<headius>
well, FFI could be improved a lot with some generation tools
<enebo>
nirvdrum: for perf stuff especially
<headius>
lopex: right, but TR wouldn't be able to support that at all
<nirvdrum>
Although, there's opportunity for divergence. I've hit a json bug in the json-pure variant.
<enebo>
nirvdrum: for wrapping imagemagick or HTML5 parser it is more of a commitment
<nirvdrum>
So, I can appreciate why one might not want to maintain parallel versions of a gem.
<headius>
pure-Ruby versions of everything would be nice but again most of them are wrapping a library
<enebo>
yeah my point above since who is going to rewrite imagemagick in ruby
<enebo>
(note: image_voodoo is pretty nice)
<headius>
if we'd put more effort into getting FFI generators to work nicely, FFI probably would be the standard option for wrapping a lib
<nirvdrum>
Someone reported a compatibility issue for hamlit with TruffleRuby. We'll get the extension working, but just running haml on TR is faster than running hamlit on MRI.
<lopex>
and then the argument gets back to mri apis
<enebo>
but much less than imagemagick
<nirvdrum>
headius: FFI is a fine option as well. It's a shame people still overlook it.
<enebo>
nirvdrum: haha you remember our cext support
<nirvdrum>
I could even warm up to Fiddle.
<lopex>
nirvdrum: actually in case you've looked at python api would it be easier ?
<nirvdrum>
Yeah. It was interesting.
<enebo>
nirvdrum: people would get the cext du jour to work then complain we were not quick then we would say you can use this version...it was an endless issue
<headius>
C ext over JNI is death of a million cuts...the C API is diced so small
<lopex>
nirvdrum: or it;s just a uncontrolled c thing
<nirvdrum>
lopex: Sorry, I don't understand the question.
<enebo>
so if cext is slower than pure Ruby you will endlessly be having to close issues telling people to use something else.
<headius>
python API would definitely not be easier because it exposes reference-counting APIs
<lopex>
nirvdrum: would python c api be easier fo sulong ?
<headius>
for the same reason that MRI's conservative GC and raw pointer access is a hassle
<lopex>
gc is one thing
<nirvdrum>
lopex: I don't know. I could ask Tim. He's leading up the Graal.Python effort.
<nirvdrum>
He also wrote the JRuby C ext. support.
<lopex>
headius: ah, right DataWrapStruct is gc dependent
<headius>
no alternative Python or PHP impl has managed to do a good enough job supporting C exts because both have refcounting C APIs
<nirvdrum>
He seems to be a glutton for punishment.
<headius>
lopex: you can also allocate objects on the heap and walk away from them
<headius>
if they're in the GC'ed heap MRI will eventually clean them up
<headius>
so you have to handle abandoned object handles in C exts
<lopex>
headius: but you can release some requirements somewhat, for paying less
<headius>
the weakref cleanup of handles alone was a huge overhead for our C API
<lopex>
headius: featurewise
<lopex>
headius: tell me, who really depends on ObjectSpace perf wise
<headius>
not objectspace
<headius>
just handles to managed objects
<lopex>
walking mark&sweep ?
<headius>
which might be allocated from C code
<lopex>
ah
<headius>
if they live on managed heap they have to have a lazily-collected handle
<lopex>
just dangling ?
<headius>
if they live on unmanaged heap they have to be copied across
<headius>
and calling across either way has to deal with managed VM boundary
<lopex>
right
<headius>
yes, dangling is the common case
<lopex>
which from sulong point of view are ok
<headius>
because there's no APIs for bounding the lifetime of a reference into the managed heap
<headius>
well, how is that any more ok in Sulong?
<lopex>
yeah, understood
<lopex>
headius: because sulong manages the heap ?
<headius>
but it's still an unmanaged heap with raw pointer access and dangling allocations
<headius>
I assume Sulong has some smarts here but it's really a native heap
<lopex>
hm
<enebo>
dinner
<lopex>
headius: I though sulong is just a layer for ast and c exts dont matter here
<lopex>
*thought
<lopex>
headius: c malloc is java alloc or no ?
<headius>
well java alloc is on managed heap
<headius>
it's definitely not that
<lopex>
ok
<headius>
at a minimum because you need those pointers not to move
<lopex>
right
<headius>
I mean again, if you can see the whole world maybe you can prove more
<headius>
but most exts call out to C functions and libraries
<lopex>
headius: and you cant get that from llvm bitcode
<headius>
so pointesrs have to be pointers
<lopex>
right ?
<headius>
llvm bitcode probably tells you a ton but at some point a pointer is just a number
<lopex>
yeah, I think I get it now
<lopex>
so just c semantics gets in the way
<lopex>
game over
<headius>
I don't have all the info about Sulong but this is my understanding
<headius>
it's basically just running the C code via llvm bitcode along side the JVM/Substrate/GraalVM stuff
<lopex>
right without java semantics
<headius>
I think most of the smarts are in reducing the copying and calling across that boundary because it has visibility across
<headius>
yeah
<headius>
so it can, say, be much less overhead per call than JNI
<headius>
since it can know if something's a leaf or makes other calls etc
<headius>
but eventually you're still dealing with two heaps
<headius>
and opaque pointer refs
<headius>
and unknowable concurrency issues :-D
<lopex>
so if c takes pointer to poiner in some less obvious way, then no clue
<headius>
right
<headius>
raw pointers passed to C libs, callback functions, managed objects represented as pointers for an arbitrary amount of time...it's really an awful API to support
<nirvdrum>
lopex: AFK for a couple hours. I'll be happy to answer any questions when I get back. Or just mention me in your message and my client will highlight it.
<lopex>
nirvdrum: sure
<lopex>
jeeze on how much I underestimated the effort
<lopex>
headius: but potentially having whole world c program sulong might do better ?
<lopex>
it's the lib calls that are unsafe now
<lopex>
headius: provided sulon is aware of all c
<lopex>
which is alias analysis which is hard and there we go ?
<lopex>
I just know alisa analysis is a killer for lots of c optz
<lopex>
and it's not even mentioning GC
<lopex>
seems kinda hard
<lopex>
headius: assuming AA is just pointer to pointer streal
<lopex>
*steal
<lopex>
headius: this is madness
<headius>
lopex: maybe, I don't know
<headius>
that would work for things like json, oj, ox libraries that don't call any libs
<headius>
they have to call libc though presumably
<headius>
so...pointers again for some of it
<lopex>
yeah, but allocs inbetwen are hell
<headius>
right
<lopex>
or alloced passed between ext end runtime
<headius>
there's always a native edge for any extension, even if it doesn't explicitly use some third-party lib
<lopex>
jeeeze
<lopex>
never thought of that
<lopex>
wrt optz
<headius>
I know TR+Sulong has some smarts to know when to leave a String in native or managed to reduce calling across that boundary but there's always going to be a boundary I assume
<lopex>
ok, but for the most part sulong can be useful, just guessing from code patterns right ?
<lopex>
not every c program does those things
<headius>
I assume so...if only as a "smarter JNI"
<headius>
oh, yeah I dunno if it's better than LLVM or not
<headius>
that's fairly subjective
<lopex>
and the mmemory model, and gc
<lopex>
etc
<lopex>
helll
<headius>
right
<headius>
this is why we abandoned C ext effort...twice
<headius>
and one of those efforts actually worked
<headius>
most of the overhead was the copying (which Truffle might be able to help profile and improve) and JNI overhead like safe points (which could be reduced but not eliminated
<lopex>
yeah, calling into c dynamically will always be useful
<lopex>
till the end of c
<headius>
so I guess we're waiting to see if the finely-diced MRI API will ever be fast and thread-safe
<lopex>
yeah
<headius>
yeah
<lopex>
mri needs to change
<headius>
if we could use sulong to improve the FFI situation, for example
<headius>
like use the clang plumbing already in Sulong (I think?) to generate the FFI binding
<headius>
lopex: the TR C ext support also has to patch many (most?) extensions
<headius>
it's just too broad and invasive an API
<lopex>
I guess I always looked at sulong / truffle optimistically like calling into some ddb driver with options which would get optimized internally
<lopex>
all those checks lile (this option is set) would be constant propagated
<lopex>
headius: right ?
<headius>
yeah it probably can do some of that
<headius>
if it has complete info about stuff it does great...but C is the definition of incomplete information :-)
<headius>
so I dunno
<lopex>
headius: imagine something like live rebel, with that ideal graph visualizer for perf on production
<lopex>
and STABLE
<lopex>
nirvdrum: ^^
<lopex>
headius: yeah, but some parts of c code can be statically proved to be safe
<lopex>
but yeah, it;s just the assert boundaries
<headius>
I guess it remains to be seen how far that can go...I really don't know
<headius>
we'll let TR folks figure it all out and then just borrow it :-)
<lopex>
yeah
<lopex>
seems very hard
<headius>
Rubinius folks had to figure this all out too, and they still had really poor C ext perf
<lopex>
I wonder how far it would go if Evan had the money
<headius>
well, let's remove Evan from the equation too...if there were enough money and man-hours
<lopex>
he is brilliant
<lopex>
not enough evan's ?
<headius>
I'm pretty sure he didn't stop because he wouldn't be paid anymore
<headius>
Brian kept working on it under EY for a good while after Evan left
<lopex>
and ruby specs are still dead ?
<headius>
if it were me I'd have left because it was too big a problem to try to solve
<headius>
like, if the primary challenge I faced every day was C exts...I'd quit too
<lopex>
mhm
<headius>
rubyspec was adopted by MRI and is actually more active than ever
<lopex>
oh, cool
<headius>
it is now called ruby/spec because someone requested it not be called rubyspec
<headius>
🙄
<lopex>
well, at some point ruby was most speced language right ?
<lopex>
at least dynamic one
<lopex>
headius: on jruby site there should be info on failed ruby impls
<lopex>
microsoft, apple, SAP
<lopex>
gemstone ?
<headius>
yeah
<headius>
I dunno if I'd call rubymotion failed...but it wasn't really a true ruby impl anyway
<headius>
I have no idea if SAP BlueRuby still exists
<lopex>
but those above are
<headius>
don't forget topaz
<lopex>
yeah
<lopex>
ah
<lopex>
righto
<headius>
opal is still going
<headius>
partial impl obviously
<headius>
xruby
<lopex>
yeah, and it;s almost interesting
<lopex>
headius: there;s react api for it
<headius>
I think the OMR effort is dead
<lopex>
wrt opal
<headius>
Cardinal was Ruby for Parrot
<lopex>
ah, I recall
<headius>
I'm not sure if rbx should be considered failed or not
<lopex>
dormant ?
<headius>
failed may be the wrong word...ruby impls that are not usable today
<lopex>
but opal goes strongs wrt dev
<headius>
yeah
<lopex>
headius: I tried whole react app on it
<headius>
I find this more interesting than trying to build a better MRI
<lopex>
and it worked
<lopex>
ah
<headius>
JRuby is better than MRI at many things but also has the value add of running on any JVM, running with parallel threading, accessing all the JVM libraries, deploying to JVM servers
<lopex>
headius: once you said strings will become symbols
<lopex>
so read only
<headius>
Opal's value add is being on JS in the same way
<headius>
heh yeah...it's going that way
<headius>
.freeze hack and the pragma are both efforts to move Ruby code toward immutable strings