<headius[m]>
Oh yes I saw you tweeting about playing with indy
<headius[m]>
ok this is in my queue after triaging recent bugs
<kares[m]>
yeah I got some more experiments in the queue ... for later
<kares[m]>
how was the vacation, batteries recharged :) ?
<headius[m]>
I think so!
<headius[m]>
we should chat more about indy stuff
<headius[m]>
I want to make it better integrated and possible for us to have a mix of indy and non-indy call sites as they profile to be hot
<headius[m]>
there's so many more things we could use indy for but using it fully at every call site is a waste
<kares[m]>
yeah it either all indy sites or not
<kares[m]>
sounds like a waste - for some calls it seems to make more sense
<kares[m]>
like primitive ops ~ fixnum/float (in the PR)
<headius[m]>
right, and then each indy linkage is 6-deep, with potentially a dozen adaptations to do framing and validation and what have you
<kares[m]>
there aren't many of those and they seem to be all faster
<headius[m]>
yeah direct calls to key core stuff like fixnum and float ops
<headius[m]>
or calls to small target methods...sort of an inlining size threshold, but also recognizing that it's going to be the small fast methods that compose better and get the most of of the site linkage
<kares[m]>
makes sense - I think non-direct stuff just gets slower
<kares[m]>
how do you recognize such method?
<kares[m]>
byte code check?
<headius[m]>
there'd be some metric, maybe IR size...but given that at link time we have the target method in hand we could make better choices
<headius[m]>
the only choices made now are just based on the type of method...core, ruby, field/var
<headius[m]>
kares: Ahh I see NormalCachingCallSite2
subbu is now known as subbu|lunch
<headius[m]>
I also prefer the Bimorphic naming
<headius[m]>
unsure about method names though
<headius[m]>
I'd call the cache entries "primary" and "secondary" but that's pretty long to tag on the end of a method name
<headius[m]>
kares: relating to indy stuff...one of the other changes I want to explore is eliminating our non-indy CallSite entirely and instead using an indy call site
<headius[m]>
then the compile.invokedynamic flag would change to a gradient of how heavily we customize those call sites to the caller and callee...simplest form would just bind DynamicMethod.call and have a simple monomorphic cache
<headius[m]>
I think I also so you looking at Dynalink...that may still be an option but it always seemed more complicated than necessary to me
victori_ has joined #jruby
<headius[m]>
we really do need a generic library for doing all these indy site customizations
<headius[m]>
dynalink is also now unmaintained, so that's an issue
victori has quit [Read error: Connection reset by peer]
<headius[m]>
kares: reviewed
<headius[m]>
I'll comment separately about the naming
<headius[m]>
commented
<headius[m]>
basically I'm not thrilled about the "2" moniker but I don't have a great suggestion...and the numeric moniker pattern is elsewhere for e.g. reified instance vars
<rtyler>
I'm good at naming things, how can I help? :)
subbu|lunch is now known as subbu
enebo has left #jruby [#jruby]
<headius[m]>
I'm terrible at it, so you probably can
<headius[m]>
we have a single-method cache called NormalCachingCallSite...now kares introduced a subclass that's a bimorphic site and just appended a "2" to the class and method names, which is fine but not very descriptive
<headius[m]>
naming the first one MonomorphicCallSite and the second BimorphicCallSite seems good but not sure about the additional methods
<headius[m]>
my first thought was "secondary" since it's the secondary element in the cache, but it's a long word
<headius[m]>
getSecondaryCache vs getCache2 etc
<headius[m]>
so there it is 😄
<rtyler>
I feel dumb in that I don't know what "bimorphic" is supposed to mean :/
<headius[m]>
enebo: so I'm still looking at this net thing and it just seems to me like a plain connection refused, but I can't figure out why
<headius[m]>
the finishConnect logic in JDK is no help because it bottoms out in a native method, which is where the exception comes from
<headius[m]>
rtyler: mono versus bi, morphic meaning "shape"...it's just saying the shape of the call site is that it has two possible cached methods
<headius[m]>
there would be tri or quad beyond bi but as you would expect talking about 3+ targets at a call site has less interest than 1 or 2 targets
<headius[m]>
in this case it's bimorphic to accommodate math methods that might fall into either integer (fixnum usually) or float logic
<headius[m]>
enebo: I've step debugged into finishConnect and there's very little to indicate anything except that it's a plain old refused connection 😕
<headius[m]>
it has the right address and all
<enebo[m]>
the channel is open after the select according to the channel via isOpen so what does that mean
<rtyler>
headius[m]: that's helpful, it looks to be a name y'all have just made up rather than a definition I could find :P
<headius[m]>
enebo: why don't I see anything bound to 4567
<headius[m]>
isOpen just checks if the fd is set
<headius[m]>
even in MRI
<headius[m]>
oh but the channel you mean
<headius[m]>
that I don't know...the select should indicate it's ready to connect (or error) and then it errors
<rtyler>
headius[m]: would NormalCachingCallSite/DualCachingCallSite not make sense?
<headius[m]>
the socket is "open" but not connected...so maybe that meaning is ok?
<headius[m]>
rtyler: Dual isn't bad
<headius[m]>
I haven't seen monomorphic/bimorphic used much outside VM impls and papers
<headius[m]>
VM jargon
<headius[m]>
heh, maybe DualCachingCallSite and "getOtherCache"
<rtyler>
ah, that's why it was tricky for me to find
<headius[m]>
seems like a dead end if we added more shapes though
<headius[m]>
enebo: I'm going to try running server in JRuby again
<headius[m]>
my network-fu is not great but a wiresharky thing might show if this is trying the right addresses
<headius[m]>
I don't get this
<headius[m]>
where is the socket
<headius[m]>
WHERE'S THE SOCKET
<rtyler>
try using sudo
* rtyler
ducks
<headius[m]>
hah
<headius[m]>
faye doesn't use some weird socket does it?
<rtyler>
define weird? it might be opening a socket with async flags set
<enebo[m]>
since you ran jruby as server start it and run mri against it
<enebo[m]>
does it negotiate handling both v4 and v6 somehow?
<headius[m]>
I have no idea how MRI does IP4/6
<enebo[m]>
but it MRI can hit us with ipv4 and them with ipv6 it seems it would have some magic to do the right thing
<headius[m]>
it's likely some logic in localhost lookup that differs
<headius[m]>
huh riot recognizes localhost as an address
<headius[m]>
enebo: worksforme...I'm going to look into the other items I tabbed now
<enebo[m]>
Exception `IOError' at org/jruby/ext/socket/RubyTCPSocket.java:167 - Failed to open TCP connection to ::1:4567 (Address family not supported by protocol)
<headius[m]>
oh I'll try JRuby server quick though
<headius[m]>
assuming it actually works
<headius[m]>
enebo: [::1]
<headius[m]>
in a UR:L
<headius[m]>
heh, URL
<enebo[m]>
jruby -d -rjson -rnet/http -e 'message = "hello"; uri = URI.parse("http://[::1]:4567/faye"); p Net::HTTP.post_form(uri, :message => message.to_json)'
<headius[m]>
oddly I didn't need it to connect in that command line
<headius[m]>
and netstat shows we're binding to tcp4
<enebo[m]>
I am not making statement I have not tried JRuby as server yet
<headius[m]>
right
<headius[m]>
it works
<enebo[m]>
ok so JRuby work v4 and v6 so long as it is the server and client
<headius[m]>
I get connection refused if I try IP6 from JRuby client to JRuby server
<headius[m]>
yeah
<headius[m]>
so my diagnosis is that this is just IP4/6 mismatch
<enebo[m]>
ok so you do match me...whew
<enebo[m]>
we are just doing v4 all the time and v6 is not working more or less
<enebo[m]>
so v4 -> v4 ok v6 -> anything is family not supported error
<enebo[m]>
derp so v6 does work for me
<enebo[m]>
JRuby (v6) -> MRI server : OK
<enebo[m]>
I had something commented out I shouldn't
<enebo[m]>
JRuby (v4) -> MRI server : Connection closed error
<headius[m]>
yeah
<enebo[m]>
MRI as client works using V4 or V6
<headius[m]>
I have very little logic of what the appropriate negotiation is for 4 vs 6 in a typical setup...we may be doing something wrong, but this case is defiinitely trying different stuff on both ends
<headius[m]>
mmm
<headius[m]>
could be resolv.rb ip6 difference perhaps, or just the way JDK presents alternative addresses to us
<headius[m]>
so there may still be a bug there but this is not a new behavior
<enebo[m]>
So good thing we at least work with appropriate matching addr versions but they obviously do some more negiotiation
<enebo[m]>
yeah
<headius[m]>
get the reporter to do some investigation into how MRI managed to negotiate the connection even with non-IP6 client string
<headius[m]>
I'd suspect the two things I listed above
<headius[m]>
that initSocket move may still be valid fwiw...obviously without it the select is not doing what I intended it to do
<headius[m]>
worth throwing on a branch and running CI
<enebo[m]>
select does nothing if it ie below
<enebo[m]>
it will 100% hit an else branch which returns true
<enebo[m]>
So that is valid in that it will select once and return true
<enebo[m]>
no doubt there is some case where it maybe select multiple times
<enebo[m]>
probably not ordinary case on an unburdened dev machine
<enebo[m]>
getByName feels like it should handle this
<headius[m]>
I can't figure out what's different in that environment that would cause it to bail out
<headius[m]>
basically looks like under some environments, str.encode! with undef and invalid => :replace does not do the replace and bails out at first bad char, then raising error about incomplete transcoding
<headius[m]>
I can't repro on any normal linux env...so far only repro under the docker image he provided
<headius[m]>
image/container/config whatever
<headius[m]>
I have tried forcing LANG and file.encoding in the docker env and it still breaks...tried forcing my MacOS env to match the docker env and it still passes
<headius[m]>
23 bytes left is just about the number of bytes following the first bad char
<headius[m]>
yeah index of first bad byte is 24 from end
<headius[m]>
enebo: you also if you have any suggestions
<enebo[m]>
So a docker image and only that image (although he did report it against mojave originally) will not properly finish and it seems to fail on first bad character
<enebo[m]>
This is pure Java code doing transcoding so something it causing what works elsewhere to not there
<enebo[m]>
main culprits would be file.encoding, default_{internal, external}, LANG (although this would likely be reflected in file.encoding), and whatelse?
<enebo[m]>
I would have to think source string being transcoded would not have encoding it normally has
<enebo[m]>
The other weird bit is invalid: replace....I mean is there a place where we transcode but it is benath or not honoring that somewhere?
<headius[m]>
right
<headius[m]>
so on that env file.encoding ends up being some weird ANSI code page string
<enebo[m]>
Maybe we assume extension byte then read something which is invalid for what we think then blow up vs bailing and saying it is invalid
<headius[m]>
LANG is not set
<enebo[m]>
yeah either case we probably need to know what that strings encoding is internally
<headius[m]>
if I set LANG to en_US.UTF-8 it does not help. If I force -Dfile.encoding=UTF-8 it does not help.
<headius[m]>
similarly if I set those to the weird option locally it does not fail
<enebo[m]>
how about -Eutf-8?
<headius[m]>
also no effect
<headius[m]>
so walking through the script he has in his dockerfile, basically the logic reads in this Ruby source that's not UTF-8
<headius[m]>
it reads it in using r:utf-8 so the resulting string does claim to be UTF-8
<headius[m]>
but I think since internal is also UTF-8 it doesn't try to transcode so the read just blindly reads the bytes
<lopex>
headius[m]: is that the mac utf-8 issue ?
<headius[m]>
in any case that doesn't seem to be the bug...in both envs we read the file in and call it UTF-8 but it has bad bytes
<headius[m]>
when doing the transcode later, those bytes should get replaced, but they don't...we just bail on the transcode and then error
<headius[m]>
lopex: so far I've only been able to repro in a linux docker
<lopex>
headius[m]: but there was a separate one for mac-utf8 afaik
<headius[m]>
in this case centos...which shouldn't be that weird
<enebo[m]>
is jcodings invlida really set to '?' like java charsets?
<lopex>
you mean replacement ?
<headius[m]>
there are default ones yeah
<enebo[m]>
yeah I mean replacement when invalid
<headius[m]>
the default in ascii is ? and in the unicode encodings it's a special <?> char
<enebo[m]>
headius: I was just wondering if we have some path where it cares about the charsets of the JVM in any way
<headius[m]>
the result on my local system, if I p the output of encode, is "This is a test. I\uFFFDt\uFFFDrn\uFFFDti\uFFFDn\uFFFDliz\uFFFDti\uFFFDn\n"
<headius[m]>
\uFFFD being the special replacement char
<enebo[m]>
headius: I assumed it was pure jcodings but I don't know if jcodings ever has a necesary relationship with java charsets
<enebo[m]>
hmm and it is complaining about the source string and not what it is writing? Could that be possible? maybe that replacement is invalid
<enebo[m]>
I am just suggesting random stuff :)
<headius[m]>
well the encode! expects that when the transcode is complete, the "in" position will equal the end of the input string
<headius[m]>
if everythign replaces right, it should
<headius[m]>
but in this env, it stops at the first bad byte, so there's bytes left and it raises
<enebo[m]>
you did full backtrace on this too
<headius[m]>
yeah
<headius[m]>
can't really dig deeper without hooking up step debugging in docker
<headius[m]>
just wish I could repro the damn thing outside of docker
<headius[m]>
our ability to help is limited since we don't have access to a ppc env but let's see if we can figure it out together
<headius[m]>
so I saw you did a JRuby build with updated jnr stuff
<Trevor159>
I am using an arm env if that matters
<headius[m]>
Oh you're getting a similar error to this bug, but on ARM?
<Trevor159>
yes, I commented on the issue as well with my system
<headius[m]>
oh I see
<headius[m]>
ok
<headius[m]>
Trevor159: so you tried 9.2.7.0?
<headius[m]>
it might help if you opened a separate bug since this one is about PPC
<headius[m]>
I thought ARM was working on recent JRuby
<Trevor159>
I am using the bundled version with logstash 7.1.1 which i believe is 9.2.7.0
<headius[m]>
do you have a logstash issue also?
<headius[m]>
first step would be to try JRuby 9.2.7.0 directly and see if that also fails to load on your environment
<headius[m]>
without logstash
<headius[m]>
Trevor159: open a new bug for your issue so we can keep all this information there
<headius[m]>
I know JRuby works on arm64 (aarch64) but I don't have an armv7l environment handy to try
<Trevor159>
I have two different devices
<Trevor159>
it does not work on both
<Trevor159>
one is 64 bit
<Trevor159>
it also does not have an OS on it atm
<Trevor159>
I should clarify they are both arm
<headius[m]>
ok
<headius[m]>
show us the error you get trying to start up (use gist.github.com)
<headius[m]>
I'll double check my arm64 env
<Trevor159>
I am looking into jruby stand alone, if the issue persists I will open a new issue
<headius[m]>
ok
<headius[m]>
if it does not, open an issue with logstash and copy me
<headius[m]>
I handle most of the native/FFI stuff on JRuby
<Trevor159>
There already is an issue at least tied to the issue I linked prior
<headius[m]>
yeah PPC though
<headius[m]>
we have had ARM support longer than PPC (I'm not even sure we work properly on all PPC envs)
<Trevor159>
Jun 24 22:14:56 odroid logstash[4100]: [2019-06-24T22:14:56,067][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) load error: ffi/ffi -- java.lang.NullPointerException: null
<headius[m]>
mmm
<Trevor159>
still need to try stand alone
<headius[m]>
yeah that will get you a better error
<headius[m]>
logstash is hiding some details
<headius[m]>
my rebuild on arm64 is almost done
<headius[m]>
Trevor159: the command line to test: bin/jruby -e 'require "ffi/ffi"'
<headius[m]>
that appears to work fine for me on arm64
<headius[m]>
Linux bouncer 4.4.105-mainline-rev1 #1 SMP Sat Dec 9 19:10:34 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux
<headius[m]>
actually jruby -rffi -e 1 should be enough to trigger the error if it's there
<headius[m]>
`jruby -rffi -e 1`
<Trevor159>
I am being a little slow gimme a sec :d
<headius[m]>
NPE should never happen regardless so we have something to fix there
<headius[m]>
Trevor159: no worries...I'm back from vacay this week so I'll be here every day
<lopex>
headius[m]: ah, I also looked at it, spent an hour and came up with nothing
<lopex>
so many layers
<headius[m]>
it's strange
<lopex>
the transcoder seemed ok though
<headius[m]>
I'm wondering if it's failing to get a valid UTF-16 replacement character on some environment, since that would cause it to giv eup on replacement
<lopex>
might be the data again ?
<lopex>
unlikely though
<headius[m]>
but it works here
<headius[m]>
it's seems like it has to be some encoding mismatch but I don't know where
<lopex>
but it failed for me
<headius[m]>
oh you can repro locally?
<headius[m]>
without docker
<lopex>
reproed it on wndows
<headius[m]>
well that's interesting
<headius[m]>
another data point that doesn't give me any clues
<lopex>
you mean the fixture "This is a test. Iñtërnâtiônàlizætiøn\n" ?
<headius[m]>
you could try to step through the outer encoding loops and see why it doesn't do replacement
<headius[m]>
I think I've sorted out that much...it should either be raising an error about invalid chars (without replace) or just replacing them...instead it does neither and raises this ArgumentError
<lopex>
-rwxrwx---+ 1 lopex Brak 83 Apr 8 00:04 encoding.rb
<headius[m]>
I have a patch locally that may help but not fix
<lopex>
to prove I have it locally :P
<headius[m]>
I noticed a place MRI raises error that we don't
<headius[m]>
I'll show my patch and maybe it changes things for you
<headius[m]>
Trevor159: hard to tell from that output, but perhaps you could just use a binary JRuby?
<headius[m]>
failure to build is a separate issue 😆
<Trevor159>
fair
<headius[m]>
like download a tarball from jruby.org or use a ruby installer
<headius[m]>
I'll have to sign off soon for the evening but if you can reproduce using a binary dist using that command line above (also commented on the PPC bug) you should open a new issue and we'll investigate
<headius[m]>
I'll try to find an armv7l env to try
<headius[m]>
ooo
<headius[m]>
my arm cloud has docs about armv7l but I con't see it in available instances ☹️
<headius[m]>
damn, raspi3 is also arm 8
<Trevor159>
am I just missing it or does rbenv not have jruby 9.2.7.0
<headius[m]>
hmmm what else do I have lying around here
<headius[m]>
Trevor159: it definitely should have it
<headius[m]>
jruby-9.2.7.0
<headius[m]>
anything that can install Rubies should know about 9.2.7.0
<Trevor159>
ruby-build: definition not found: jruby-9.2.7.0
<headius[m]>
if that works then we need to get logstash folks involved
<Trevor159>
does jruby just take that as an argument?
<headius[m]>
lopex: I suppose the next step is to tweak his docker rig so I can fetch and build JRuby master
<headius[m]>
Trevor159: just download that file and run it like `jruby gettimeofday.rb1`
<headius[m]>
er, minus the 1
<Trevor159>
it runs
<headius[m]>
ok
<headius[m]>
I'd recommend opening a new bug with logstash, let them know JRuby FFI seems to work on its own, and tag me
<headius[m]>
it is likely some od packaging or configuration issue with their embedded JRuby
<rtyler>
headius[m]: you don't have some raspberry pis running around like everybody else does? :P
<headius[m]>
I do but they're all aarch64/armv8
<headius[m]>
my beaglebone might not be though
<headius[m]>
aha
<Trevor159>
issue made, thanks for your help
<headius[m]>
Trevor159: If it comes back around, seems like maybe my old beaglebone board is armv7
<headius[m]>
we'll see if logstash folks have any thoughts though
<headius[m]>
and with that I'll call it a day
<Trevor159>
Thanks for the help
<headius[m]>
👍
<headius[m]>
At least we know now that it should work
<lopex>
rpi4 is there
<lopex>
up to 4gigs of ram
<rtyler>
I thought the older raspberry pis were armv7, I still haven't ditched mine >_<
<headius[m]>
Very likely!
<headius[m]>
Oldest I have is a 3
<headius[m]>
4GB rpi sounds nice
<lopex>
the spec is quite cool
<rtyler>
agreed.
<rtyler>
headius[m]: I'm on the bus home, I believe I have a raspberry 1 model B or two at home, but I am having trouble figuring out if those are armv7
<rtyler>
most tech bloggers don't seem to include that
<headius[m]>
They probably are, I think they introduced 64-bit in raspberry pi 3
<headius[m]>
Or 2
<rtyler>
I'll probably be back online after chores and whatnot in around an hour and half. Would it help for me to try to reproduce something on these devices, or do you need direct access (also doable)
<headius[m]>
At this point, it sounds like JRuby itself is ok but logstash fails...but I'm not sure how to repro with logstash
<Trevor159>
literally just run it
<rtyler>
what's the "it" there? :P
<Trevor159>
logstash
<Trevor159>
the tarball
<Trevor159>
ill check if u even need a config
<rtyler>
I probably won't have time for 20 questions later :D do you have that issue you were referring to previously that would give me some repro steps?