sidx64_ has quit [Read error: Connection reset by peer]
<JulesIvanicGitte>
yes
<JulesIvanicGitte>
Even the `9.2.6.0` tag doesn’t compile for me 🤔
lucasb has joined #jruby
rperez has joined #jruby
<rperez>
Hello!! I have problems with ActiveMailer and jruby, can anyone help me or give any recomendations?
jmalves has joined #jruby
<JulesIvanicGitte>
@rperez Can we have more details plz ?
<rperez>
Thanks! I have this error: "no constructor for arguments (org.jruby.runtime.Frame,org.jruby.RubyClass,org.jruby.runtime.scopes.DynamicScope2,org.jruby.runtime.backtrace.BacktraceElement) on Java::OrgJrubyRuntime::Binding"
<rperez>
with this instruction: UsuarioMailer.bienvenido_email(@user).deliver_later
<rperez>
I send an email with mail gem, there isn't problem. I think it is an incompatibility with ActionMailer configuration and Jruby
<JulesIvanicGitte>
(edited) ... leak to ... => ... leak problem to ...
<JulesIvanicGitte>
(edited) ... garbage collect problem => ... garbage collection problem
<JulesIvanicGitte>
(edited) It’d move my thread leak problem to a memory garbage collection problem => I’m a total newbie but why do you boot a thread per fiber ? It doesn’t seems to be the definition of fibers
<kares[m]>
not sure what you mean ... LW means not backed by a thread?
<JulesIvanicGitte>
yes
<JulesIvanicGitte>
just a simple instance of a class
<kares[m]>
to correct myself ... I was testing bounded pool (in JRuby 1.7) and it did not make a difference
<kares[m]>
but you need to switch execution context and pop back
<kares[m]>
how do you want to do that light-weightly?
<JulesIvanicGitte>
I really don’t know how to implement Fibers. I don’t know this subject so I can really tell.
<JulesIvanicGitte>
How I understand it for now is that Fibers are logical threads
<kares[m]>
its possible in a compiled language -> kotlin does it
<JulesIvanicGitte>
because they’re just a simple object they can be GCed
<kares[m]>
but you need to know ahead of time which parts (functions/methods) will be fiber-ish
<kares[m]>
fibers are gc-d just fine
<kares[m]>
the threads end up being re-used
<JulesIvanicGitte>
not in my case sadly :/
<kares[m]>
that likely means you have a leak somewhere ...
<kares[m]>
sometimes its hard to resolve without having all of the code at hand
<kares[m]>
well at least for me - guessing from thread/heap dumps :)
<headius[m]>
Oh sorry I have unpushed fixes
<JulesIvanicGitte>
😅
<headius[m]>
I was fixing in the car on the way to my halfway vacay
<JulesIvanicGitte>
Driving and coding, you have to choose :D
<JulesIvanicGitte>
(I think you have to be French to understand the joke 😅)
rperez has joined #jruby
<headius[m]>
haha
<headius[m]>
I get it
<JulesIvanicGitte>
:)
<headius[m]>
did you run into something that hit that bug?
<headius[m]>
it should only affect when passing an Enumerator to Java as an Iterator
<JulesIvanicGitte>
No I didn't
<headius[m]>
ok
<headius[m]>
well I pushed fixes for it anyway...almost mergeable
<headius[m]>
I see there's some questions about why we implement fibers this way
<headius[m]>
the main problem here is that at least one library you're running, biz, creates these enumerators, starts them up, and then just walks away from them
rperez has quit [Client Quit]
<headius[m]>
in CRuby's case they can do that because they swap the entire fiber off the thread and use the same thread to execute other stuff
<headius[m]>
we can't do that because JVM has no such capability, so every fiber has its own thread and if you walk away from it it's still there holding a thread
<headius[m]>
The coroutines in Kotlin and Scala are done using bytecode tricks and static typing...every function that needs to be pauseable and resumable in a coroutine has to be generated with special bytecode that doesn't actually deepen the call stack
<headius[m]>
kares pretty much covered the reasons why we can't do what they do
<headius[m]>
the next interesting project in this area would be to use openjdk project loom for our fibers
<JulesIvanicGitte>
Hum. Super interesting :)
<JulesIvanicGitte>
Thanks for taking the time to explain all this!
<JulesIvanicGitte>
> the next interesting project in this area would be to use openjdk project loom for our fibers
<JulesIvanicGitte>
Yup totally agree
<headius[m]>
Jules Ivanic (Gitter): so I gathered from skimming messages that the branch did not appear to help your situation
<headius[m]>
I've pushed last fixes I know of
<JulesIvanicGitte>
No it doesn’t help.
<JulesIvanicGitte>
Maybe that you pushed eveything, it’ll. Don’t know. I have to rebuild a JRuby version and test but I’m on something else right now.
<JulesIvanicGitte>
We’re also exploring the “biz” side of the problem
<headius[m]>
I doubt these fixes will change anything
<JulesIvanicGitte>
maybe there’s a bug in the “biz” gem
<headius[m]>
it's possible they could do something in a more lightweight way but my reading of that library is that they heavily lean on enumerator-as-fiber
<headius[m]>
I can't imagine it's very efficient in CRuby either
<headius[m]>
but it wouldn't use up too many threads
<headius[m]>
Jules Ivanic (Gitter): are you able to reproduce the leaking outside of your app?
<headius[m]>
like with a simple script calling biz
<JulesIvanicGitte>
> are you able to reproduce the leaking outside of your app?
<headius[m]>
that would be a big help
<JulesIvanicGitte>
Didn’t try
<JulesIvanicGitte>
hum good idea
<headius[m]>
ok that would be really helpful
<headius[m]>
if we can come up with specific cases leading to those leaking threads we can mitigate it, or if there's a bug we can fix that
<headius[m]>
I have another possible theory that if these enumerators really are left hanging, it might be possible to .rewind to reset them and free the thread
<headius[m]>
sort of like an enum.abort
<headius[m]>
rewind may save us for these scoped cases that only want partial enumerator results
<headius[m]>
Jules Ivanic (Gitter): can you post your biz config somewhere?
<JulesIvanicGitte>
> ok that would be really helpful
<JulesIvanicGitte>
I’m asking to one of my dev if he can help us. Not sure he’ll accept. It’s 8pm here :/
<JulesIvanicGitte>
Huhu we’re lucky
<JulesIvanicGitte>
he’ll write the script
lucasb has quit [Quit: Connection closed for inactivity]
<JulesIvanicGitte>
He’s finishing eating
<JulesIvanicGitte>
(edited) ... finishing eating => ... finishing to eat
<JulesIvanicGitte>
> can you post your biz config somewhere?
<JulesIvanicGitte>
Which configuration ?
<headius[m]>
well when I played with biz the first thing it needed was a config
<JulesIvanicGitte>
because of `:each>>:map>:reject>:flat_map>:reject>>:until` ?
<JulesIvanicGitte>
ok
<headius[m]>
yeah that's what it amounts to internally as it stacks each level of transformation
<headius[m]>
so it has this infinite generator, multiple levels of lazy on top to filter and alter results, and then finally a terminating condition of some kind
<headius[m]>
I'm trying to sort out how far to_a gets...and whether it's supposed to be rewinding something
sidx64 has joined #jruby
<headius[m]>
yikes
<headius[m]>
yeah this library is the heaviest use of enumerators, generators, and lazy I've seen
<headius[m]>
ok I think I'm starting to sort out where the issue is
<headius[m]>
it's largely this "periods" enum they use in linear.rb
<headius[m]>
in this case that's week_periods
<headius[m]>
that appears to be the primary place where they end up doing #next, and then those periods enums are left hanging
<headius[m]>
so the to_a does drain the Linear enumerator, but it flat_maps against this other enumerator and never shuts it down
<headius[m]>
each layer of a lazy enumerator does create one of these anonymous block-based enumerators that cause us trouble
<headius[m]>
BUT
<headius[m]>
if you only use that lazy with .each or other normal internal iteration methods, it will be fine
<headius[m]>
the outer one, the one you get from Linear, does get run this way
<headius[m]>
but it is constructed with two other enumerators, "periods" and "shifts", and it only pulls a few items from each during its run
<headius[m]>
so I think the "fix" or workaround would be to get the outer enumerator to know about those inner ones so you can reset the whole thing at once
<headius[m]>
I'm trying to get it to propagate a rewind call all the way in
drbobbeaty has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
sidx64 has quit [Ping timeout: 246 seconds]
yaauie has joined #jruby
<yaauie>
What can I do to accelerate someone helping me get started with the Open3 issue I filed a few weeks ago (https://github.com/jruby/jruby/issues/5642#)? It's blocking Logstash installs on Java 11 and preventing our test suite from validating behaviour. I'm glad to do the brunt of the work, but need pointers with someone more familiar with JRuby core.
<JulesIvanicGitte>
@yaauie sorry to ask but why don’t you stay on JDK8 ?
<yaauie>
Java 8 has reached end-of-life and is no longer providing security-related updates. Our customers and community are wanting to run on supported JDKs
<JulesIvanicGitte>
untrue
<JulesIvanicGitte>
AdoptOpenJDK continue to support JDK8
<JulesIvanicGitte>
I strongly advise you to use their builds
nirvdrum has quit [Ping timeout: 245 seconds]
<JulesIvanicGitte>
The official OpenJDK8 build aren’t up to date
<JulesIvanicGitte>
The latest version is u202 while OpenJDK only provides u181
<JulesIvanicGitte>
Some important fixes made in JDK10 about its integration with Docker where backported in JDK8u191
<JulesIvanicGitte>
(edited) Some important fixes made in JDK10 about its integration with Docker where backported in JDK8u191 => The official OpenJDK8 builds aren’t up to date
nirvdrum has joined #jruby
<JulesIvanicGitte>
(edited) The official OpenJDK8 builds aren’t up to date => Some important fixes made in JDK10 about its integration with Docker were backported in JDK8u191
<yaauie>
Arguments about forks and distributions of the JDK aside, I need to support Oracle JDK11 and OpenJDK11, and am hitting bugs with the Open3 implementation while doing so.
<JulesIvanicGitte>
can’t help on this side. Sorry.
rperez has joined #jruby
rperez has quit [Client Quit]
<headius[m]>
yaauie: hi there
<headius[m]>
yaauie: if that was your reason for moving to 9+ then you really don't need to
<headius[m]>
there are many companies other than Oracle providing 8 builds with security updates, usually with lower or no cost
<headius[m]>
we will get 9 working well for these things but short term workaround for you could be to use 8
<headius[m]>
I do follow that you need to support 11
<headius[m]>
and I want to help :-)
<headius[m]>
it's just a really busy time right now before RubyKaigi and RailsConf
<headius[m]>
kares: I wonder if you have some cycles to help yaauie look into this Java 9+ Open3.popen2 hanging?
<headius[m]>
Jules Ivanic (Gitter): Ok so I have spent the time on this I can for now
<headius[m]>
I'm going to update the issue with everything I know and we should get the Biz folks involved
<headius[m]>
I think we can make Biz behave better if it cleans up after itself when these lazy enumerators complete
<headius[m]>
I'd say we take your small example and start a conversation with Biz folks about what we can do to improve how it cleans up these temporary enumerators
<headius[m]>
yaauie: so
<yaauie>
@headius I saw your replies on the ticket. Thanks much for following up and getting it kicked off. We're currently pointing our users to the Java 8 workaround for installation failures on Debian, and looking at ways we can route around the problem in the short-term by spawning the sub-processes ourselves without Open3's convenience.
<headius[m]>
so if we use pure native pipe descriptors it seems to work fine
<headius[m]>
I propose a likely root cause there along with a patch to do just that
<headius[m]>
we have not used pure native IO.pipe up to now because most IO were still just using normal JDK channels...but for subprocess stuff, if we can't unwrap those channels we can't properly interact with the native process calls like posix_spawn
<JulesIvanicGitte>
@headius Thanks ! :)
<headius[m]>
yaauie: JRuby patch and Ruby workaround provided
<headius[m]>
we'll have to discuss the right path forward
<headius[m]>
sorry I didn't get time to look into this...these things are often only 15 minute deals for me (since I wrote 90% of this stuff) but as I mentioned it's been a busy month 😓