[Caml-list] In need of serious help regarding threading

[Caml-list] In need of serious help regarding threading

[Jonathan Roewen]

Hi,

I hope someone with experience in the guts of the ocaml systhreads
package can help me out.

I've tried adapting the win32_threads.c source in systhreads to work
with DST. I've implemented a C co-op threading module, which appears
to function correctly according to the tests I've done.

However, I can't seem to be able to get ocaml threads to play along
nicely. I'm out of ideas, and really need some help >_<

All code is available online (http://glek.net/subversion/os/kernel/)
and can be checked out with an svn client (using URL above). For
building, you need to first do make boot (takes a while on lesser
machines like my PIII 600). Also, I copied hooks.h and signals.h from
the byterun folder to ocaml/libs/ocaml/caml (needed private values for
signals.h). And removed the bytecode variants of whatever was present.

The file with the ocaml threading code is
http://glek.net/subversion/os/kernel/ocamlthreads.c, and the C co-op
threading code is http://glek.net/subversion/os/kernel/threads.c

I would very much appreciate anyone who can take the time to look at
this, and discover what I'm undoubtedly doing wrong. This is a major
area of DST to be implemented, and without threading in OCaml, we're
pretty much stuck in the middle of no-man's land.

For once this is a legitimate cry for help ;-) I know I've sent some
stupid, stupid emails in the past....

Thanks to all willing to help,

Jonathan

Re: [Caml-list] In need of serious help regarding threading

[Jonathan Roewen]

It appears (with the help of a friend good with a debugger) that
there's something going wrong with my code, and how it's interacting
with the GC.

I have a debug compiled version of my kernel, the disassembly, sources
to the C threading code, and some notes from the guy about the
debugging @ http://moonbeam.purevoid.org/~jonathan/ocaml/

If someone can point out what I'm doing wrong, it'd be the biggest
life-saver I could ever imagine!

Jonathan

On 8/19/05, Jonathan Roewen <jonathan.roewen@gmail.com> wrote:
> Hi,
> 
> I hope someone with experience in the guts of the ocaml systhreads
> package can help me out.
> 
> I've tried adapting the win32_threads.c source in systhreads to work
> with DST. I've implemented a C co-op threading module, which appears
> to function correctly according to the tests I've done.
> 
> However, I can't seem to be able to get ocaml threads to play along
> nicely. I'm out of ideas, and really need some help >_<
> 
> All code is available online (http://glek.net/subversion/os/kernel/)
> and can be checked out with an svn client (using URL above). For
> building, you need to first do make boot (takes a while on lesser
> machines like my PIII 600). Also, I copied hooks.h and signals.h from
> the byterun folder to ocaml/libs/ocaml/caml (needed private values for
> signals.h). And removed the bytecode variants of whatever was present.
> 
> The file with the ocaml threading code is
> http://glek.net/subversion/os/kernel/ocamlthreads.c, and the C co-op
> threading code is http://glek.net/subversion/os/kernel/threads.c
> 
> I would very much appreciate anyone who can take the time to look at
> this, and discover what I'm undoubtedly doing wrong. This is a major
> area of DST to be implemented, and without threading in OCaml, we're
> pretty much stuck in the middle of no-man's land.
> 
> For once this is a legitimate cry for help ;-) I know I've sent some
> stupid, stupid emails in the past....
> 
> Thanks to all willing to help,
> 
> Jonathan
>

Re: [Caml-list] In need of serious help regarding threading

[Jonathan Roewen]

Out of curiosity: is my lack of the "tick" thread the reason everything breaks?

Jonathan

On 8/19/05, Jonathan Roewen <jonathan.roewen@gmail.com> wrote:
> It appears (with the help of a friend good with a debugger) that
> there's something going wrong with my code, and how it's interacting
> with the GC.
> 
> I have a debug compiled version of my kernel, the disassembly, sources
> to the C threading code, and some notes from the guy about the
> debugging @ http://moonbeam.purevoid.org/~jonathan/ocaml/
> 
> If someone can point out what I'm doing wrong, it'd be the biggest
> life-saver I could ever imagine!
> 
> Jonathan
> 
> On 8/19/05, Jonathan Roewen <jonathan.roewen@gmail.com> wrote:
> > Hi,
> >
> > I hope someone with experience in the guts of the ocaml systhreads
> > package can help me out.
> >
> > I've tried adapting the win32_threads.c source in systhreads to work
> > with DST. I've implemented a C co-op threading module, which appears
> > to function correctly according to the tests I've done.
> >
> > However, I can't seem to be able to get ocaml threads to play along
> > nicely. I'm out of ideas, and really need some help >_<
> >
> > All code is available online (http://glek.net/subversion/os/kernel/)
> > and can be checked out with an svn client (using URL above). For
> > building, you need to first do make boot (takes a while on lesser
> > machines like my PIII 600). Also, I copied hooks.h and signals.h from
> > the byterun folder to ocaml/libs/ocaml/caml (needed private values for
> > signals.h). And removed the bytecode variants of whatever was present.
> >
> > The file with the ocaml threading code is
> > http://glek.net/subversion/os/kernel/ocamlthreads.c, and the C co-op
> > threading code is http://glek.net/subversion/os/kernel/threads.c
> >
> > I would very much appreciate anyone who can take the time to look at
> > this, and discover what I'm undoubtedly doing wrong. This is a major
> > area of DST to be implemented, and without threading in OCaml, we're
> > pretty much stuck in the middle of no-man's land.
> >
> > For once this is a legitimate cry for help ;-) I know I've sent some
> > stupid, stupid emails in the past....
> >
> > Thanks to all willing to help,
> >
> > Jonathan
> >
>

Re: [Caml-list] In need of serious help regarding threading (GC problem!)

[Jonathan Roewen]

Perhaps no one really cares, but I'll keep on going -- hopefully
someone will give in and try help me ;-) Or no-one really knows what's
wrong...

Anyways, all that aside, I've been working on test-cases some more,
eliminating code that could possibly be getting in the way, and have
now:

let start_ocaml_os gfx_mode =
  Thread.thread_initialize ();
  ignore (Thread.create
    (fun () -> while true do Console.printf "argh..."; Thread.yield (); done)
    ()
  );
  ignore (Thread.create 
    (fun () -> while true do Console.printf "boo!"; Thread.yield (); done)
    ()
  );
  Console.launch (); (* this just clears the screen pretty much, not
needed atm *)
  
  while true do (*Gc.full_major ();*) Thread.yield () done;

"start_ocaml_os" is the entry point to the ocaml kernel. The only
other ocaml code run before this point is during initialising the
ocaml runtime.

What happens is that with Gc.full_major() commented, the system
quickly falls over and dies (strange lockup, which apparently is
somewhere inside GC runtime code -- noted in previous email). If I
uncomment, the system runs flawlessly, my two ocaml threads going nuts
printing to the console. Left it running for around 5minutes before I
got truly bored and convinced it wasn't going to fall over like
before.

So, this proves that this is a GC issue, but I have no idea where to
begin. I followed the win32 systhreads implementation fairly closely,
including hints for GC that win32 code uses. (BTW, I found the posix
version a bit harder to consider implementing with the use of
condition variables, which is why I chose the win32 implementation).

Can some threading and/or GC expert have a decent look at my code
please? Again, URL to current code:
http://moonbeam.purevoid.org/~jonathan/ocaml/, and svn repo:
http://glek.net/subversion/os/kernel/

Kindest Regards,

Jonathan

On 8/19/05, Jonathan Roewen <jonathan.roewen@gmail.com> wrote:
> Out of curiosity: is my lack of the "tick" thread the reason everything breaks?
> 
> Jonathan
> 
> On 8/19/05, Jonathan Roewen <jonathan.roewen@gmail.com> wrote:
> > It appears (with the help of a friend good with a debugger) that
> > there's something going wrong with my code, and how it's interacting
> > with the GC.
> >
> > I have a debug compiled version of my kernel, the disassembly, sources
> > to the C threading code, and some notes from the guy about the
> > debugging @ http://moonbeam.purevoid.org/~jonathan/ocaml/
> >
> > If someone can point out what I'm doing wrong, it'd be the biggest
> > life-saver I could ever imagine!
> >
> > Jonathan
> >
> > On 8/19/05, Jonathan Roewen <jonathan.roewen@gmail.com> wrote:
> > > Hi,
> > >
> > > I hope someone with experience in the guts of the ocaml systhreads
> > > package can help me out.
> > >
> > > I've tried adapting the win32_threads.c source in systhreads to work
> > > with DST. I've implemented a C co-op threading module, which appears
> > > to function correctly according to the tests I've done.
> > >
> > > However, I can't seem to be able to get ocaml threads to play along
> > > nicely. I'm out of ideas, and really need some help >_<
> > >
> > > All code is available online (http://glek.net/subversion/os/kernel/)
> > > and can be checked out with an svn client (using URL above). For
> > > building, you need to first do make boot (takes a while on lesser
> > > machines like my PIII 600). Also, I copied hooks.h and signals.h from
> > > the byterun folder to ocaml/libs/ocaml/caml (needed private values for
> > > signals.h). And removed the bytecode variants of whatever was present.
> > >
> > > The file with the ocaml threading code is
> > > http://glek.net/subversion/os/kernel/ocamlthreads.c, and the C co-op
> > > threading code is http://glek.net/subversion/os/kernel/threads.c
> > >
> > > I would very much appreciate anyone who can take the time to look at
> > > this, and discover what I'm undoubtedly doing wrong. This is a major
> > > area of DST to be implemented, and without threading in OCaml, we're
> > > pretty much stuck in the middle of no-man's land.
> > >
> > > For once this is a legitimate cry for help ;-) I know I've sent some
> > > stupid, stupid emails in the past....
> > >
> > > Thanks to all willing to help,
> > >
> > > Jonathan
> > >
> >
>

Re: [Caml-list] In need of serious help regarding threading

[Xavier Leroy]

> It appears (with the help of a friend good with a debugger) that
> there's something going wrong with my code, and how it's interacting
> with the GC.

Please don't get offended by what I'm going to say, but I have the
feeling that you're attacking extremely hard problems without adequate
debugging tools and without enough understanding of the OCaml runtime
system.

I'll try to provide some explanations nonetheless, but please don't
bombard this list with too many cries for help.

What the debugging session shows is a problem with return address
determination during the stack scanning performed by the GC.  To find
heap pointers contained in the stack, the GC scans it one frame at a
time, using compiler-generated frame descriptors to locate the
pointers.  The frame descriptors are keyed to the return address in
the Caml code through a hash table (variable frame_descriptors, hash table
lookup at lines 135-141 and 249-255 in file asmrun/roots.c).

Your run appears to be looping in the hash table lookup, indicating
that 1- the return address being looked up (variable retaddr) is not
in the table (this should never happen in normal operation), and 2-
your environment lets you dereference the NULL pointer without
crashing (bad idea!).

A good way to debug this is to print the value of the "retaddr" local
variable at lines 134 and 249 in asmrun/roots.c and correlate it with your
disassembly.  It should always refer to code addresses immediately
following a "call camlModule__function" or a "call caml_call_gc"
instruction.  While you're at it, print also the "sp" variable: it
should stay within the stack of a thread.  The problem is likely to
come from wrong values of the bottom_of_stack and last_return_address
starting points for the stack walk.

Your second test (Gc.full_major() in the main thread) further suggests
that the problem does not occur if the main thread is the one calling
the GC.  Try to put Gc.full_major() in another thread to see what
happens.  That could narrow the problem to the saving and restoring of
caml_bottom_of_stack and caml_last_return_address globals during
context switches.

Finally, notice that your stacks are tiny (4096 words???).  Unless
they are protected by guard pages, expect a lot of trouble when they
overflow (they will).

- Xavier Leroy

Re: [Caml-list] In need of serious help regarding threading

[Julien Verlaguet]

I have been writing a cooperative thread system for ocaml, it only
works on linux and macosx though ...

you can find it at www.pps.jussieu.fr/~emmanuel/Public/Dev/HirondML/

I am giving you this link because I have had the same type of problems
at a time, it gave me a lot of headackes and sleepless nights ... so I
hope this will help

two little notes :

Ignore the thread migration mechanism if you are only interested in
the thread system

Do not expect nice code, the C code is full of hacks and disgusting
things that I don't understand myself anymore

I hope this helps ...

J