Re: Interpreter vs hardware threads

[Julian Assange <proff@iq.org>]

Xavier Leroy <Xavier.Leroy@inria.fr> writes:

> >    IIRC, Linux native pthreads, for one, aren't at the moment meant to
> > be used in huge numbers---the flood-test performance of those Linux
> > Java VMs which map Java threads onto them supposedly suffers compared
> > with those that don't.  But Xavier would be able to tell us more about
> > that :).
> 
> My pleasure :-)  It is true that threads in LinuxThreads are not as
> lightweight as some would like, mostly because they map 1-to-1 to
> kernel processes, and the Linux kernel limits the number of processes
> to 512 (for the super-user) or 256 (for normal users).  Also, kernel
> scheduling takes time linear in the number of processes (just like the
> OCaml bytecode-thread scheduler, which was strongly inspired by that
> of the Linux kernel), so context switching times increase as more
> threads are run.

I recently worked on a project in C that used a FSM to simulate
several thousand concurrent IO threads, as part of a concurrent
version of dig (able to recursively zone transfer all of *.au in under
three minutes). Due to the complicated, but inherently forward moving
nature of the operations involved (i.e do this. if there is no error
or timeout, do the next step, and so on for about 30 steps) together
with inter-state resources (i.e various queues, buffers and so on) this
ended up as an extremely large and unintuitive piece of code. A lot of
the added complexity was a result of having to explicity deallocate
data and maintain reference counts, and manually follow allocation
dependencies, which in the circumstances of multiple interdependent
event queues and states became extremely tedious.

What is the source of the linearity of thread context switches in
ocaml?  Are there ways to eliminate it? If so I'd be happy to have a
look at doing so.


Cheers,
Julian.

--
Stefan Kahrs in [Kah96] discusses the
   notion of completeness--programs which never go wrong can be
   type-checked--which complements Milner's notion of
   soundness--type-checked programs never go wrong [Mil78].