Re: Threading and SharedMem (Re: [Caml-list] Re: Is OCaml fast?)

[Gerd Stolpmann <info@gerd-stolpmann.de>]

Am Dienstag, den 30.11.2010, 13:55 +0100 schrieb
oliver@first.in-berlin.de:
> On Tue, Nov 30, 2010 at 09:10:36AM +0100, Stephan Houben wrote:
> > On 11/29/2010 04:33 PM, Oliver Bandel wrote:
> > >Zitat von "Gerd Stolpmann" <info@gerd-stolpmann.de>:
> > >
> > >>Am Montag, den 29.11.2010, 17:12 +0100 schrieb Oliver Bandel:
> > >>>Zitat von "Gerd Stolpmann" <info@gerd-stolpmann.de>:
> > >>>
> > 
> > >>>You use shared mem(?), but you link only to *.ml files,
> > >>>and I see no *.c there.
> > 
> > >>>How can this be done?
> > >>>
> > >>>At least not via the libs that are shipped with OCaml?!
> > 
> > Actually it can be done using the libs that ship with OCaml
> > (Unix and Bigarray), although it is not 100% POSIX :
> > 
> > let create_shared_genarray kind layout dims =
> >   let fd = Unix.openfile "/dev/zero" [Unix.O_RDWR] 0
> >   in let ar = Bigarray.Genarray.map_file fd kind layout true dims
> >   in Unix.close fd; ar
> > 
> > 
> > The resulting bigarray object is shared among subsequent forks.
> 
> Hmhhh... we started talking about Threads and SharedMem.
> You mean even fork.... hmhhh

Independent processes are right now the only way to use several cores.
You can organize shared memory between processes, but it is tricky.
That's what I try to ease with my Netmulticore library.

> > This relies on the fact that mmap-ing /dev/zero is equivalent
> > to an anonymous mmap.
> > 
> > http://en.wikipedia.org/wiki//dev/zero
> > 
> > Well, at least it works on Linux.
> 
> In APUE it's mentioned that memory mapped regions are inherited
> by a child, when forking it. So it should work on all Unix-systems too.

Yes, but is not defined by POSIX what mapping /dev/zero means.

> There is one problem with this... when you have forked, then
> you obviously have separated processes and also in each process
> your own ocaml-program with it's own GC running...
> 
> ..with such a mem-mapping trick (never used Bigarray, so I'm astouned it uses
> mmap) 

Bigarrays can use any memory with fixed addresses. That's the essence
here: Bigarrays are not moved around by the GC.

> you then have independent processes, working on shared mem without
> synchronisation.
> 
> This is a good possibility to get corrupted data, and therefore unreliable behaviour.
> 
> So, you have somehow to create a way of communicating of these processes.

So you need inter-process synchronization primitives, like POSIX
semaphores.

> This already is easily done in the Threads-module, because synchronisation
> mechanisms are bound there to the OCaml API and can be used easily.
> 
> In the Unix module there is not much of ths IPC stuff...

But in Ocamlnet's netsys module.

> (A thread-specific GC for thread-specific variables would help here,
>  making global locks only necessary when accessing global used variables.
>  But I don't know if such a way would be possible without changing the GC-stuff
>  itself.)

The global lock does not protect user variables, but the Ocaml runtime,
e.g. the state of the memory manager/garbage collector. Also it eases
code generation - the memory image needs not be in a consistent state
all the time (i.e. all pointers meaningful), but only when the runtime
gets a hand on it. Removing this lock has far-reaching consequences.

The oc4mc (ocaml for multicore) project used a separate minor heap per
thread, which actually eases the task a lot - memory is in most cases
allocated in the minor heap anyway. Many variables keeping the state of
the runtime are then thread-local.

Gerd

> 
> 
> Ciao,
>    Oliver
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 


-- 
------------------------------------------------------------
Gerd Stolpmann, Bad Nauheimer Str.3, 64289 Darmstadt,Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------