Re: [Caml-list] Faking concurrency using Unix forks and pipes

[Gerd Stolpmann <info@gerd-stolpmann.de>]

Am Mittwoch, den 30.05.2007, 09:13 +0200 schrieb Florian Hars:
> Jon Harrop schrieb:
> > Has anyone implemented a parallel map function in OCaml using Unix forks, 
> > pipes and maybe marshalling?
> 
> I've been toying with something like
> 
> module type MR =
>   functor (D1: DICT_TYPE) ->
>     functor (D2: DICT_TYPE) ->
> sig
>   val mapreduce :
>     (D1.key -> D1.value -> (D2.key * 'a) list) ->
>     ( D2.key -> 'a list -> D2.value option) ->
>     D1.t ->
>     D2.t
> end

Interesting example... We are going to implement a distributed mapreduce
on a cluster using ocamlnet (and its components rpc and netplex).

Netplex is a framework to listen to a number of sockets and to
start/manage processes dealing with the TCP connections (using a
generalized pre-fork model). We are already using it for our distributed
web crawler.

Gerd

> and have one implementation that forks a set of worker processes
> and then marshalls data of types
> type 'a command = Quit | Execute of 'a
> type ('a, 'b) response = Result of 'a | Error of 'b
> through a pipe.
> 
> Not very robust right now, but it seems to work if none of the
> workers dies prematurely...
> 
> Yours Florian.
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------