From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr X-Spam-Level: X-Spam-Status: No, score=0.1 required=5.0 tests=AWL autolearn=disabled version=3.1.3 Received: from concorde.inria.fr (concorde.inria.fr [192.93.2.39]) by yquem.inria.fr (Postfix) with ESMTP id CCF2BBC69 for ; Wed, 30 May 2007 13:31:46 +0200 (CEST) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.188]) by concorde.inria.fr (8.13.6/8.13.6) with ESMTP id l4UBVkIT001107 for ; Wed, 30 May 2007 13:31:46 +0200 Received: from [84.59.96.136] (helo=gate.lan.gerd-stolpmann.de) by mrelayeu.kundenserver.de (node=mrelayeu4) with ESMTP (Nemesis), id 0ML21M-1HtMP035Ba-00013L; Wed, 30 May 2007 13:31:42 +0200 Received: from flakew.lan.gerd-stolpmann.de (fw.lan.gerd-stolpmann.de [192.168.1.1]) by gate.lan.gerd-stolpmann.de (Postfix) with ESMTP id 7B265C16C; Wed, 30 May 2007 13:31:42 +0200 (CEST) Subject: Re: [Caml-list] Faking concurrency using Unix forks and pipes From: Gerd Stolpmann To: Florian Hars Cc: Jon Harrop , caml-list@inria.fr In-Reply-To: <465D2436.5060703@bik-gmbh.de> References: <200705300442.59906.jon@ffconsultancy.com> <465D2436.5060703@bik-gmbh.de> Content-Type: text/plain Date: Wed, 30 May 2007 13:31:41 +0200 Message-Id: <1180524702.1470.40.camel@localhost.localdomain> Mime-Version: 1.0 X-Mailer: Evolution 2.6.1 Content-Transfer-Encoding: 7bit X-Provags-ID: V01U2FsdGVkX1/U7P61Hu8VLS6KSC0itGQAlytWFEQqVYSnTMb nUbYtSef/InbZTPz1rc0o9yocqsqNhah5Q8aiNM7niUyqNUQ7Y osWm6DjvHmFzo42oJZZn1E+KyHnORaO X-Miltered: at concorde with ID 465D60A2.000 by Joe's j-chkmail (http://j-chkmail . ensmp . fr)! X-Spam: no; 0.00; forks:01 gerd:01 stolpmann:01 hars:01 ocaml:01 forks:01 marshalling:01 toying:01 functor:01 functor:01 sig:01 val:01 ocamlnet:01 sockets:01 model:01 Am Mittwoch, den 30.05.2007, 09:13 +0200 schrieb Florian Hars: > Jon Harrop schrieb: > > Has anyone implemented a parallel map function in OCaml using Unix forks, > > pipes and maybe marshalling? > > I've been toying with something like > > module type MR = > functor (D1: DICT_TYPE) -> > functor (D2: DICT_TYPE) -> > sig > val mapreduce : > (D1.key -> D1.value -> (D2.key * 'a) list) -> > ( D2.key -> 'a list -> D2.value option) -> > D1.t -> > D2.t > end Interesting example... We are going to implement a distributed mapreduce on a cluster using ocamlnet (and its components rpc and netplex). Netplex is a framework to listen to a number of sockets and to start/manage processes dealing with the TCP connections (using a generalized pre-fork model). We are already using it for our distributed web crawler. Gerd > and have one implementation that forks a set of worker processes > and then marshalls data of types > type 'a command = Quit | Execute of 'a > type ('a, 'b) response = Result of 'a | Error of 'b > through a pipe. > > Not very robust right now, but it seems to work if none of the > workers dies prematurely... > > Yours Florian. > > _______________________________________________ > Caml-list mailing list. Subscription management: > http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list > Archives: http://caml.inria.fr > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs > -- ------------------------------------------------------------ Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany gerd@gerd-stolpmann.de http://www.gerd-stolpmann.de Phone: +49-6151-153855 Fax: +49-6151-997714 ------------------------------------------------------------