From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=AWL autolearn=disabled version=3.1.3 Received: from concorde.inria.fr (concorde.inria.fr [192.93.2.39]) by yquem.inria.fr (Postfix) with ESMTP id 39005BC69 for ; Wed, 30 May 2007 14:03:21 +0200 (CEST) Received: from furbychan.cocan.org (furbychan.cocan.org [80.68.91.176]) by concorde.inria.fr (8.13.6/8.13.6) with ESMTP id l4UC3K5Z009575 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO) for ; Wed, 30 May 2007 14:03:20 +0200 Received: from rich by furbychan.cocan.org with local (Exim 3.35 #1 (Debian)) id 1HtMtb-0007Wi-00 for ; Wed, 30 May 2007 13:03:19 +0100 Date: Wed, 30 May 2007 13:03:19 +0100 To: caml-list@inria.fr Subject: Re: [Caml-list] Faking concurrency using Unix forks and pipes Message-ID: <20070530120310.GA24085@furbychan.cocan.org> References: <200705300442.59906.jon@ffconsultancy.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200705300442.59906.jon@ffconsultancy.com> User-Agent: Mutt/1.5.9i From: Richard Jones X-Miltered: at concorde with ID 465D6808.001 by Joe's j-chkmail (http://j-chkmail . ensmp . fr)! X-Spam: no; 0.00; forks:01 0100,:01 ocaml:01 forks:01 marshalling:01 ocaml:01 parsing:01 parsing:01 overnight:98 heap:01 unix:01 unix:01 wrote:01 caml-list:01 concurrency:02 On Wed, May 30, 2007 at 04:42:59AM +0100, Jon Harrop wrote: > > Has anyone implemented a parallel map function in OCaml using Unix forks, > pipes and maybe marshalling? > > This seems like an easy way to get concurrency in OCaml... Don't particularly have any code to show but kind of: at Merjis we did something like this using both MPI and the Ancient module. (Not at the same time). MPI was used to parallelise a simple log parsing job across a cluster of machines. This wasn't very successful as it turned out because I got bitten by Amdahl's law - the thing that was taking the time was the serial parsing part of the job. Second approach was to parse the logfiles overnight into a file-backed memory heap using Ancient. Then we used several processes to do the analysis. The processes were forked, but accessed the file-backed shared memory directly (so no pipes). Rich. -- Richard Jones Red Hat