From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail4-relais-sop.national.inria.fr (mail4-relais-sop.national.inria.fr [192.134.164.105]) by walapai.inria.fr (8.13.6/8.13.6) with ESMTP id p7HJQLxp011363 for ; Wed, 17 Aug 2011 21:26:21 +0200 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgADAKQVTE5KfVM2kGdsb2JhbABBAZkpNoZ9AYgOCBQBAQEBCQkNBxQEIYFAAQEBAQIBEgIsARsQAggDAQMBCwYFCxoMAhMiAREBBQEKEgYTEhCHTgSZEgqMN4JVhRg7iG0CAwaDUAGCcQSCUZBCjFk8g3w X-IronPort-AV: E=Sophos;i="4.68,240,1312149600"; d="scan'208";a="105698956" Received: from mail-gw0-f54.google.com ([74.125.83.54]) by mail4-smtp-sop.national.inria.fr with ESMTP/TLS/RC4-SHA; 17 Aug 2011 21:26:15 +0200 Received: by gwb15 with SMTP id 15so637706gwb.27 for ; Wed, 17 Aug 2011 12:26:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=ueqK+a+myZMp6UMHcI6MLGlcA7jKdws+CjUoaFnyrJw=; b=bmT2N0eDkoKKEfzJ2Hxvi8k3sxdOwAYicnrwGARHhio4eZtr/x9sBAz8lgaXui6fyW zey7CB+hYcAtfaZF6PskoXzby40nFRr9ApgSO9WB1muyLj8j49GnTRGv0KU0z1S8KUe7 VqQn837Lbn2+4HRn6C+7hvvEbAQkarVnJh2LE= Received: by 10.236.177.70 with SMTP id c46mr4355503yhm.219.1313609174133; Wed, 17 Aug 2011 12:26:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.146.167.12 with HTTP; Wed, 17 Aug 2011 12:25:53 -0700 (PDT) In-Reply-To: <20110817171035.GA8231@voyager> References: <20110817171035.GA8231@voyager> From: Ashish Agarwal Date: Wed, 17 Aug 2011 15:25:53 -0400 Message-ID: To: Roberto Di Cosmo Cc: caml-list@inria.fr, marcod@di.unipi.it Content-Type: multipart/alternative; boundary=20cf303f6a1832abe004aab878c8 Subject: Re: [Caml-list] [ANN]: Parmap --20cf303f6a1832abe004aab878c8 Content-Type: text/plain; charset=ISO-8859-1 Thanks. This is very exciting. I tried a simple test but get an error: # #require "extlib";; # #require "parmap";; # Parmap.parfold (+) (List.of_enum (1 -- 1000)) 0 ~ncores:1;; Signal -10 The function List.of_enum is from Batteries. I use it just to create a long list of integers. Also, you need to add extlib to the META file. On Wed, Aug 17, 2011 at 1:10 PM, Roberto Di Cosmo wrote: > Dear all, > a few lines to announce the availability of a minimalistic library > which > can be useful to exploit your multicore processor with minimal > modifications to > your OCaml programs. > > In a nutshell > ------------- > > If you want to use your many cores to accelerate an operation which happens > to > be a map, fold or map/fold (map-reduce), just use Parmap's parmap, parfold > and > parmapfold primitives in place of the standard List.map and friends, and > specify > the number of subprocesses to use by the optional parameter ncores. > > For example, in the classical Mandelbrot example present in the example > directory, > the line > > Parmap.parmap pixel tasks ~ncores:i > > allows to spawn i separate processes, each working on 1/ith of the list > tasks. > > Rationale > --------- > > The principle of Parmap is very simple: when you call one of the three > available > primitives, map, fold, and mapfold , your OCaml sequential program forks > in n > subprocesses (you choose the n), and each subprocess performs the > computation on > the 1/n of the data, returing the results through a shared memory area > to the > parent process, that resumes execution once all the children have > terminated, > and the data has been recollected. > > This means that you *must* run your program on a *single* multicore > machine. > Repeat after us: Parmap is not meant to run on a cluster, see one of the > many > available (re)implementations of the map-reduce schema for that. > > By forking the parent process on a sigle machine, the children get > access, for > free, to all the data structures already built, even the imperative ones, > and as > far as your computation inside the map/fold does not produce side effects > that > need to be preserved, the final result will be the same as > performing the > sequential operation, the only difference is that you might get it faster. > > Of course, if you happen to have open channels, or files, or other > connections > that should only be used by the parent process, your program may behave > in a > very wierd way: as an example, *do not* open a graphic window before > calling a > Parmap primitive, and *do not* use this library if your program > is > multi-threaded! > > The OCaml code is quite simple and does not rely on any external C > library: all > the magic is done by your operating system's fork and memory mapping > mechanisms. > One could gain some speed by implementing a marshal/unmarshal operation > directly > on bigarrays, but we did not do this yet. > > > How to get it > ------------- > > Project home: https://gitorious.org/parmap > > To compile and install: > > git clone git://gitorious.org/parmap/parmap.git > make > make install > > Enjoy > > -- Marco Danelutto and Roberto Di Cosmo > > P.S.: special thanks to Pierre Chambart for useful discussions on this code > > > > > > > -- > Caml-list mailing list. Subscription management and archives: > https://sympa-roc.inria.fr/wws/info/caml-list > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs > > --20cf303f6a1832abe004aab878c8 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks. This is very exciting. I tried a simple test but get an error:
=
# #require "extlib";;
# #require &q= uot;parmap";;
# Parmap.parfold (+) (List.of_enum (1 -- 1000)= ) 0 ~ncores:1;;
Signal -10

The function List.of_enum is from = Batteries. I use it just to create a long list of integers.

<= /div>
Also, you need to add extlib to the META file.



On Wed, Aug 17, 2011 at= 1:10 PM, Roberto Di Cosmo <roberto@dicosmo.org> wrote:
Dear all,
=A0 =A0 =A0 a few lines to announce the availability of a minimalistic lib= rary which
can be useful to exploit your multicore processor with minimal modification= s to
your OCaml programs.

In a nutshell
-------------

If you want to use your many cores to accelerate an operation which happens= to
be a map, fold or map/fold (map-reduce), just use Parmap's parmap, parf= old and
parmapfold primitives in place of the standard List.map and friends, and sp= ecify
the number of subprocesses to use by the optional parameter ncores.

For example, in the classical Mandelbrot example present in the example dir= ectory,
the line

=A0 =A0 =A0 =A0Parmap.parmap pixel tasks ~ncores:i

allows to spawn i separate processes, each working on 1/ith of the list tas= ks.

Rationale
---------

The principle of Parmap is very simple: when you call one of the three avai= lable
primitives, map, fold, and =A0mapfold , your OCaml =A0sequential program fo= rks =A0in n
subprocesses (you choose the n), and each subprocess performs the computati= on on
the =A01/n of the data, returing =A0the results through a =A0shared memory = area to the
parent process, that resumes =A0execution once all =A0the children =A0have = terminated,
and the data has been recollected.

This means that you *must* run your program on a *single* multicore machine= .
Repeat after us: Parmap is not meant to run on a cluster, see one of the ma= ny
available (re)implementations of the map-reduce schema for that.

By forking the parent process =A0on a sigle =A0machine, the children get ac= cess, for
free, to all the data structures already built, even the imperative ones, a= nd as
far as your computation =A0inside the map/fold =A0does not produce side eff= ects that
need =A0to be =A0preserved, the =A0final result will =A0 be the same =A0as = =A0performing the
sequential operation, the only difference is that you might get it faster.<= br>
Of course, if you happen =A0to have open =A0channels, or files, or other co= nnections
that should only be =A0used by the parent =A0process, your program =A0may b= ehave in =A0a
very wierd way: as an example, *do =A0not* open a =A0graphic window before = calling a
Parmap primitive, and =A0 *do =A0 not* =A0use =A0this =A0library =A0 if =A0= your =A0program =A0 =A0is
multi-threaded!

The OCaml code is quite simple and does not rely on any =A0external C libra= ry: all
the magic is done by your operating system's fork and memory mapping me= chanisms.
One could gain some speed by implementing a marshal/unmarshal operation dir= ectly
on bigarrays, but we did not do this yet.


How to get it
-------------

Project home: ht= tps://gitorious.org/parmap

To compile and install:

=A0git clone git://gitorious.org/parmap/parmap.git
=A0make
=A0make install

Enjoy

-- Marco Danelutto and Roberto Di Cosmo

P.S.: special thanks to Pierre Chambart for useful discussions on this code=






--
Caml-list mailing list. =A0Subscription management and archives:
https://sympa-roc.inria.fr/wws/info/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs


--20cf303f6a1832abe004aab878c8--