[Caml-list] [ANN]: Parmap

[Caml-list] [ANN]: Parmap

[Roberto Di Cosmo]

Dear all,
       a few lines to announce the availability of a minimalistic library which
can be useful to exploit your multicore processor with minimal modifications to
your OCaml programs.

In a nutshell
-------------

If you want to use your many cores to accelerate an operation which happens to
be a map, fold or map/fold (map-reduce), just use Parmap's parmap, parfold and
parmapfold primitives in place of the standard List.map and friends, and specify
the number of subprocesses to use by the optional parameter ncores.

For example, in the classical Mandelbrot example present in the example directory,
the line 

	Parmap.parmap pixel tasks ~ncores:i

allows to spawn i separate processes, each working on 1/ith of the list tasks.

Rationale
---------

The principle of Parmap is very simple: when you call one of the three available
primitives, map, fold, and  mapfold , your OCaml  sequential program forks  in n
subprocesses (you choose the n), and each subprocess performs the computation on
the  1/n of the data, returing  the results through a  shared memory area to the
parent process, that resumes  execution once all  the children  have terminated,
and the data has been recollected.

This means that you *must* run your program on a *single* multicore machine.
Repeat after us: Parmap is not meant to run on a cluster, see one of the many
available (re)implementations of the map-reduce schema for that.

By forking the parent process  on a sigle  machine, the children get access, for
free, to all the data structures already built, even the imperative ones, and as
far as your computation  inside the map/fold  does not produce side effects that
need  to be  preserved, the  final result will   be the same  as  performing the
sequential operation, the only difference is that you might get it faster.

Of course, if you happen  to have open  channels, or files, or other connections
that should only be  used by the parent  process, your program  may behave in  a
very wierd way: as an example, *do  not* open a  graphic window before calling a
Parmap primitive, and   *do   not*  use  this  library   if  your  program    is
multi-threaded!

The OCaml code is quite simple and does not rely on any  external C library: all
the magic is done by your operating system's fork and memory mapping mechanisms.
One could gain some speed by implementing a marshal/unmarshal operation directly
on bigarrays, but we did not do this yet.


How to get it
-------------

Project home: https://gitorious.org/parmap

To compile and install:

 git clone git://gitorious.org/parmap/parmap.git
 make
 make install

Enjoy

-- Marco Danelutto and Roberto Di Cosmo

P.S.: special thanks to Pierre Chambart for useful discussions on this code

Re: [Caml-list] [ANN]: Parmap

[Ashish Agarwal]

[-- Attachment #1: Type: text/plain, Size: 3578 bytes --]

Thanks. This is very exciting. I tried a simple test but get an error:

# #require "extlib";;
# #require "parmap";;
# Parmap.parfold (+) (List.of_enum (1 -- 1000)) 0 ~ncores:1;;
Signal -10

The function List.of_enum is from Batteries. I use it just to create a long
list of integers.

Also, you need to add extlib to the META file.



On Wed, Aug 17, 2011 at 1:10 PM, Roberto Di Cosmo <roberto@dicosmo.org>wrote:

> Dear all,
>       a few lines to announce the availability of a minimalistic library
> which
> can be useful to exploit your multicore processor with minimal
> modifications to
> your OCaml programs.
>
> In a nutshell
> -------------
>
> If you want to use your many cores to accelerate an operation which happens
> to
> be a map, fold or map/fold (map-reduce), just use Parmap's parmap, parfold
> and
> parmapfold primitives in place of the standard List.map and friends, and
> specify
> the number of subprocesses to use by the optional parameter ncores.
>
> For example, in the classical Mandelbrot example present in the example
> directory,
> the line
>
>        Parmap.parmap pixel tasks ~ncores:i
>
> allows to spawn i separate processes, each working on 1/ith of the list
> tasks.
>
> Rationale
> ---------
>
> The principle of Parmap is very simple: when you call one of the three
> available
> primitives, map, fold, and  mapfold , your OCaml  sequential program forks
>  in n
> subprocesses (you choose the n), and each subprocess performs the
> computation on
> the  1/n of the data, returing  the results through a  shared memory area
> to the
> parent process, that resumes  execution once all  the children  have
> terminated,
> and the data has been recollected.
>
> This means that you *must* run your program on a *single* multicore
> machine.
> Repeat after us: Parmap is not meant to run on a cluster, see one of the
> many
> available (re)implementations of the map-reduce schema for that.
>
> By forking the parent process  on a sigle  machine, the children get
> access, for
> free, to all the data structures already built, even the imperative ones,
> and as
> far as your computation  inside the map/fold  does not produce side effects
> that
> need  to be  preserved, the  final result will   be the same  as
>  performing the
> sequential operation, the only difference is that you might get it faster.
>
> Of course, if you happen  to have open  channels, or files, or other
> connections
> that should only be  used by the parent  process, your program  may behave
> in  a
> very wierd way: as an example, *do  not* open a  graphic window before
> calling a
> Parmap primitive, and   *do   not*  use  this  library   if  your  program
>    is
> multi-threaded!
>
> The OCaml code is quite simple and does not rely on any  external C
> library: all
> the magic is done by your operating system's fork and memory mapping
> mechanisms.
> One could gain some speed by implementing a marshal/unmarshal operation
> directly
> on bigarrays, but we did not do this yet.
>
>
> How to get it
> -------------
>
> Project home: https://gitorious.org/parmap
>
> To compile and install:
>
>  git clone git://gitorious.org/parmap/parmap.git
>  make
>  make install
>
> Enjoy
>
> -- Marco Danelutto and Roberto Di Cosmo
>
> P.S.: special thanks to Pierre Chambart for useful discussions on this code
>
>
>
>
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

[-- Attachment #2: Type: text/html, Size: 4581 bytes --]

Re: [Caml-list] [ANN]: Parmap

[Roberto Di Cosmo]

Dear Ashish,
	thanks for the report; these issues are fixed in the latest
commit on https://gitorious.org/parmap  (bfdf714)

Let me know if you find any more issues

All the best

--Roberto

On Wed, Aug 17, 2011 at 03:25:53PM -0400, Ashish Agarwal wrote:
> Thanks. This is very exciting. I tried a simple test but get an error:
> 
> # #require "extlib";;
> # #require "parmap";;
> # Parmap.parfold (+) (List.of_enum (1 -- 1000)) 0 ~ncores:1;;
> Signal -10
> 
> The function List.of_enum is from Batteries. I use it just to create a long
> list of integers.
> 
> Also, you need to add extlib to the META file.
> 
> 
> 
> On Wed, Aug 17, 2011 at 1:10 PM, Roberto Di Cosmo <roberto@dicosmo.org> wrote:
> 
>     Dear all,
>           a few lines to announce the availability of a minimalistic library
>     which
>     can be useful to exploit your multicore processor with minimal
>     modifications to
>     your OCaml programs.
> 
>     In a nutshell
>     -------------
> 
>     If you want to use your many cores to accelerate an operation which happens
>     to
>     be a map, fold or map/fold (map-reduce), just use Parmap's parmap, parfold
>     and
>     parmapfold primitives in place of the standard List.map and friends, and
>     specify
>     the number of subprocesses to use by the optional parameter ncores.
> 
>     For example, in the classical Mandelbrot example present in the example
>     directory,
>     the line
> 
>            Parmap.parmap pixel tasks ~ncores:i
> 
>     allows to spawn i separate processes, each working on 1/ith of the list
>     tasks.
> 
>     Rationale
>     ---------
> 
>     The principle of Parmap is very simple: when you call one of the three
>     available
>     primitives, map, fold, and  mapfold , your OCaml  sequential program forks
>      in n
>     subprocesses (you choose the n), and each subprocess performs the
>     computation on
>     the  1/n of the data, returing  the results through a  shared memory area
>     to the
>     parent process, that resumes  execution once all  the children  have
>     terminated,
>     and the data has been recollected.
> 
>     This means that you *must* run your program on a *single* multicore
>     machine.
>     Repeat after us: Parmap is not meant to run on a cluster, see one of the
>     many
>     available (re)implementations of the map-reduce schema for that.
> 
>     By forking the parent process  on a sigle  machine, the children get
>     access, for
>     free, to all the data structures already built, even the imperative ones,
>     and as
>     far as your computation  inside the map/fold  does not produce side effects
>     that
>     need  to be  preserved, the  final result will   be the same  as  
>     performing the
>     sequential operation, the only difference is that you might get it faster.
> 
>     Of course, if you happen  to have open  channels, or files, or other
>     connections
>     that should only be  used by the parent  process, your program  may behave
>     in  a
>     very wierd way: as an example, *do  not* open a  graphic window before
>     calling a
>     Parmap primitive, and   *do   not*  use  this  library   if  your  program
>        is
>     multi-threaded!
> 
>     The OCaml code is quite simple and does not rely on any  external C
>     library: all
>     the magic is done by your operating system's fork and memory mapping
>     mechanisms.
>     One could gain some speed by implementing a marshal/unmarshal operation
>     directly
>     on bigarrays, but we did not do this yet.
> 
> 
>     How to get it
>     -------------
> 
>     Project home: https://gitorious.org/parmap
> 
>     To compile and install:
> 
>      git clone git://gitorious.org/parmap/parmap.git
>      make
>      make install
> 
>     Enjoy
> 
>     -- Marco Danelutto and Roberto Di Cosmo
> 
>     P.S.: special thanks to Pierre Chambart for useful discussions on this code
> 
> 
> 
> 
> 
> 
>     --
>     Caml-list mailing list.  Subscription management and archives:
>     https://sympa-roc.inria.fr/wws/info/caml-list
>     Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>     Bug reports: http://caml.inria.fr/bin/caml-bugs
> 
> 
> 

-- 
--Roberto Di Cosmo
 
------------------------------------------------------------------
Professeur               En delegation a l'INRIA
PPS                      E-mail: roberto@dicosmo.org
Universite Paris Diderot WWW  : http://www.dicosmo.org
Case 7014                Tel  : ++33-(0)1-57 27 92 20
5, Rue Thomas Mann       
F-75205 Paris Cedex 13   Identica: http://identi.ca/rdicosmo
FRANCE.                  Twitter: http://twitter.com/rdicosmo
------------------------------------------------------------------
Attachments:
   MIME accepted
   Word deprecated, http://www.rfc1149.net/documents/whynotword
------------------------------------------------------------------
Office location:
 
Bureau 6C08 (6th floor)
175, rue du Chevaleret, XIII
Metro Chevaleret, ligne 6
------------------------------------------------------------------