Re: [Caml-list] Nproc: process pools for OCaml (request for suggestions)

[Martin Jambon <martin.jambon@ens-lyon.org>]

On 11/30/2011 01:35 AM, Jerome Vouillon wrote:
> Hi,
> 
> On Tue, Nov 29, 2011 at 06:54:46PM -0800, Martin Jambon wrote:
>> I would like to publicize Nproc, which is an implementation of process
>> pools for OCaml based on fork, pipes, Marshal and Lwt:
>>
>>   https://github.com/MyLifeLabs/nproc
>>
>> Using Nproc involves:
>>
>> 1. Creating a pool of N processes, N being chosen by the user.
>> 2. Running tasks:
>>   a. Submitting a task (f, x) of any type.
>>   b. Defining what to do when the result becomes available.
> 
> Marco Danelutto and Roberto Di Cosmo have written a small library to
> perform parallel maps and folds on multi-core machines. This is
> complementary to your library. You should look at the implementation:
> communication is performed by marshalling to a shared memory area for
> better performances, and pipes are used only for synchronization.
>
>      https://gitorious.org/parmap/
>
>

Thank you, this is interesting. I hadn't look at the implementation
although I knew about parmap but wanted something with an Lwt-ready
interface. I also wanted something that could handle continuous streams,
without having to create a process for each stream item (assuming fork()
is more expensive than we want).


> I have also written a more low-level library, where you can control
> which process runs each task. This is useful when the processes have
> to work with a lot of data that you don't want to duplicate. You can
> get the code with the following command:
> 
>      darcs clone http://www.pps.jussieu.fr/~vouillon/coinst/darcs/dev/
> 
> The file of interest is task.ml.

I see. Thanks.


Martin