[Caml-list] parany: a minimalistic library to parallelize any kind of computation

[Francois BERENGER <berenger@bioreg.kyushu-u.ac.jp>]

Dear caml riders,

I am pleased to announce parany, a kind of minimalistic and more generic 
version of parmap.

Yes, a minimalistic version of parmap is possible! :D

Parany can be found here:

https://github.com/UnixJunkie/parany

The super simple interface is:
---
(* the demux function must throw End_of_input once it's done *)
exception End_of_input

val run: nprocs:int ->
          demux:(unit -> 'a) ->
          work:('a -> 'b) ->
          mux:('b -> unit) -> unit
---

This is a beta release, so please don't expect excellent performance and 
rock stability. Also, maybe it can crash in some cases (tell me).

The difference with parmap is that parany is supposed to be able
to work with very large files (that you cannot load in memory at once)
or infinite streams of things to process.

Managing such cases is doable with parmap but requires some programming
and is not optimal in terms of parallelization performance (you have to 
stop all work when loading in memory a part of your file and
fork all your workers again after that).

Parany relies on the shared data structure Netmcore_queue from
Gerd Stolpmann's excellent ocamlnet library to take care of all the magic.

Contributions are welcome to improve performance and stability, while 
not degrading code quality.

Also, since this is a minimalistic library, I am not sure
new features will be accepted. ;)

If there is some interest, I can put it into opam, let me know.

Regards,
F.