Re: [Caml-list] Request for feedback: Procord, a library to delegate tasks to other processes

["Török Edwin" <edwin+ml-ocaml@etorok.net>]

On 07/19/2013 05:35 PM, Romain Bardou wrote:
> Le 19/07/2013 16:21, Török Edwin a écrit :
>> On 07/19/2013 04:05 PM, Romain Bardou wrote:
>>> Hello,
>>>
>>> I plan on writing yet another library to help with concurrency in OCaml.
>>> The motivations for this library, and the interface I have in mind, are
>>> available here:
>>>
>>> http://romain.bardou.fr/procord/api/Procord.html
>>>
>>> Before actually implementing the library, I would be very happy to
>>> receive feedback. I am interested to know, among others:
>>> - whether I miss important information which would make the very
>>> existence of this library stupid (such as, it already exists);
>>> - whether I forgot some important use case;
>>
>> Processing streams of data in parallel without having to read all the data in memory first.
> 
> Good point. I'm not sure how to deal with this and I actually think it
> calls for a slightly different paradigm. Although maybe one could
> eventually add send/receive functions to send/receive to/from an 'a
> process, and the same for the worker. It would allow to transmit more
> than one input and more than one output.

Yeah I think it'd need a concept that feeding input to a worker won't necessarely produce an immediate output (for example: compressing a file). And you'd need a way to signal EOF. 

I don't know how this could be done in a generic way (for an 'a), but maybe something could be done for strings: provide the worker with a way to repeatedly read strings (like Pervasives.input), and to repeatedly write output (like Pervasives.output). 
If you can use pipes then they can even be Pervasives.input and Pervasives.output themselves, otherwise
some buffering logic will be needed.

The nice thing about this is that you could quite easily turn something like Cryptokit's transform into a worker, even if you are reading/writing from/to the network.

> 
>> You could sort of do this with your current interface but only on Unix (i.e. mkfifo and pass filename).
>> Don't know what'd work on Windows, perhaps creating a (named) pipe, and sending the file descriptor to a newly spawned child?
> 
> I don't think there are named pipes on Windows but I may be wrong. Maybe
> it can be emulated using files anyway. I don't think sending a file
> descriptor is a good idea but sending a file name is of course possible.
> 
>> Also it'd be nice to have something similar to the reduce in map-reduce.
> 
> That's definitely something I want to be able to add easily. It won't be
> in the initial version, as I have no need for it right now, but it is
> important for me that it can easily be added, maybe as a new layer on
> top of Procord itself.
> 
>>> - whether the names I chose have better ubiquitous equivalents;
>>> - whether you believe I should choose another interface entirely, for
>>> instance, if you don't like functors.
>>
>> I'd prefer if there was also a Lwt-like monadic interface, but it is not absolutely required (it could probably be done on top of your already existing interface).
> 
> That is also something I want to be able to add easily, but I don't want
> Procord to depend on anything and to force one library. The user should
> be able to choose Async or Lwt or whatever. This is why I propose
> get_file_descriptors and process_status. It should be easy to write a
> Lwt using that.
> 
> It would be possible to provide separate modules Procord_lwt and
> Procord_async as interfaces though.
> 
> Thanks for your inputs!
> 
> Best regards,
>