From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 7B00B7ED25 for ; Fri, 19 Jul 2013 17:06:38 +0200 (CEST) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of edwin+ml-ocaml@etorok.net) identity=pra; client-ip=176.9.138.55; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="edwin+ml-ocaml@etorok.net"; x-sender="edwin+ml-ocaml@etorok.net"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of edwin+ml-ocaml@etorok.net designates 176.9.138.55 as permitted sender) identity=mailfrom; client-ip=176.9.138.55; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="edwin+ml-ocaml@etorok.net"; x-sender="edwin+ml-ocaml@etorok.net"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of postmaster@mail.etorok.net designates 176.9.138.55 as permitted sender) identity=helo; client-ip=176.9.138.55; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="edwin+ml-ocaml@etorok.net"; x-sender="postmaster@mail.etorok.net"; x-conformance=sidf_compatible; x-record-type="v=spf1" X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgMFAMlU6VGwCYo3/2dsb2JhbABbgwY1wRiBDxZ0giQBAQQBDCYBDQEBNgIPCxgJFg8JAwIBAgE3AQ0TCAKIBgoIpFaEQgEFjXMGkBaDfokljjuBKYR6iyqBWYE8 X-IPAS-Result: AgMFAMlU6VGwCYo3/2dsb2JhbABbgwY1wRiBDxZ0giQBAQQBDCYBDQEBNgIPCxgJFg8JAwIBAgE3AQ0TCAKIBgoIpFaEQgEFjXMGkBaDfokljjuBKYR6iyqBWYE8 X-IronPort-AV: E=Sophos;i="4.89,702,1367964000"; d="scan'208";a="26648258" Received: from mail.etorok.net ([176.9.138.55]) by mail2-smtp-roc.national.inria.fr with ESMTP; 19 Jul 2013 17:06:37 +0200 Received: from [172.30.42.2] (unknown [79.114.32.194]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.etorok.net (Postfix) with ESMTPSA id A4A3046D7 for ; Fri, 19 Jul 2013 17:06:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=etorok.net; s=mailout; t=1374246396; bh=6qWBJcebyedlE9CzQ95hXTPrPbRp4qw2zrqETSjvero=; l=3558; h=Date:From:To:References:In-Reply-To:From; b=dqzMUuSMlSBx+QG2qm6fRW6jfokRE/VvyxQrDTEKdpMDqC4Zwar23xxxBbk4nelI2 usC82PCs8A4YimXM3Ofo1X+lkfvfAosGH/R65RMxaCDWLf3GiKPjkkTckngjc+Wpkx x9muFsSCBtlrlhzzApcWeUSpsDNISlh5DhOPVmFk= Message-ID: <51E955FB.2020707@etorok.net> Date: Fri, 19 Jul 2013 18:06:35 +0300 From: =?ISO-8859-1?Q?T=F6r=F6k_Edwin?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130630 Icedove/17.0.7 MIME-Version: 1.0 To: caml-list@inria.fr References: <51E9398A.9010402@inria.fr> <51E94B76.6070207@etorok.net> <51E94EBA.609@inria.fr> In-Reply-To: <51E94EBA.609@inria.fr> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 0.97.8 at mail X-Virus-Status: Clean Subject: Re: [Caml-list] Request for feedback: Procord, a library to delegate tasks to other processes On 07/19/2013 05:35 PM, Romain Bardou wrote: > Le 19/07/2013 16:21, Török Edwin a écrit : >> On 07/19/2013 04:05 PM, Romain Bardou wrote: >>> Hello, >>> >>> I plan on writing yet another library to help with concurrency in OCaml. >>> The motivations for this library, and the interface I have in mind, are >>> available here: >>> >>> http://romain.bardou.fr/procord/api/Procord.html >>> >>> Before actually implementing the library, I would be very happy to >>> receive feedback. I am interested to know, among others: >>> - whether I miss important information which would make the very >>> existence of this library stupid (such as, it already exists); >>> - whether I forgot some important use case; >> >> Processing streams of data in parallel without having to read all the data in memory first. > > Good point. I'm not sure how to deal with this and I actually think it > calls for a slightly different paradigm. Although maybe one could > eventually add send/receive functions to send/receive to/from an 'a > process, and the same for the worker. It would allow to transmit more > than one input and more than one output. Yeah I think it'd need a concept that feeding input to a worker won't necessarely produce an immediate output (for example: compressing a file). And you'd need a way to signal EOF. I don't know how this could be done in a generic way (for an 'a), but maybe something could be done for strings: provide the worker with a way to repeatedly read strings (like Pervasives.input), and to repeatedly write output (like Pervasives.output). If you can use pipes then they can even be Pervasives.input and Pervasives.output themselves, otherwise some buffering logic will be needed. The nice thing about this is that you could quite easily turn something like Cryptokit's transform into a worker, even if you are reading/writing from/to the network. > >> You could sort of do this with your current interface but only on Unix (i.e. mkfifo and pass filename). >> Don't know what'd work on Windows, perhaps creating a (named) pipe, and sending the file descriptor to a newly spawned child? > > I don't think there are named pipes on Windows but I may be wrong. Maybe > it can be emulated using files anyway. I don't think sending a file > descriptor is a good idea but sending a file name is of course possible. > >> Also it'd be nice to have something similar to the reduce in map-reduce. > > That's definitely something I want to be able to add easily. It won't be > in the initial version, as I have no need for it right now, but it is > important for me that it can easily be added, maybe as a new layer on > top of Procord itself. > >>> - whether the names I chose have better ubiquitous equivalents; >>> - whether you believe I should choose another interface entirely, for >>> instance, if you don't like functors. >> >> I'd prefer if there was also a Lwt-like monadic interface, but it is not absolutely required (it could probably be done on top of your already existing interface). > > That is also something I want to be able to add easily, but I don't want > Procord to depend on anything and to force one library. The user should > be able to choose Async or Lwt or whatever. This is why I propose > get_file_descriptors and process_status. It should be easy to write a > Lwt using that. > > It would be possible to provide separate modules Procord_lwt and > Procord_async as interfaces though. > > Thanks for your inputs! > > Best regards, >