From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id D66CA8018B for ; Tue, 4 Jul 2017 10:56:21 +0200 (CEST) Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=berenger@bioreg.kyushu-u.ac.jp; spf=None smtp.mailfrom=berenger@bioreg.kyushu-u.ac.jp; spf=None smtp.helo=postmaster@h4.hosting4.cc.kyushu-u.ac.jp Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of berenger@bioreg.kyushu-u.ac.jp) identity=pra; client-ip=133.5.13.5; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="berenger@bioreg.kyushu-u.ac.jp"; x-sender="berenger@bioreg.kyushu-u.ac.jp"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of berenger@bioreg.kyushu-u.ac.jp) identity=mailfrom; client-ip=133.5.13.5; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="berenger@bioreg.kyushu-u.ac.jp"; x-sender="berenger@bioreg.kyushu-u.ac.jp"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@h4.hosting4.cc.kyushu-u.ac.jp) identity=helo; client-ip=133.5.13.5; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="berenger@bioreg.kyushu-u.ac.jp"; x-sender="postmaster@h4.hosting4.cc.kyushu-u.ac.jp"; x-conformance=sidf_compatible IronPort-PHdr: =?us-ascii?q?9a23=3A75sgixaVb4/5489jQrtZo+H/LSx+4OfEezUN459i?= =?us-ascii?q?sYplN5qZr8mybnLW6fgltlLVR4KTs6sC0LuJ9fi4EUU7or+5+EgYd5JNUxJXwe?= =?us-ascii?q?43pCcHRPC/NEvgMfTxZDY7FskRHHVs/nW8LFQHUJ2mPw6arXK99yMdFQviPgRp?= =?us-ascii?q?OOv1BpTSj8Oq3Oyu5pHfeQtFiT6/bL9oMBm6sRjau9ULj4dlNqs/0AbCrGFSe+?= =?us-ascii?q?RRy2NoJFaTkAj568yt4pNt8Dletuw4+cJYXqr0Y6o3TbpDDDQ7KG81/9HktQPC?= =?us-ascii?q?TQSU+HQRVHgdnwdSDAjE6BH6WYrxsjf/u+Fg1iSWIdH6QLYpUjul86pmRh3lhS?= =?us-ascii?q?keOzIl/2zcl8h8gaJHrB6koRF03ozab5yPNPdmY63TY90aS2pCUMhfWSNODYGz?= =?us-ascii?q?YJcAAecaIeZVrZPwq0cSoRawBwShAv7kxD9Shn/x2K03y+QvERvc0wwmA90Ot3?= =?us-ascii?q?XUrM7oP6oPXu670KbGwy3CYf1ZxTn29Y/FfQs/rvGWQ71wd8XRxlc1Fw7elVqQ?= =?us-ascii?q?qIvlPymL2eQCqWSb7OphVf+0i24ntgF9uyWvyt02hYbVnI4VyEjE+Dx/zY0oK9?= =?us-ascii?q?O4T0t7bsSlEJtWryyaOIp2Qt8iQ2F1oyk20KEJuZm+fCUM1Z8pxAbfZuSaf4WG?= =?us-ascii?q?7B/vTuacLS1liH9rYr6yhw6+/VWux+DyTMW50ExGojRYntTPrHwByhPe58uBR/?= =?us-ascii?q?Bg5EmuwyyP2BrW6uxcIUA7i67bK5k5z74wmZcfq0TDHijtmEXqja+WcEok9+ey?= =?us-ascii?q?5+v7frrqvJycNo9pigHlMqQuh9WzAfkiPgcUQ2eX4+a81KX78U34RrVFkOE2n7?= =?us-ascii?q?HEvJ3UK8kXvK60DxVI3ost8RqyDy+q0NECknkGKFJFdgiHj4/sO1zWO//3E+2/?= =?us-ascii?q?jEqqkDdk2f/GJKbsDY7MLnjEirfuY7l951RfyAsz0NBf/JJUCqwYLPLuQEP+rs?= =?us-ascii?q?DXAgU7Mwys2+rnEsly1psCWWKTBa+UKL/dvkWN5uIrOuWMYI4VuC3hK/U+/P7v?= =?us-ascii?q?jXo5mUcHcqWz3JsXbmq4HvV8LEmDb3rsmIRJLWBftQM7SKnuiUaeeT9VfXe7Ga?= =?us-ascii?q?wmtR8hD4fzI53KQIuqgafJ9gqBM7t5S1oOXlqBC3rueIiec/0NbzifLYligyEf?= =?us-ascii?q?VaXkVsko3lesrFmpmPJcMuPI93hA5trY399v6riOzRw=3D?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0AlBADuV1tZlwUNBYVcHAEBBAEBCgEBF?= =?us-ascii?q?wEBBAEBCgEBhBEDhHqLDJBeliKCEQEnhXQCg0QWAQEBAQEBAQEBAQESAQEBAQE?= =?us-ascii?q?GGAZXgjMMDIJOAwMjBBFRJQImAgIfOBMIAQGKKhGwE4FsOodtGoNAAQsBIAWBC?= =?us-ascii?q?4IchVgLiCSCR4JhBYdIB4oBjTYehymMPoIMhUqDToZ6lTMmDYEugQqFbIIBihE?= =?us-ascii?q?BAQE?= X-IPAS-Result: =?us-ascii?q?A0AlBADuV1tZlwUNBYVcHAEBBAEBCgEBFwEBBAEBCgEBhBE?= =?us-ascii?q?DhHqLDJBeliKCEQEnhXQCg0QWAQEBAQEBAQEBAQESAQEBAQEGGAZXgjMMDIJOA?= =?us-ascii?q?wMjBBFRJQImAgIfOBMIAQGKKhGwE4FsOodtGoNAAQsBIAWBC4IchVgLiCSCR4J?= =?us-ascii?q?hBYdIB4oBjTYehymMPoIMhUqDToZ6lTMmDYEugQqFbIIBihEBAQE?= X-IronPort-AV: E=Sophos;i="5.40,307,1496095200"; d="scan'208";a="281939640" Received: from hosting4.cc.kyushu-u.ac.jp (HELO h4.hosting4.cc.kyushu-u.ac.jp) ([133.5.13.5]) by mail2-smtp-roc.national.inria.fr with ESMTP; 04 Jul 2017 10:55:51 +0200 Received: from [192.168.2.36] (unknown [133.5.218.148]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: berenger@bioreg.kyushu-u.ac.jp) by h4.hosting4.cc.kyushu-u.ac.jp (hde-lc-postfix) with ESMTPSA id 070A12AC895; Tue, 4 Jul 2017 17:55:48 +0900 (JST) (envelope-from berenger@bioreg.kyushu-u.ac.jp) To: caml-list@inria.fr References: <20170703140437.8FFD7121499@mcclellan.cs.miami.edu> From: Francois BERENGER Message-ID: <978f88e8-6e9c-9f54-5988-0933a4cac833@bioreg.kyushu-u.ac.jp> Date: Tue, 4 Jul 2017 17:55:47 +0900 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 MIME-Version: 1.0 In-Reply-To: <20170703140437.8FFD7121499@mcclellan.cs.miami.edu> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: [Caml-list] parany: a minimalistic library to parallelize any kind of computation Dear caml riders, I am pleased to announce parany, a kind of minimalistic and more generic version of parmap. Yes, a minimalistic version of parmap is possible! :D Parany can be found here: https://github.com/UnixJunkie/parany The super simple interface is: --- (* the demux function must throw End_of_input once it's done *) exception End_of_input val run: nprocs:int -> demux:(unit -> 'a) -> work:('a -> 'b) -> mux:('b -> unit) -> unit --- This is a beta release, so please don't expect excellent performance and rock stability. Also, maybe it can crash in some cases (tell me). The difference with parmap is that parany is supposed to be able to work with very large files (that you cannot load in memory at once) or infinite streams of things to process. Managing such cases is doable with parmap but requires some programming and is not optimal in terms of parallelization performance (you have to stop all work when loading in memory a part of your file and fork all your workers again after that). Parany relies on the shared data structure Netmcore_queue from Gerd Stolpmann's excellent ocamlnet library to take care of all the magic. Contributions are welcome to improve performance and stability, while not degrading code quality. Also, since this is a minimalistic library, I am not sure new features will be accepted. ;) If there is some interest, I can put it into opam, let me know. Regards, F.