Re: [Caml-list] Question about Lwt/Async

[Malcolm Matalka <mmatalka@gmail.com>]

Gerd Stolpmann <info@gerd-stolpmann.de> writes:

> Am Mittwoch, den 09.03.2016, 07:35 +0000 schrieb Malcolm Matalka:
>> Jeremie Dimino <jdimino@janestreet.com> writes:
>> 
>> > On Tue, Mar 8, 2016 at 12:47 PM, Yaron Minsky <yminsky@janestreet.com>
>> > wrote:
>> >
>> >> Jeremie, other than having some different back-ends available (e.g., glib
>> >> main loop), how different are the approaches to backend management between
>> >> Async and Lwt?
>> >>
>> >
>> > ​The backend interfaces are slightly different​, but we just need a bit of
>> > glue in the middle. Essentially the difference is that with Lwt you provide
>> > one callback per fd and watch (read or write), while with Async you have a
>> > global callback.
>> >
>> > ​Right now what we need to change in Async to make this work is:
>> >
>> > - allow to provide a backend ​programmatically; right now you can only
>> > choose between the predefined epoll and select ones
>> > - make the scheduler ignore fds returned by the backend that are not
>> > handled by async
>> 
>> For what it's worth, which isn't much right now, I've been slowly
>> developing an interface point for event loops and user facing code.  The
>> rough idea is to present "asynchronous system calls" like an OS would,
>> so user facing code has an interface to program against and the
>> underlying event loop can change as someone wants, libev, libuv, direct
>> epoll or kqueue, etc.  So Async and Lwt libraries could be implemented
>> in terms of this interface and share the same event loop, to cooperate
>> nicely.  So far I haven't implemented anything using the interface
>> except for a barely functional test to demonstrate that it even works,
>> so it's quite raw.  And it's clearly deficient on a few things, but I
>> think the idea is sound and would alleviate some of the pain of deciding
>> to use Lwt or Async and if it works on JS or Windows or My Favorite OS
>> (just flip out the underlying scheduler implementation).
>> 
>> The work in progress around the interface can be found below, any
>> constructive feedback would be appreciated.
>> 
>> https://bitbucket.org/acslab/abb_scheduler_inf/src
>> 
>
> Very academic. The reality is different. Most of these operations are
> only provided as synchronous calls anyway in all OS I know (and you can
> only provide a non-blocking version by using helper threads). The only
> operations you can do something about are those reading/writing a file
> descriptor, but even here there is a strong OS dependency, e.g. on
> Windows async operations are very restricted, limiting implementation
> options drastically. The truth is that you cannot abstract the OS
> away.

I'm not sure how thoroughly you read the link, but each call takes a
callback to indicate when the operation is complete.  I put "system
call" in quotes because it's attempting to define a what an asynchronous
set of system calls would look like.  It also defines a "Scheduler"
which maps to an event loop.

>
> And what if I need linkat and not link? And what about calling my
> favorite C library that uses blocking I/O? E.g. I'm often preferring a
> variant of Unix.read/write using bigarrays as buffer.

Deferring something to a thread is lacking currently, one of the
"obvious deficiencies" I mentioned.  I'm not sure how this is any
different from the given state of event loops in Ocaml.

>
> I'd prefer a reduced approach for interoperability: Focus on event loops
> and ways to read/write, and accept that everything else must be dealt
> with using helper threads.

This is entirely possible given the interface I linked to, I don't see
this as being a reason not to like the interface.  In fact, the demo
implementation of the interface in the link I gave does this, even
dirtier.  It runs everything in a thread and uses some queues and cond
variables to kick off callbacks.

>
> Sorry for not being constructive. I don't like the approach (and I also
> don't like Lwt and Async, and by the way these are not the only kids on
> the block).

Definitely, I'd like more options, the hope is that an interface like
this would provide a meeting point for the various libraries.

>
> Gerd