[Caml-list] Attn: Development Editor, Latest OCaml Weekly News

Alan Schmitt
2020-06-09T08:28:55Z

[-- Attachment #1: Type: text/plain, Size: 28667 bytes --]

Hello

Here is the latest OCaml Weekly News, for the week of June 02 to 09,
2020.

Table of Contents
─────────────────

Multicore Update: April 2020, with a preprint paper
BAP 2.1.0 Release
Migrating an Async project to Lwt, a short primer
jose 0.4.0
OCaml 4.11.0, second alpha release
OCaml Workshop 2020: Call for Volunteers
Introduction to Lwt
Other OCaml News
Old CWN

Multicore Update: April 2020, with a preprint paper
═══════════════════════════════════════════════════

Archive:
[https://discuss.ocaml.org/t/multicore-update-april-2020-with-a-preprint-paper/5630/26]

Continuing this thread, Daniel Bünzli asked and KC Sivaramakrishnan replied
───────────────────────────────────────────────────────────────────────────

One thing that I didn’t get from the paper is how exactly
`ConcurMinor' breaks the current FFI and the impact it
would have on the existing eco-system, on a scale from “it
affect all projects” to “only people doing *that* fancy
thing” :–) ?

All the projects that use the C API. The details are here:
[https://github.com/ocaml-multicore/ocaml-multicore/wiki/C-API-changes]

At the end of the paper it seems you make the point that
`ParMinor' is the solution to go with for the time
being. Does this means you are going to leave behind the
work done on `ConcurMinor' or do you intend to continue to
maintain it ?

We don't intend to maintain it. It is quite a bit of work to maintain
and port the changes across two different GCs. `ParMinor' GC is now
at 4.11 branch point (the default multicore compiler is 4.10 +
ParMinor now). The `ConcMinor' is at 4.06.1.

Given that `ConcMinor' breaks the C API, the ecosystem would have to
be fixed for `ConcMinor' to be useful. The code changes are indeed
intricate; the differences are not just in the minor GC, but the
compilers internal use of the C API. It will be quite a bit of work to
keep both GCs in the same source distribution.

Guillaume Munch-Maccagnoni then said
────────────────────────────────────

Given that `ConcMinor' breaks the C API, the ecosystem
would have to be fixed for `ConcMinor' to be useful.

I do not think this is necessarily true.

Here is why I think so, but be warned that this is preliminary as I do
not have time to explore this idea further on my own at the moment.

State in Rust
╌╌╌╌╌╌╌╌╌╌╌╌╌

Breaking the C API is a consequence of deciding that all
single-threaded shared mutable state must assume they are also shared
between threads. So a new read barrier is used to promote values when
read from another thread. But for data types that were correct up to
now, users must also be careful to avoid races from now on… for
instance by avoiding sharing values of such types between domains.

One lesson of Rust is that there are different kinds of mutable state,
for different usages, with different means to achieve thread-safety.

The closest there is to current OCaml's `mutable' is the notion of
single-threaded multiple-writers mutable state (_`Cell'_). It is made
thread-safe in Rust by statically preventing values containing `Cell'
from crossing thread boundaries (by virtue of not having the _`Send'
trait_). The same restriction is used to make some data structures
more efficient by avoiding the cost of synchronisation (cf. the
reference-counting pointer `Rc' vs. the atomic reference-counting
pointer `Arc').

This is not enough by itself, and Rust offers other kinds of state for
communicating and sharing values between threads.

_`UnsafeCell'_ like Ocaml multicore's `mutable' (though yours is safe
thanks to the work on the memory model): it has almost no restriction
and can be sent across domains, but the user is likewise told to
“avoid data races”. It is rarely used alone, but together with type
abstraction it can be used to program safe concurrent data structures.

Lastly, the default notion of state in Rust is linear state, which can
be sent freely across threads. Thread-safety is ensured by restricting
aliasing using the ownership and borrowing discipline.

A backwards-compatible concurrent collector?
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

If I had to imagine a backwards-compatible OCaml with static control
of interference à la Rust based on `ConcMinor', it would distinguish
the three kinds of state (concretely with other keywords in addition
to `mutable'). `mutable' would keep its current meaning of
single-domain, multiple-writers state and not require a read barrier,
and in particular preserve the API. (I count systhreads as
single-threaded for this purpose, since here it means "sharing the
same minor heap".)

Programs could progressively transition to other kinds of state when
parallelising the program. Concretely, a data structure like
`Stack.t', instead of becoming racy, would keep its current meaning,
but users could replace it with a linear stack or a concurrent stack,
two data structures distinct from the first one, when parallelizing
their programs.

So how could this fit with the current plans? It is not entirely clear
to me. If people start to rely on parallelism in an unstructured way
(e.g. no clear distinction between different kinds of data types
arising from different ways of ensuring thread-safety) then one will
also lose the ability to retrofit `ConcMinor' in a
backwards-compatible manner (by losing the information that the
current `mutable' API is single-threaded). The API breakage of
`ConcMinor' which might only be virtual right now (if I trust this
preliminary, not fully-explored idea) will become real. (Further
difficulties arise with the emulation of the `Thread' library with
domains, but this could be changed later.)

But if users are provided in advance with a general direction for a
model of control of interference this might happen differently. And
eventually having such a model is desirable in any case, as it helps
parallelizing programs (for instance the Firefox people reported that
they had attempted and failed twice to parallelise the CSS engine in
C++ before succeeding with Rust). Furthermore, in an imaginary
retrofitting of `ConcMinor', one could imagine enforcing something
like the `Send' trait at the level of the read barrier until there is
a better way (there would be two kinds of barriers, one of which would
raise an exception if a state happened to be incorrectly shared across
domains, and not be required in the FFI).

I find `ConcMinor' interesting from a systems programming perspective
compared to the stop-the-world collector because it could (I hope)
offer possibilities such as having a low-latency domain communicating
with a higher-latency domain. Moreover the performance cost of the
read barrier might be lower in this scheme if it could be removed for
all but the concurrent data structures.

BAP 2.1.0 Release
═════════════════

Archive: [https://discuss.ocaml.org/t/ann-bap-2-1-0-release/5906/1]

Ivan Gotovchits announced
─────────────────────────

The Carnegie Mellon University Binary Analysis Platform ([CMU BAP]) is
a suite of utilities and libraries that enables analysis of programs
that are represented as machine code (aka binaries). CMU BAP is
written in OCaml and uses plugin-based architecture to enable
extensibility. We also have a domain-specific language, called Primus
Lisp, that we use to write analysis, specify verification conditions,
interact with the built-in SMT solver, and model the semantics of
machine instructions and functions.

The 2.1.0 Release is very rich in [new features] but the most
prominent addition is the new [symbolic executor] mode for the Primus
framework. We also significantly updated the Primus framework,
integrated it with our new Knowledge Base, which was introduced in the
BAP 2.0 release; we made our interpreter much faster; we added the
systems and components facilities, inspired by Common Lisp; and we
implemented a gradual type checker for Primus Lisp with type
inference. We also added an ability to represent machine instructions
as intrinsic functions so now it is possible to express their
semantics using Primus Lisp since we added IEEE754 primitives to the
Lisp interpreter.

As usual, we upgraded BAP to the newer versions of the Core library
and OCaml (we now support OCaml versions from 4.07 to 4.09). We also
significantly improved our build times and added an optional omake
backend, which we are using in-house.

From the user perspective, one of the key features of BAP as an
analysis platform is that you can run BAP on binaries that you can't
run otherwise, either because they need special hardware or software,
or need to interact with the outside world. In the past couple of
months, we have run BAP on various firmware and found numerous
zero-day vulnerabilities, particular, we were able to find critical
vulnerabilities in the VxWorks operating system that runs on,
potentially, billions of devices including mission-critical and
military appliances.

As always, questions, suggestions, and opinions are very welcome!

[CMU BAP] https://github.com/BinaryAnalysisPlatform/bap

[new features]
https://github.com/BinaryAnalysisPlatform/bap/releases/tag/v2.1.0

[symbolic executor]
https://github.com/BinaryAnalysisPlatform/bap/pull/1105

Migrating an Async project to Lwt, a short primer
═════════════════════════════════════════════════

Archive:
[https://discuss.ocaml.org/t/migrating-an-async-project-to-lwt-a-short-primer/5908/1]

Michael Bacarella announced
───────────────────────────

Consider this a post where I think aloud about my experience migrating
an Async project to Lwt. I've spent about a weekend doing such a
thing, and if, in the process of talking about it here I can save a
few people an hour or two (or perhaps inspire confidence to take such
a project on in the first place) then it will have been worthwhile.

This wouldn't be a complete post if I didn't also mention @dkim's
[translation of Real World OCaml's Async examples to Lwt]

This was born out of a previous effort where I [tried to mix Lwt and
Async in the same project]. This didn't go so well, so I tried
converting the whole thing to Lwt, and it turns out adapting to Lwt if
you're an Async person is actually much easier than I thought it would
be.

[translation of Real World OCaml's Async examples to Lwt]
https://github.com/dkim/rwo-lwt

[tried to mix Lwt and Async in the same project]
https://discuss.ocaml.org/t/best-practices-on-mixing-lwt-and-async/5372

Basics
╌╌╌╌╌╌

Both libraries involve promises/futures. Async calls its promises
`Deferred.t', whereas in Lwt they're called `Lwt.t'.

In Async you start your program by saying `never_returns (Scheduler.go
())' or `Command.async_spec' after you set up your initial
`Deferred.t'.

In Lwt you say `Lwt_main.run' on a top-level `Lwt.t' argument. Note
you can re-run `Lwt_main.run' in a single program as many times as you
want, but perhaps you shouldn't run multiple `Lwt_main.run' in
parallel.

There's an easy correspondence between basic operators.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Async Lwt
────────────────────────────────────────────────────
`Deferred.bind' `Lwt.bind'
`Deferred.return' `Lwt.return'
`>>=' `>>='
`Deferred.map' `Lwt.map'
`>>|' `>|='
`Deferred.don't_wait_for' `Lwt.async'
`In_thread.run' `Lwt_preemptive.detach'
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Starvation worries
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

The most important difference between Async and Lwt is that *fulfilled
promises are acted on immediately*, whereas Async kinda punts them to
the end of a work queue and runs their thunks later.

A return loop like this starves the rest of Lwt:

┌────
│ open Lwt.Infix
│
│ let main () =
│ let rec loop () =
│ Lwt.return ()
│ >>= fun () ->
│ loop ()
│ in
│ Lwt.async (loop ());
│ Lwt_io.printlf "this line never prints!"
│ ;;
│
│ let () = Lwt_main.run main ;;
└────

whereas the corresponding Async loop does not starve:

┌────
│ open! Async
│
│ let main () =
│ let rec loop () =
│ Deferred.return ()
│ >>= fun () ->
│ loop ()
│ in
│ don't_wait_for (loop ());
│ printf "this line does print!\n";
│ return ()
│ ;;
│
│ let () =
│ let cmd = Command.async_spec ~summary:"" Command.Spec.empty main in
│ Command.run cmd
│ ;;
└────

Fortunately there's a workaround. You can get something closer to the
Async-style behavior in Lwt by using `Lwt.yield ()' instead of
`Lwt.return ()'.

Spawning threads
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

From time to time you may need to run something in a system thread.
In Async you say `In_thread.run', whereas in Lwt you say
`Lwt_preemptive.detach'. For simple things they're pretty much
interchangeable, but one stumbling point for me was that in Async you
can create a named thread and always use that for the `In_thread.run',
with multiple simultaneous dispatches to that thread becoming
sequenced.

This is really useful for interacting with libraries that aren't so
thread friendly.

Lwt's detach doesn't provide an easy way to do this out of the box,
but I think you can still deal with thread unfriendly libraries by
using the `Lwt_preemptive.run_in_main' call.

Basically, never exit the detach thread you started to interact with
your library, and instead have it block on promise that gets filled
through run_in_main. In this way you can sequence your detached Lwt
thread similarly to Async.

Happy to explain further if this is unclear.

Other libraries
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

`Async.Unix' has a somewhat built-up conception of the UNIX API,
whereas `Lwt_main' is more a direct mapping of ocaml's `Unix' module
to promises.

Async `Clock.every' and `Clock.after' don't have exact analogs, but
you can make new versions pretty simply.

Example of a shallow imitation of Async `Clock.every'
┌────
│ let every span f =
│ Lwt.async (fun () ->
│ let span = Time.Span.to_sec span in
│ let rec loop () =
│ f ();
│ Lwt_unix.sleep span
│ >>= fun () ->
│ loop ()
│ in
│ loop ())
│ ;;
└────

*Open questions*

I haven't sorted out a good Lwt substitute that's as comfortable as
Async Pipe yet. Though some combination of Lwt_stream, Lwt_sequence
and `lwt-pipe' might fit the bill. If you just happen to know already
feel free to cluephone.

Closing remarks
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

This is basically everything? I'm almost suspicious that I'm not
having more problems, but will happily accept grace when it arises.

Raphaël Proust then said
────────────────────────

I haven’t sorted out a good Lwt substitute that’s as
comfortable as Async Pipe yet. Though some combination of
Lwt_stream, Lwt_sequence and `lwt-pipe' might fit the
bill. If you just happen to know already feel free to
cluephone.

The Tezos project has a pipe-like module:
[https://gitlab.com/tezos/tezos/-/blob/master/src/lib_stdlib/lwt_pipe.mli]
It hasn't been released as a standalone library (yet) but it is
released as part of the `tezos-stdlib' package.

I haven't used Async's pipe, so I don't know how close of a match it
is.

jose 0.4.0
══════════

Archive: [https://discuss.ocaml.org/t/ann-jose-0-4-0/5909/1]

Ulrik Strid announced
─────────────────────

A new release of JOSE has been published to opam

The following changes has been made
• RFC7638: Implement thumbprints @undu
• Make kid optional in the header and jwk to align better with the
spec, this is a breaking change

I have started dog fooding the library for a OpenID Connect client
which hopefully will help with the design going forward.

OCaml 4.11.0, second alpha release
══════════════════════════════════

Archive:
[https://discuss.ocaml.org/t/ocaml-4-11-0-second-alpha-release/5910/1]

octachron announced
───────────────────

A new alpha version of OCaml 4.11.0 has been published. Compared to
the first alpha version, this version contains the following new bug
fixes:

• *additional fixes* [6673], [1132], [+9617]: Relax the handling of
explicit polymorphic types (Leo White, review by Jacques Garrigue
and Gabriel Scherer)

• *additional fixes* [7364], [2188], [+9592], [+9609]: improvement of
the unboxability check for types with a single
constructor. Mutually-recursive type declarations can now contain
unboxed types. This is based on the paper
[https://arxiv.org/abs/1811.02300]

• [7817], [9546]: Unsound inclusion check for polymorphic variant
(Jacques Garrigue, report by Mikhail Mandrykin, review by Gabriel
Scherer)

• [9549], [9557]: Make -flarge-toc the default for PowerPC and
introduce -fsmall-toc to enable the previous behaviour. (David
Allsopp, report by Nathaniel Wesley Filardo, review by Xavier Leroy)

• [9320], [9550]: under Windows, make sure that the Unix.exec*
functions properly quote their argument lists. (Xavier Leroy, report
by André Maroneze, review by Nicolás Ojeda Bär and David Allsopp)

• [9490], [9505]: ensure proper rounding of file times returned by
Unix.stat, Unix.lstat, Unix.fstat. (Xavier Leroy and Guillaume
Melquiond, report by David Brown, review by Gabriel Scherer and
David Allsopp)

• [8676], [9594]: turn debugger off in programs launched by the
program being debugged (Xavier Leroy, report by Michael Soegtrop,
review by Gabriel Scherer)

• [9552]: restore ocamloptp build and installation (Florian Angeletti,
review by David Allsopp and Xavier Leroy)

• [7708], [9580]: Ensure Stdlib documentation index refers to
Stdlib. (Stephen Dolan, review by Florian Angeletti, report by
Hannes Mehnert)

• [9189], [9281]: fix a conflict with Gentoo build system by removing
an one-letter Makefile variable. (Florian Angeletti, report by Ralph
Seichter, review by David Allsopp and Damien Doligez)

The compiler can be installed as an OPAM switch with one of the
following commands
┌────
│ opam switch create ocaml-variants.4.11.0+alpha2 --repositories=default,beta=git+https://github.com/ocaml/ocaml-beta-repository.git
└────
or
┌────
│ opam switch create ocaml-variants.4.11.0+alpha2+<VARIANT> --repositories=default,beta=git+https://github.com/ocaml/ocaml-beta-repository.git
└────
where <VARIANT> is replaced with one of these: afl, flambda, fp,
fp+flambda

The source code for the alpha is also available at these addresses:

• [https://github.com/ocaml/ocaml/archive/4.11.0+alpha2.tar.gz]
• [https://caml.inria.fr/pub/distrib/ocaml-4.11/ocaml-4.11.0+alpha2.tar.gz]

If you find any bugs, please report them here:
[https://github.com/ocaml/ocaml/issues]

[6673] https://github.com/ocaml/ocaml/issues/6673

[1132] https://github.com/ocaml/ocaml/issues/1132

[+9617] https://github.com/ocaml/ocaml/issues/9617

[7364] https://github.com/ocaml/ocaml/issues/7364

[2188] https://github.com/ocaml/ocaml/issues/2188

[+9592] https://github.com/ocaml/ocaml/issues/9592

[+9609] https://github.com/ocaml/ocaml/issues/9609

[7817] https://github.com/ocaml/ocaml/issues/7817

[9546] https://github.com/ocaml/ocaml/issues/9546

[9549] https://github.com/ocaml/ocaml/issues/9549

[9557] https://github.com/ocaml/ocaml/issues/9557

[9320] https://github.com/ocaml/ocaml/issues/9320

[9550] https://github.com/ocaml/ocaml/issues/9550

[9490] https://github.com/ocaml/ocaml/issues/9490

[9505] https://github.com/ocaml/ocaml/issues/9505

[8676] https://github.com/ocaml/ocaml/issues/8676

[9594] https://github.com/ocaml/ocaml/issues/9594

[9552] https://github.com/ocaml/ocaml/issues/9552

[7708] https://github.com/ocaml/ocaml/issues/7708

[9580] https://github.com/ocaml/ocaml/issues/9580

[9189] https://github.com/ocaml/ocaml/issues/9189

[9281] https://github.com/ocaml/ocaml/issues/9281

OCaml Workshop 2020: Call for Volunteers
════════════════════════════════════════

Archive:
[https://discuss.ocaml.org/t/ocaml-workshop-2020-call-for-volunteers/5913/1]

Ivan Gotovchits announced
─────────────────────────

The OCaml Workshop will be held in the virtual format this year, which
poses new challenges and requires people with special talents and
training. The Organizing Committee is seeking for members who will
volunteer to fill one (or more) of the following roles:

1. AV Editor
2. Session Host
3. Transcribers/Interpreter
4. Content Manager
5. Accessibility Chair

The roles are described in details below. We are asking prospective
Organizing Committee members to contact the Organizing Committee chair
([ivg@ieee.org]([mailto:ivg@ieee.org])), indicating which role(s) they
are ready to take.

[AV Editor]
╌╌╌╌╌╌╌╌╌╌╌

AV (Audio/Video) editors are responsible for previewing the
presentations and providing help and feedback to the authors. Ideally
we target for one editor per talk.

[AV Editor] https://icfp20.sigplan.org/home/ocaml-2020#av-editor

◊ [Duties]

• Preview and (if necessary) post-process or (ask the author to shoot
again) the pre-recorded videos.
• Advise authors and help in choice of software and hardware, teach
how to set up the camera, light, make sure that the audio is of good
quality and, in general, channel our quality guidelines.
• Ensure that all videos are of the same quality, the audio levels are
the same, and that everything is loud and clear.

[Duties] https://icfp20.sigplan.org/home/ocaml-2020#duties

[Session Hosts]
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

Session hosts will assist session chairs in streaming the pre-recorded
videos as well as helping and moderating the Q&A sessions and the
panel session. They will also be responsible for security and be ready
to react to potential threats and wrongdoers. Since we will broadcast
sessions in several time zones we need several hosts for each session.

[Session Hosts] https://icfp20.sigplan.org/home/ocaml-2020#session-hosts

◊ [Duties]

• Moderating the text chats
• Controlling microphones in the video-conferencing
• Watching for the time
• Performing sound checks
• Welcoming and otherwise guiding participants

[Duties] https://icfp20.sigplan.org/home/ocaml-2020#duties

[Transcribers / Interpreters]
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

We would like to have at least English transcriptions for each talk
and translations to other languages are very welcome. Transcriptions
enable accessibility as well as potentially increase the audience and
publicity as they could be indexed by the search engines.

[Transcribers / Interpreters]
https://icfp20.sigplan.org/home/ocaml-2020#transcribers-interpreters

◊ [Duties]

• Create transcriptions for videos, potentially in other languages.

[Duties] https://icfp20.sigplan.org/home/ocaml-2020#duties

[Content Manager]
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

The content manager will be responsible for maintaining the web
presence of the conference on [https://ocaml.org/]. We plan to have
all videos available, as well as maintain a page for each submitted
work.

[Content Manager]
https://icfp20.sigplan.org/home/ocaml-2020#content-manager

[Accessibility Chair]
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

We are striving to make the conference accessible to everyone and we
are looking for volunteers who have experience in online
accessibility.

[Accessibility Chair]
https://icfp20.sigplan.org/home/ocaml-2020#accessibility-chair

◊ [Duties]

• Helping with the selection of accessible platforms and tools.
• Working with attendees to ensure the necessary access services are
included.
• Establishing best practices for preparing and running accessible
sessions.

[Duties] https://icfp20.sigplan.org/home/ocaml-2020#duties

Introduction to Lwt
═══════════════════

Archive: [https://discuss.ocaml.org/t/introduction-to-lwt/5940/1]

Raphaël Proust announced
────────────────────────

I've published
[https://raphael-proust.github.io/code/lwt-part-1.html], a 2-part
introduction to Lwt.

The main aim of the introduction is to give a good mental model of
what promises are, how they behave and how to use them. It assumes
basic familiarity with OCaml.

Don't hesitate to ask questions or share feedback.

Other OCaml News
════════════════

From the ocamlcore planet blog
──────────────────────────────

Here are links from many OCaml blogs aggregated at [OCaml Planet].

• [Using ASCII waveforms to test hardware designs]

[OCaml Planet] http://ocaml.org/community/planet/

[Using ASCII waveforms to test hardware designs]
https://blog.janestreet.com/using-ascii-waveforms-to-test-hardware-designs/

Old CWN
═══════

If you happen to miss a CWN, you can [send me a message] and I'll mail
it to you, or go take a look at [the archive] or the [RSS feed of the
archives].

If you also wish to receive it every week by mail, you may subscribe
[online].

[Alan Schmitt]

[send me a message] mailto:alan.schmitt@polytechnique.org

[the archive] http://alan.petitepomme.net/cwn/

[RSS feed of the archives] http://alan.petitepomme.net/cwn/cwn.rss

[online] http://lists.idyll.org/listinfo/caml-news-weekly/

[Alan Schmitt] http://alan.petitepomme.net/

[-- Attachment #2: Type: text/html, Size: 46346 bytes --]