[Caml-list] Attn: Development Editor, Latest OCaml Weekly News

Alan Schmitt
2020-02-04T08:47:46Z

[-- Attachment #1: Type: text/plain, Size: 16468 bytes --]

Hello

Here is the latest OCaml Weekly News, for the week of January 28 to
February 04, 2020.

Table of Contents
─────────────────

Multicore OCaml: January 2020 update
Use Case for Ephemerons?
`json-data-encoding' version 0.8 (was `ocplib-json-typed')
Developer position at Abacus Medicine, Copenhagen
Camlp5 version 7.11 release (4.10 compatibility)
Old CWN

Multicore OCaml: January 2020 update
════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/multicore-ocaml-january-2020-update/5090/1>

Anil Madhavapeddy announced
───────────────────────────

Welcome to the January 2020 news update from the Multicore OCaml team!
We're going to summarise our activites monthly to highlight what we're
working on throughout this year. This update has kindly been assembled
by @shakthimaan and @kayceesrk.

The most common question we get is how to contribute to the overall
multicore effort. As I [noted last year], we are now in the process of
steadily upstreaming our efforts to mainline OCaml. Therefore, the
best way by far to contribute is to test for regressions or
opportunities for improvements in the patches that are outstanding in
the main OCaml repository.

A secondary benefit would be to review the PRs in the [multicore
repository], but those tend to be more difficult to evaluate
externally as they are being spotted as a result of stress testing at
the moment. A negative contribution would be to raise discussion of
orthogonal features or new project management mechanisms – this takes
time and effort to reply to, and the team has a very full plate
already now that the upstreaming has begun. We don't want to prevent
those discussions from happening of course, but would appreciate if
they were directed to the general OCaml bugtracker or another thread
on this forum.

We'll first go over the OCaml PRs and issues, then cover the multicore
repository and our Sandmark benchmarking infrastructure. A new
initiative to implement and test new parallel algorithms for Multicore
OCaml is also underway.

[noted last year]
<https://discuss.ocaml.org/t/multicore-prerequisite-patches-appearing-in-released-ocaml-compilers-now/4408>

[multicore repository]
<https://github.com/ocaml-multicore/ocaml-multicore/pulls>

OCaml
╌╌╌╌╌

◊ Ongoing

• [ocaml/ocaml#9082] Eventlog tracing system

Eventlog is a proposal for a new tracing facility for OCaml runtime
that provides metrics and counters, and uses the Binary Trace Format
(CTF). The next step to get this merged is to incubate the tracing
features in separate runtime variant, so it can be selected at
application link time.

• [ocaml/ocaml#8984] Towards a new closure representation

A new layout for closures has been proposed for traversal by the
garbage collector without the use of a page table. This is very much
useful for Multicore OCaml and for performance improvements. The PR
is awaiting review from other developers, and can then be rebased
against trunk for testing and merge.

• [ocaml-multicore/ocaml-multicore#187] Better Safe Points

A patch to regularly poll for inter-domain interrupts to provide
better safe points is actively being reviewed. This is to ensure
that any pending interrupts are notified by the runtime system.

• Work is underway on improving the marshaling (runtime/extern.c) in
upstream OCaml to avoid using GC mark bits to represent visitedness,
and to use a hash table (addrmap) implementation.

[ocaml/ocaml#9082] <https://github.com/ocaml/ocaml/pull/9082>

[ocaml/ocaml#8984] <https://github.com/ocaml/ocaml/pull/8984>

[ocaml-multicore/ocaml-multicore#187]
<https://github.com/ocaml-multicore/ocaml-multicore/issues/187>

◊ Completed

The following PRs have been merged to upstream OCaml trunk:

• [ocaml/ocaml#8713] Move C global variables to a dedicated structure

This PR moves the C global variables to a "domain state"
table. Every domain requires its own table of domain local
variables, and hence this is required for Multicore runtime.

This uncovered a number of [compatability issues] with the C header
files, which were all included in the recent OCaml 4.10.0+beta2
release via the next item.

• [ocaml/ocaml#9253] Move back `caml_*' to thematic headers

The `caml_*' definitions from runtime/caml/compatibility.h have been
moved to provide a compatible API for OCaml versions 4.04 to
4.10. This change is also useful for Multicore domains that have
their own state.

[ocaml/ocaml#8713] <https://github.com/ocaml/ocaml/pull/8713>

[compatability issues] <https://github.com/ocaml/ocaml/issues/9205>

[ocaml/ocaml#9253] <https://github.com/ocaml/ocaml/pull/9253>

Multicore OCaml
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

The following PRs have been merged into the Multicore OCaml trees:

• [ocaml-multicore/ocaml-multicore#275] Fix lazy behaviour for
Multicore

A `caml_obj_forward_lazy()' function is implemented to handle lazy
values in Multicore Ocaml.

• [ocaml-multicore/ocaml-multicore#269] Move from a global
`pools_to_rescan' to a domain-local one

During stress testing, a segmentation fault occurred when a pool was
being rescanned while a domain was allocating in to it. The rescan
has now been moved to the domain local, and hence this situation
will not occur again.

• [ocaml-multicore/ocaml-multicore#268] Fix for a few space leaks

The space leaks that occurred during domain spawning and termination
when performing the stress tests have been fixed in this PR.

• [ocaml-multicore/ocaml-multicore#272] Fix for DWARF CFI for
non-allocating external calls

The entry to `caml_classify_float_unboxed' caused a corrupted
backtrace, and a fix that clearly specifies the boundary between
OCaml and C has been provided.

• An effort to implement a synchronized minor garbage collector for
Multicore OCaml is actively being researched and worked
upon. Benchmarking for a work-sharing parallel stop-the-world branch
against multicore trunk has been performed along with clearing
technical debt, handling race conditions, and fixing segmentation
faults. The C-API reversion changes have been tested and merged into
the stop-the-world minor GC branch for Multicore OCaml.

[ocaml-multicore/ocaml-multicore#275]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/275>

[ocaml-multicore/ocaml-multicore#269]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/269>

[ocaml-multicore/ocaml-multicore#268]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/268>

[ocaml-multicore/ocaml-multicore#272]
<https://github.com/ocaml-multicore/ocaml-multicore/pull/272>

Benchmarking
╌╌╌╌╌╌╌╌╌╌╌╌

• The [Sandmark] performance benchmarking infrastructure has been
improved for backfilling data, tracking branches and naming
benchmarks.

• Numerical parallel benchmarks have been added to the Multicore
compiler.

• An [Irmin] macro benchmark has been included in Sandmark. A test for
measuring Irmin's merge capabilities with Git as its filesystem is
being tested with different read and write rates.

• Work is also underway to implement parallel algorithms for N-body,
reverse-complement, k-nucleotide, binary-trees, fasta,
fannkuch-redux, regex-redux, Game of Life, RayTracing, Barnes Hut,
Count Graphs, SSSP and from the MultiMLton benchmarks to test on
Multicore OCaml.

[Sandmark] <http://bench2.ocamllabs.io/>

[Irmin] <https://irmin.org>

Documentation
╌╌╌╌╌╌╌╌╌╌╌╌╌

• A chapter on Parallel Programming in Multicore OCaml is being
written and an early draft will be made available to the community
for their feedback. It is based on Domains, with examples to
implement array sums, Pi approximation, and trapezoidal rules for
definite integrals.

Acronyms
╌╌╌╌╌╌╌╌

• API: Application Programming Interface
• CTF: Common Trace Format
• CFI: Call Frame Information
• DWARF: Debugging With Attributed Record Formats
• GC: Garbage Collector
• PR: Pull Request
• SSSP: Single Source Shortest Path

Nicolas Tollenaere asked
────────────────────────

If I may ask a question, I am curious about the status of integration
of effects into the type system. According to this page
<https://ocamlverse.github.io/content/future_ocaml.html#typed-algebraic-effects>,
original plan was to merge an untyped version of effect, before it was
decided to integrate them into the system. I have seen this
presentation of leo white on this matter
<https://www.janestreet.com/tech-talks/effective-programming/> along
with this one <https://www.youtube.com/watch?v=ibpUJmlEWi4> (from
2016). My understanding was that, at the time of the last
presentation, there was still some theoretical issues to be solved
(although the speaker did not seem too worried about finding some way
around eventually). I have no idea about the current status of the
project. Reading your post it seems that you are now in an integration
phase (PR reviews and all) that would imply that you're done with
(most) theoretical questions. But that could either mean that you are
integrating an untyped version of effects (and the type system is let
for future development) or that you have indeed settled on a
design. Which one is it ? Anyway, thanks for the post and the work in
general, this project seems awesome (even if I did not dive into it
too much until now)

Anil Madhavapeddy replied
─────────────────────────

Good question; our current focus in getting the runtime components
upstreamed (the "Domains" API) and some of the mechanisms that could
be used by an effect system. We haven't yet settled on a final design
for an effect extension to OCaml, but the general preference is to
skip integrating an untyped effect system if a typed version lands in
the right timescales. This will happen after all the runtime pieces
are upstreamed, which will allow everyone to use multicore parallelism
via the lower-level Domains API.

Use Case for Ephemerons?
════════════════════════

Archive: <https://discuss.ocaml.org/t/use-case-for-ephemerons/2838/3>

Continuing this old thread, Yawar Amin said
───────────────────────────────────────────

[Here's another use] (disclaimer: this is my project).

What's happening here is that I'm using an 'ephemeral cache' (i.e. a
cache backed by an ephemeron hash table, [here]) to store subscribers
to a 'topic', i.e. a pub-sub bus. You get a subscription token when
you subscribe to a topic, and part of that token is the cache key. The
cache is 'ephemeral' so as soon as the subscription token goes out of
scope, it and its corresponding subscription (concretely, the stream
and its push function) are automatically deleted from the cache.

Hence, there's no 'unsubscribe' or 'close topic' functionality–it's
assumed that you want to unsubscribe if you let the subscription token
go out of scope.

[Here's another use]
<https://github.com/yawaramin/re-web/blob/766da0c0e06652824e34416bc518ee37197a90fb/ReWeb/Topic.ml>

[here]
<https://github.com/yawaramin/re-web/blob/766da0c0e06652824e34416bc518ee37197a90fb/ReWeb/Cache.ml#L41>

`json-data-encoding' version 0.8 (was `ocplib-json-typed')
══════════════════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/ann-json-data-encoding-version-0-8-was-ocplib-json-typed/5095/1>

Raphaël Proust announced
────────────────────────

I'm happy to announce that Nomadic Labs is now in charge of the
development, maintenance and release of `json-data-encoding' – the
library previously known as `ocplib-json-typed'. Even though we are
changing to a more descriptive name, we are maintaining continuity of
version numbers. As a result, this is an announce for the version
`0.8'.

The library `json-data-encoding' lets you define encodings for a given
OCaml type, and use that encoding to encode values of that type into
JSON or decode JSON into values of that type. The library supports
multiple JSON backends: `Ezjsonm', `Yojson', native browser
representation (for `js_of_ocaml', via the package
`json-data-encoding-browser') and `BSON' (via the package
`json-data-encoding-bson').

It is available via `opam' (`opam install json-data-encoding') and
hosted on <https://gitlab.com/nomadic-labs/json-data-encoding/>

Changes from the version v0.7 include:
• extensive tests using `Crowbar' (adapted from similar tests on
`data-encoding' originally by @gasche)
• minor documentation improvements
• improved self documentation capabilities for unions' cases (work by
@smondet)
• improved schema equality (work by @rbardou)

Developer position at Abacus Medicine, Copenhagen
═════════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/developer-position-at-abacus-medicine-copenhagen/5119/1>

mokn announced
──────────────

Abacus Medicine has an open developer position. We do parallel
distribution of medicine in EU and for that we have developed a system
to handle the trading. A part of this system is developed in OCaml.

Unfortunately the job description is only in danish, but we do accept
applications in english: [Job description]

[Job description]
<https://www.jobindex.dk/jobannonce/351439/software-developer>

Camlp5 version 7.11 release (4.10 compatibility)
════════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/ann-camlp5-version-7-11-release-4-10-compatibility/5121/1>

Chet Murthy announced
─────────────────────

New release 7.11 of Camlp5. Compatible with all OCaml versions >=
4.00.0, latest OCaml version 4.10+beta2 included.

Main improvement: compatible with 4.10's blank module names and
generative functors.

Home page, including downloading and documentation at:
<https://camlp5.github.io/>

Enjoy!

N.B. I'm new to helping out with camlp5, so might have made some
mistakes; any users who find problems should contact me either
directly, or (better) thru issues on
<https://github.com/camlp5/camlp5/releases> and I'll be sure to get
right on it.

N.B.#2: There are still lots of gaps between current Ocaml, and
Camlp5's support; I'm working on fixing that, and there'll soon be a
release that brings camlp5 as up-to-date as possible with Ocaml.

Old CWN
═══════

If you happen to miss a CWN, you can [send me a message] and I'll mail
it to you, or go take a look at [the archive] or the [RSS feed of the
archives].

If you also wish to receive it every week by mail, you may subscribe
[online].

[Alan Schmitt]

[send me a message] <mailto:alan.schmitt@polytechnique.org>

[the archive] <http://alan.petitepomme.net/cwn/>

[RSS feed of the archives] <http://alan.petitepomme.net/cwn/cwn.rss>

[online] <http://lists.idyll.org/listinfo/caml-news-weekly/>

[Alan Schmitt] <http://alan.petitepomme.net/>

[-- Attachment #2: Type: text/html, Size: 29138 bytes --]