[Caml-list] Attn: Development Editor, Latest OCaml Weekly News

Alan Schmitt
2021-05-11T14:48:02Z

[-- Attachment #1: Type: text/plain, Size: 27361 bytes --]

Hello

Here is the latest OCaml Weekly News, for the week of May 04 to 11,
2021.

Table of Contents
─────────────────

Software engineer position at LexiFi (Paris)
Open source editor for iOS, iPadOS and macOS
Backend developer position at Issuu (Copenhagen)
25 years of OCaml
OCaml compiler development newsletter, issue 1: before May 2021
After so many years, I discover 'Str.bounded_full_split regexp str n'
Parser for the Scala programming language?
Old CWN

Software engineer position at LexiFi (Paris)
════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/job-software-engineer-position-at-lexifi-paris/7782/1>

Alain Frisch announced
──────────────────────

[LexiFi] is hiring! We are looking for a fully-time software engineer
to join our core development team. The vast majority of our stack is
implemented in OCaml, and we have plenty of exciting projects on a
wide range of topics.

More info on <https://www.lexifi.com/careers/software_engineer/>

[LexiFi] <https://www.lexifi.com>

Open source editor for iOS, iPadOS and macOS
════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/open-source-editor-for-ios-ipados-and-macos/7624/15>

Continuing this thread, Nathan Fallet announced
───────────────────────────────────────────────

Just updated the editor, I redesigned the macOS version, and it just
looks better and more native

<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/optimized/2X/6/6b03c462755fb37a2d5018013c3d1c8bd45f53bf_2_1380x766.jpeg>

What are your first impressions on it?

Backend developer position at Issuu (Copenhagen)
════════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/job-backend-developer-position-at-issuu-copenhagen/7793/1>

Dario Teixeira announced
────────────────────────

We are looking for a Backend Developer with experience in machine
learning – and preferably also OCaml! – to join our Research &
Development team. You will help build machine learning research
prototypes and be responsible for integrating them into new and
existing products.

At Issuu, we use OCaml extensively in our production systems. If you
love OCaml and functional programming in general, Issuu is a great
place to put your passion into real-world products!

Please find more information about this position at the following
link:
<https://jobs.lever.co/issuu/f502cb20-b216-4c67-8357-d748e1b35178>

Anentropic asked and Dario Teixeira replied
───────────────────────────────────────────

I would love to hear more about your OCaml backend stack

Well, we love to talk about our OCaml stack! :slightly_smiling_face:

We rely on the Jane Street ecosystem a lot, using Core as a Stdlib
replacement and Async for monadic concurrency.

AMQP forms the backbone of our messaging system, and therefore we use
[amqp-client] extensively.

We use both MySQL and Postgresql databases in production. For the
former we use [ppx_mysql], and for the latter, [PGOCaml]. (Thanks to
Docker, we can give PGOCaml compile-time access to the DB without
having to depend on the actual production DB.)

We currently use Protobuf for serialisation, but spend a great amount
of time complaining about it. We rely on [ocaml-protoc-plugin] to
generate the OCaml code from Protobuf definitions.

Anyway, that's just the basics of our stack. Do let me know if there's
something else you'd like to know in more detail!

[amqp-client] <https://github.com/andersfugmann/amqp-client>

[ppx_mysql] <https://github.com/issuu/ppx_mysql>

[PGOCaml] <https://github.com/darioteixeira/pgocaml>

[ocaml-protoc-plugin] <https://github.com/issuu/ocaml-protoc-plugin>

roddy asked and Dario Teixeira replied
──────────────────────────────────────

Do you use Protobuf for interop with non-OCaml systems? If
not, I'm curious about whether you've considered
[bin_prot] as an alternative; it seems like an obvious
choice if you're using Core/Async.

Yes, we use Protobuf mainly because we have a heterogeneous stack,
where besides OCaml we also have services running Python, Kotlin, or
Elixir.

[bin_prot]
<https://github.com/janestreet/bin_prot/blob/master/README.md>

Tim McGilchrist asked and Dario Teixeira
────────────────────────────────────────

I'm curious about how you structure the business code (for
want of a better word), in between the technical layers of
talking to AMQP or an SQL store. Are there larger scale
patterns like CQRS or DDD that you use to organise code?

How do you package up code for deployment? Docker / AWS something.
We're slowly migrating to a micro-service architecture (the pros and
cons of which are outside the scope of this thread; that's a can of
worms I'd rather not open…) whose cast of characters includes
"entities" (responsible for storing/retrieving data from DBs), generic
backend services that encapsulate business logic, frontend services,
and backend-for-frontend services.

We're using Docker for deployment on AWS (mostly), and slowly
migrating from Docker Swarm to Kubernetes.

25 years of OCaml
═════════════════

Archive: <https://discuss.ocaml.org/t/25-years-of-ocaml/7813/1>

Xavier Leroy announced
──────────────────────

25 years ago, on May 9th 1996, release 1.00 of the Objective Caml
language and system was announced:
<https://sympa.inria.fr/sympa/arc/caml-list/1996-05/msg00003.html>

It was already the consolidation of many years of work, integrating
Jérôme Vouillon and Didier Rémy's work on objects and classes within
Caml Special Light, itself a combination of my work on modules and
native-code compilation with earlier code taken from Caml Light,
especially Damien Doligez's GC.

Little did I know that O(bjective) Caml would still be there 25 years
later!

A lot happened during this time, including several major evolutions of
the language, and, much more importantly, the emergence of a community
of users and an ecosystem of tools and libraries. But maybe this was
just the beginning for something even bigger? We'll see…

Happy birthday, OCaml!

David Allsopp replied
─────────────────────

Most pleasingly, with a [very small number of patches], the Windows
port still works in Visual Studio 2019:

┌────
│ C:\Birthday>ocaml.exe
│ Objective Caml version 1.00
│
│ #print_endline "Happy 25th Birthday, OCaml!";;
│ Happy 25th Birthday, OCaml!
│ - : unit = ()
│ ##quit;;
│
│ C:\Birthday>type hooray.ml
│ let rec hip_hip n =
│ if n > 0 then
│ let () = print_endline "hip hip! hooray!" in
│ hip_hip (pred n)
│
│ let () = hip_hip 25
│ C:\Birthday>ocamlopt -o hooray.exe hooray.ml
│
│ C:\Birthday>hooray
│ hip hip! hooray!
│ ...
└────

[very small number of patches]
<https://github.com/dra27/ocaml/commits/25-years-of-ocaml>

On the OCaml Maling List, Roberto Di Cosmo also replied
───────────────────────────────────────────────────────

Long live OCaml!

Thanks Xavier, and to all the brilliant minds that contributed to the
evolution and adoption of this beautiful language, and system, in this
past quarter of a century.

If I may add a personal note, one truly remarkable fact is that some
rather complex code written in 1998 using OCaml 1.07 [1] could be
compiled and run last year using OCaml 4.x *without modifications*:
the only visible changes were the new warnings spotting potential
issues in the code, thanks to the many improvements to the compiler
over time.

For the curious, all the details are here:
<https://www.dicosmo.org/Articles/2020-ReScienceC.pdf>

Cheers

Roberto

[1] that was the first version including support for marshalling
closures, added in a fantastic one week-spring in Pisa exactly for
this code :-)

OCaml compiler development newsletter, issue 1: before May 2021
═══════════════════════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/ocaml-compiler-development-newsletter-issue-1-before-may-2021/7831/1>

gasche announced
────────────────

I'm happy to introduce the first issue of the "OCaml compiler
development newsletter". I asked frequent contributors to the OCaml
compiler codebase to write a small burb on what they have been doing
recently, in the interest of sharing more information on what people
are interested in, looking at and working on.

This is by no means exhaustive: many people didn't end up having the
time to write something, and it's fine. But hopefully this can give a
small window on development activity related to the OCaml compiler,
structured differently from the endless stream of [Pull Requests] on
the compiler codebase.

(This initiative is inspired by the excellent Multicore
newsletter. Please don't expect that it will be as polished or
consistent :yo-yo: .)

Note:

• Feel free of course to comment or ask questions, but I don't know if
the people who wrote a small blurb will be looking at the thread, so
no promises.

• If you have been working on the OCaml compiler and want to say
something, please feel free to post! If you would like me to get in
touch next time I prepare a newsletter issue (some random point in
the future), please let me know by email at (gabriel.scherer at
gmail).

[Pull Requests] <https://github.com/ocaml/ocaml/pulls>

@dra27 (David Allsopp)
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

Compiler relocation patches now exist. There's still a few left to
write, and they need splitting into reviewable PRs, but the core
features are working. A compiler installation can be copied to a new
location and still work, meaning that local switches in opam may in
theory be renamed and, more importantly, we can cache previously-built
compilers in an opam root to allow a new switch's compiler to be a
copy. This probably won't be reviewed in time for 4.13, although it's
intended that once merged opam-repository will carry back-ports to
earlier compilers.

A whole slew of scripting pain has lead to some possible patches to
reduce the use of scripts in the compiler build to somewhat closer to
none.

FlexDLL bootstrap has been completely overhauled, reducing build time
considerably. This will be in 4.13 (#[10135])

[10135] <https://github.com/ocaml/ocaml/pull/10135>

@nojb (Nicolás Ojeda Bär)
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

I am working on #[10159], which enables debug information in
`-output-complete-exe' binaries. It uses [incbin] under Unix-like
system and some other method under Windows.

[10159] <https://github.com/ocaml/ocaml/pull/10159>

[incbin] <https://github.com/graphitemaster/incbin>

@gasche (Gabriel Scherer)
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

I worked on bringing more PRs to a decision (merge or close). The
number of open PRs has gone from 220-ish to 180, which feels nice.

I have also contributed to @Ekdohibs' project [camlboot], which is a
"bootstrap-free" implementation of OCaml able to compile the OCaml
compiler itself. It currently targets OCaml 4.07 for various
reasons. We were able to do a full build of the OCaml compiler, and
check that the result produces bootstrap binaries that coincide with
upstream bootstraps. This gives extremely strong confidence that the
OCaml bootstrap is free from "trusting trust" attacks. For more
details, see our [draft paper].

[camlboot] <https://github.com/Ekdohibs/camlboot>

[draft paper] <http://gallium.inria.fr/~scherer/drafts/camlboot.pdf>

with @Octachron (Florian Angeletti)
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄

I worked with Florian Angeletti on deprecating certain command-line
warning-specifier sequences, to avoid usability issues with (new in
4.12) warning names. Before `-w -partial-match' disables warning 4,
but `-w -partial' is interpreted as the sequence `w -p -w a -w r -w t
-w i -w a -w l', most of which are ignored but `-w a' silences all
warnings. Now multi-letter sequences of "unsigned" specifiers (`-p' is
signed, `a' is unsigned) are deprecated. (We first deprecated all
unsigned specifiers, but Leo White tested the result and remarked that
`-w A' is common, so now we only warn on multi-letter sequences of
unsigned specifiers.

I am working with @Octachron (Florian Angeletti) on grouping signature
items when traversing module signatures. Some items are "ghost items"
that are morally attached in a "main item"; the code mostly ignores
this and this creates various bugs in corner cases. This is work that
Florian started in September 2019 with #[8929], to fix a bug in the
reprinting of signatures. I only started reviewing in May-September
2020 and we decided to do sizeable changes, he split it in several
smaller changes in January 2021 and we merged it in April 2021. Now we
are looking are fixing other bugs with his code (#[9774],
#[10385]). Just this week Florian landed a nice PR fixing several
distinct issues related to signature item grouping: #[10401].

[8929] <https://github.com/ocaml/ocaml/pull/8929>

[9774] <https://github.com/ocaml/ocaml/pull/9774>

[10385] <https://github.com/ocaml/ocaml/pull/10385>

[10401] <https://github.com/ocaml/ocaml/pull/10401>

@xavierleroy (Xavier Leroy)
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

I fixed #[10339], a mysterious crash on the new Macs with "Apple
silicon". This was due to a ARM (32 and 64 bits)-specific
optimization of array bound checking, which was not taken into account
by the platform-independent parts of the back-end, leading to
incorrect liveness analysis and wrong register allocation. #[10354]
fixes this by informing the platform-independent parts of the back-end
that some platform-specific instructions can raise. In passing, it
refactors similar code that was duplicating platform-independent
calculations (of which instructions are pure) in platform-dependent
files.

I spent quality time with the Jenkins continuous integration system at
Inria, integrating a new Mac Mini M1. For unknown reasons, Jenkins
ran the CI script in x86-64 emulation mode, so we were building and
testing an x86-64 version of OCaml instead of the intended ARM64
version. A bit of scripting later (8b1bc01c3) and voilà, arm64-macos
is properly tested as part of our CI.

Currently, I'm reading the "safe points" proposal by Sadiq Jaffer
(#[10039]) and the changes on top of this proposed by Damien Doligez.
It's a necessary step towards Multicore OCaml, so we really need to
move forward on this one. It's a nontrivial change involving a new
static analysis and a number of tweaks in every code emitter, but
things are starting to look good here.

[10339] <https://github.com/ocaml/ocaml/pull/10339>

[10354] <https://github.com/ocaml/ocaml/pull/10354>

[10039] <https://github.com/ocaml/ocaml/pull/10039>

@mshinwell (Mark Shinwell)
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

I did a first pass of review on the safe points PR (#[10039]) and
significantly simplified the proposed backend changes. I've also been
involved in discussions about a new function-level attribute to cause
an error if safe points (including allocations) might exist within a
function's body, to make code that currently assumes this robust.
There will be a design document for this coming in due course.

I fixed the random segfaults that were occurring on the RISC-V Inria
CI worker (#[10349]).

In Flambda 2 land we spent two person-days debugging a problem
relating to Infix_tag! We discovered that the code in OCaml 4.12
onwards for traversing GC roots in static data ("caml_globals") is not
correct if any of the roots are closures. This arises in part because
the new compaction code (#[9728]) has a hidden invariant: it must not
see any field of a static data root more than once (not even via an
Infix_tag). As far as we know, these situations do not arise in the
existing compiler, although we may propose a patch to guard against
them. They arise with Flambda 2 because in order to compile
statically-allocated inconstant closures (ones whose environment is
partially or wholly computed at runtime) we register closures directly
as global roots, so we can patch their environments later.

[10039] <https://github.com/ocaml/ocaml/pull/10039>

[10349] <https://github.com/ocaml/ocaml/pull/10349>

[9728] <https://github.com/ocaml/ocaml/pull/9728>

@garrigue (Jacques Garrigue)
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

I have been working on a number of PRs fixing bugs in the type system,
which are now merged:
• #[10277] fixes a theoretical bug in the principality of GADT type
inference (#[10383] applies only in -principal mode)
• #[10308] fixes an interaction between local open in patterns and the
new syntax for introducing existential type variables
• #[10322] is an internal change using a normal reference inside of a
weak one for backtracking; the weak reference was an optimization
when backtracking was a seldom used feature, and was not useful
anymore
• #[10344] fixes a bug in the delaying of the evaluation of optional
arguments
• #[10347] cleans up some code in the unification algorithm, after a
strengthening of universal variable scoping
• #[10362] fixes a forgotten normalization in the type checking
algorithm

Some are still in progress:
• #[10348] improves the way expansion is done during unification, to
avoid some spurious GADT related ambiguity errors
• #[10364] changes the typing of the body of the cases of
pattern-matchings, allowing to warn in some non-principal
situations; it also uncovered a number of principality related bugs
inside the the type-checker

Finally, I have worked with Takafumi Saikawa (@t6s) on making the
representation of types closer to its logical meaning, by ensuring
that one always manipulate a normalized view in #[10337] (large
change, evaluation in progress).

[10277] <https://github.com/ocaml/ocaml/pull/10277>

[10383] <https://github.com/ocaml/ocaml/pull/10383>

[10308] <https://github.com/ocaml/ocaml/pull/10308>

[10322] <https://github.com/ocaml/ocaml/pull/10322>

[10344] <https://github.com/ocaml/ocaml/pull/10344>

[10347] <https://github.com/ocaml/ocaml/pull/10347>

[10362] <https://github.com/ocaml/ocaml/pull/10362>

[10348] <https://github.com/ocaml/ocaml/pull/10348>

[10364] <https://github.com/ocaml/ocaml/pull/10364>

[10337] <https://github.com/ocaml/ocaml/pull/10337>

@let-def (Frédéric Bour)
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

For some time, I have been working on new approaches to generate error
messages from a Menhir parser.

My goal at the beginning was to detect and produce a precise message
for the ‘let ;’ situation:
┌────
│ let x = 5;
│ let y = 6
│ let z = 7
└────
LR detects an error at the third ‘let’ which is technically correct,
although we would like to point the user at the ‘;’ which might be the
root cause of the error. This goal has been achieved, but the
prototype is far from being ready for production.

The main idea to increase the expressiveness and maintainability of
error context identification is to use a flavor of regular
expressions. The stack of a parser defines a prefix of a sentential
form. Our regular expressions are matched against it. Internal details
of the automaton does not leak (no reference to states), the regular
language is defined by the grammar alone. With appropriate tooling,
specific situations can be captured by starting from a coarse
expression and refining it to narrow down the interesting cases.

Now I am focusing on one specific point of the ‘error message’
development pipeline: improving the efficiency of ‘menhir
–list-errors’. This command is used to enumerate sentences that cover
all erroneous situations (as defined by the LR grammar). On my
computer and with the OCaml grammar, it takes a few minutes and quite
a lot of RAM. Early results are encouraging and I hope to have a PR
for Menhir soon. The performance improvement we are aiming for is to
make the command almost real time for common grammars and to tackle
bigger grammars by reducing the memory needs. For instance, in the
OCaml case, the runtime is down from 3 minutes to 2–3 seconds and
memory consumption goes from a few GiB down to 200 MiB.

Daniel Bünzli asked and gasche replied
──────────────────────────────────────

> […] @Ekdohibs’ project [camlboot ], which is a
“bootstrap-free”
> implementation of OCaml able to compile the OCaml
compiler itself. It currently targets OCaml 4.07 for
various
> reasons. We were able to do a full build of the OCaml
compiler, and check that the result produces bootstrap
> binaries that coincide with upstream bootstraps. This
gives extremely strong confidence that the OCaml
bootstrap is
> free from “trusting trust” attacks. For more details,
see our [draft paper].

Something that is not clear to me (but I read quickly) is
the impact of `guile` itself being not bootstrapped yet.
Could there be a *very* elaborate attack (with probability
0 of existing) on both the guile and ocaml bootstrap or is
there something in the whole scheme that prevents it ?

Yes, currently Guile needs to be trusted, and it would be possible
that a bootstrapping virus in Guile would break our correctness
result. (It would need to reproduce itself through our compiler and
interpreter that were written after Guile itself, but I think in
theory this could be done with an almost-infinitely clever program
analysis.) Of course, an attack at the source level (inserting
malicious source, instead of malicious binaries) is also possible
anywhere in the chain. Our main reason for using Guile is that this
is the high-level language community most active on
debootstrapping-towards-the-metal (through the Guix connection), so we
believe it is more likely to manage debootstrapping and maintain it in
the longer run.

(The seed that Guile depends on is its macro-expander, which is
written using macros itself. In theory one may perform the
macro-expansion of the expander, and then manually review the two
versions to verify the absence of attack there.)

[camlboot ] <https://github.com/Ekdohibs/camlboot>

[draft paper] <http://gallium.inria.fr/~scherer/drafts/camlboot.pdf>

After so many years, I discover 'Str.bounded_full_split regexp str n'
═════════════════════════════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/after-so-many-years-i-discover-str-bounded-full-split-regexp-str-n/7838/1>

UnixJunkie said
───────────────

This is so useful and powerful:
┌────
│ #require "str";;
│ Str.bounded_full_split (Str.regexp "[()]") "toto (titi, tata (et tutu)) vont au parc (en courant)" 1024;;
│ - : Str.split_result list =
│ [Str.Text "toto "; Str.Delim "("; Str.Text "titi, tata "; Str.Delim "(";
│ Str.Text "et tutu"; Str.Delim ")"; Str.Delim ")"; Str.Text " vont au parc ";
│ Str.Delim "("; Str.Text "en courant"; Str.Delim ")"]
└────

Still finding hidden pearls in the stdlib after so many years!
:slight_smile:

Parser for the Scala programming language?
══════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/parser-for-the-scala-programming-language/7541/18>

Deep in this thread, Yoann Padioleau announced
──────────────────────────────────────────────

I ended up porting the recursive descent parser in the Scala compiler
to OCaml … I think it was the fastest way to get a working parser
from OCaml …

<https://github.com/returntocorp/pfff/blob/develop/lang_scala/parsing/Parser_scala_recursive_descent.ml>

Old CWN
═══════

If you happen to miss a CWN, you can [send me a message] and I'll mail
it to you, or go take a look at [the archive] or the [RSS feed of the
archives].

If you also wish to receive it every week by mail, you may subscribe
[online].

[Alan Schmitt]

[send me a message] <mailto:alan.schmitt@polytechnique.org>

[the archive] <https://alan.petitepomme.net/cwn/>

[RSS feed of the archives] <https://alan.petitepomme.net/cwn/cwn.rss>

[online] <http://lists.idyll.org/listinfo/caml-news-weekly/>

[Alan Schmitt] <https://alan.petitepomme.net/>

[-- Attachment #2: Type: text/html, Size: 43022 bytes --]