[Caml-list] Attn: Development Editor, Latest OCaml Weekly News

Alan Schmitt
2022-09-27T07:17:24Z

[-- Attachment #1: Type: text/plain, Size: 29726 bytes --]

Hello

Here is the latest OCaml Weekly News, for the week of September 20 to
27, 2022.

Table of Contents
─────────────────

Esperanto, when OCaml meets Cosmopolitan
OBazl Toolsuite - tools for building OCaml with Bazel
Orsetto: structured data interchange languages (release 1.1.2)
Interest in a Http_sig library?
Outreachy summer ’22 closing commemoration session on 23rd Sept
findlib-1.9.6
Interesting OCaml Articles
Other OCaml News
Old CWN

Esperanto, when OCaml meets Cosmopolitan
════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/ann-esperanto-when-ocaml-meets-cosmopolitan/10501/1>

Calascibetta Romain announced
─────────────────────────────

I am delighted to present the first *experimental* release of
Esperanto. This project is a new OCaml _toolchain_ that creates
binaries compiled with the [Cosmopolitan C library] and linked with
the [αcτµαlly pδrταblε εxεcµταblε] link script. The binary produced is
then portable to different platforms:

<https://worker.jart.workers.dev/redbean/linux.png>
<https://worker.jart.workers.dev/redbean/windows10.png>
<https://worker.jart.workers.dev/redbean/msdos60.png>
<https://worker.jart.workers.dev/redbean/macos.png>
<https://worker.jart.workers.dev/redbean/freebsd64.png>
<https://worker.jart.workers.dev/redbean/openbsd.png>
<https://worker.jart.workers.dev/redbean/netbsd2.png>

The main objective of Esperanto is to provide a toolchain capable of
producing a portable binary from an existing project. This would allow
to finally be able to distribute software for all these platforms
without having to:
1) manage multiple platforms orthogonally, the Cosmopolitan C library
offers you the POSIX API for all platforms (including Windows)
2) Produce several versions of the same software for each platform.
Only the binary is needed to run on all platforms

Cosmopolitan *does not* however produce a binary with a multi-platform
assembler. At this stage, our distribution only supports the `x86_64'
assembler (the most common one) but we are working on the possibility
to produce a binary with different assemblers.

I would like to give special thanks to Justine, the author of the
Cosmopolitan project (to develop [redbean], a small portable HTTP
server) for her excellent work.

[Cosmopolitan C library] <https://justine.lol/cosmopolitan/>

[αcτµαlly pδrταblε εxεcµταblε] <https://justine.lol/ape.html>

[redbean] <https://redbean.dev/>

A _toolchain_
╌╌╌╌╌╌╌╌╌╌╌╌╌

In OCaml, the “toolchain” principle allows the existence of several
compilers within an OPAM switch and to choose one of them when it
comes to cross-compiling a project. This principle, even though it is
not clearly defined and even though its use remains very limited,
exists through the `ocamlfind` tool.

You can find these toolchains in your switch:
┌────
│ $ ls $(opam var lib)/findlib.conf.d/
│ esperanto.conf solo5.conf
└────

From our experience with Mirage as well as the work done in `dune'
regarding cross-compilation, the choice to propose a new _toolchain_
in order to allow cross-compilation of projects with OPAM is both a
historical choice but also the most relevant one in our opinion [1].

◊ Why we need to cross-compile?

The term cross-compilation can be misunderstood if we only consider
the question of the assembler issued by the compiler (does it match
the host assembler or not). In our case, cross-compilation is a
broader term that implies the use of external artefacts to the
compiler that are different from the default and the use of compiler
options that must be used throughout the production of the final
binary.

In other words, even though we are emitting the same assembler, we are
doing so in a different “context” which requires the definition of a
new _toolchain_ which includes our artefacts and compiler options.

One of these artefacts is of course the C library used by the compiler
which will then be systematically used by the _runtime caml_, the well
named `libasmrun.a'. This is why, for example, there is a version of
OCaml with [musl]. So there must be a version of OCaml with
Cosmopolitan.

This new _toolchain_ also allows you to include the necessary options
for compiling C files because, yes, you can compile a C file with
`ocamlopt'.

In order to provide a coherent _workflow_ for a project, we need to
provide not only a `libasmrun.a' compiled with our Cosmopolitan C
library but also an OCaml compiler capable of invoking the C compiler
with the right options required by Cosmopolitan.

Finally, we also need to describe in this _toolchain_ how to link the
object files together to actually produce a portable binary using the
APE script.

[musl] <https://musl.libc.org/>

◊ A simple example with this new _toolchain_

Installing Esperanto is very easy with OPAM. It will install the
cross-compiler and the necessary files so that `ocamlfind~/~dune' can
recognise this new _toolchain_:
┌────
│ $ opam install esperanto
└────

Finally, let’s try to produce a simple binary that displays “Hello
World!”:
┌────
│ $ cat >main.ml <<EOF
│ let () = print_endline "Hello World!"
│ EOF
│ $ ocamlfind -toolchain opt main.ml
│ $ objcopy -S -O binary a.out
│ $ file a.out
│ a.out: DOS/MBR boot sector
└────

The binary produced can already be executed. However, there are still
some issues that have been fixed since then but which are probably not
yet integrated in your system. They concern `zsh' and `binfmt_misc' in
particular.

The first problem with `zsh' is that it does not recognise the binary
correctly. This problem has been fixed in the latest version of
`zsh.5.9.0'.
┌────
│ $ zsh --version
│ zsh 5.8.1
│ $ zsh
│ $ ./a.out
│ zsh: exec format error: ./a.out
└────

The second problem concerns `binfmt_misc' which intervenes upstream at
the execution of your programs in order to choose how to execute them.
In this case, `binfmt_misc' recognises Cosmopolitan binaries as
Windows software by default.

Here too, a solution is available and described by the author of
Cosmopolitan here: [APE loader]

[APE loader] <https://justine.lol/apeloader/#binfmt_misc>

◊ Execution & Assimilation

If you are not concerned by the above problems, you can simply run
the program:
┌────
│ $ ./a.out
│ Hello World!
└────

There is a final solution that requires a little explanation of what
αcτµαlly pδrταblε εxεcµταblε is. Indeed, the latter makes it
possible to create a polyglot binary whose first point of entry is
not your program but a small program which tries to recognize on
which platform the binary tries to run.

After this recognition, this little program will “inject” values
corresponding to the platform in which you try to run your program
in order to finally let Cosmopolitan manage the translation between
its interface and the real POSIX interface that your system offers.

Of course, this step has a cost as it adds an indirection between
what your program wants to do and what is available on the system
running your program. However, APE offers a very special option that
allows the program to be assimilated to the platform in which it
wants to run.
┌────
│ $ file a.out
│ a.out: DOS/MBR boot sector
│ $ sh -c "./a.out --assimilate"
│ $ file a.out
│ a.out: ELF 64-bit LSB executable, x86-64
│ $ ./a.out
│ Hello World!
└────

This option makes your application truly native to the platform in
which you run it. This means above all that the program is *no
longer* portable.

◊ Esperanto, `dune' & `opam monorepo'

The `dune' software also incorporates this toolchain idea using the
`-x' option. More pragmatically, it is possible to define a new dune
context to use Esperanto as a compilation toolchain.

However, the original aim of Esperanto is to produce a portable
binary. This implies, among other things, that it should not depend
on remaining artefacts in order to run and, in this sense, the
compilation of your project should be a static compilation. This
means that all dependencies of your project must be available to
compile in the same context as your project.

Again, this is particularly necessary if any of your dependencies
include C files, so they need to be compiled in some way.

This is where `opam monorepo' comes in, it will simply “vendor” your
dependencies into a “duniverse” folder. Here are the steps needed to
compile a project with Esperanto. We’ll take [`decompress'] as an
example which produces a binary that can compress/decompress
documents:
┌────
│ $ git clone https://github.com/mirage/decompress
│ $ cd decompress
│ $ cat >>bin/dune <<EOF
│ (rule
│ (target decompress.com)
│ (enabled_if
│ (= %{context_name} esperanto))
│ (mode promote)
│ (deps decompress.exe)
│ (action (run objcopy -S -O binary %{deps} %{target})))
│ EOF
│ $ cat >dune-workspace <<EOF
│ (lang dune 2.0)
│ (context (default))
│ (context
│ (default
│ (name esperanto)
│ (toolchain esperanto)
│ (merlin)
│ (host default)))
│ $ opam monorepo lock --build-only
│ $ opam monorepo pull
│ $ dune build bin/decompress.com
│ $ sh -c "echo 'Hello World' | ./bin/decompress.com -d | ./bin/decompress.com"
│ Hello World
└────

[`decompress'] <https://github.com/mirage/decompress>

Issues
╌╌╌╌╌╌

Apart from the outcomes described above, however, the Esperanto
toolchain is not complete. Indeed, the OCaml distribution gives
several libraries such as `unix.cmxa' and `threads.cmxa'. A little
work has been done to make the former available. The second one is
however unavailable for the moment since Cosmopolitan only partially
implements `pthread'.

However, it seems that the author of Cosmopolotian wants to implement
the rest of the `pthread' API which will then allow us to provide
support for `threads.cmxa' and OCaml 5.

This of course makes support for the projects more limited than we
imagined (and that’s why this release is experimental) however, an
effort has already been made to lwt into Cosmopolitan’s hypothetical
future support for `pthread'.

Future
╌╌╌╌╌╌

As explained above, support for `threads.cmxa' and OCaml 5 remains the
priority. however, an effort has already been made to support [Lwt]
via Cosmopolitan’s hypothetical future support for `pthread'.

However, it is possible that Cosmopolitan could become a target for
the MirageOS project in the same way as Solo5 (or our recent
experiment on Raspberry Pi 4).

In this sense, we will surely propose an integration in MirageOS so
that projects can both produce unikernels with Solo5 or portable
binaries with Cosmopolitan.

[1] However, the question remains open at several levels, that of the
compiler, that of OPAM and of course that of `dune'. It is clear that
the current situation is not the best in terms of what we need to do
to produce such a cross-compiler. Only the feedback from Solo5 (which
requires cross-compilation) allows us to say that it is surely the
right choice for what we want to offer.

[Lwt] <https://github.com/ocsigen/lwt/>

Conclusion
╌╌╌╌╌╌╌╌╌╌

We hope that this project will facilitate the distribution of
software. You can read a more technical article about our work [here].
Finally, I would like to thank [robur.io] (an association you [can
help]) for allowing me to do this project.

*EDIT*: The author of Cosmopolitan just released Cosmopolitan with
`pthread' support. So we will definitely try to improve our
distribution to include OCaml with `threads.cmxa' support and move
forward with OCaml 5!

[here] <https://blog.osau.re/articles/esperanto.html>

[robur.io] <https://robur.io/>

[can help] <https://robur.io/Donate>

OBazl Toolsuite - tools for building OCaml with Bazel
═════════════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/obazl-toolsuite-tools-for-building-ocaml-with-bazel/10021/15>

Deep into this thread, Yawar Amin asked and Gregg Reynolds replied
──────────────────────────────────────────────────────────────────

Doesn’t dune get advertised as being able to handle
multiple programming languages, including C/C++?

There’s really no comparison. Dune evidently can use the (C ABI)
outputs of a “foreign” build (if you write the glue code needed to
make this work) but there’s no real /build/ integration, and no
hermeticity guarantees. Under Bazel different languages use different
rulesets but they’re all Bazel rulesets, so you get one dependency
graph across all languages, and if the rulesets are hermetic you get a
hermetic build. Without ABI restrictions. For example if your build
needs to run a Python (or Javascript, Ruby, whatever) tool, Bazel will
build the tool and run it for you.

Even for C I think Bazel has much better integration. The rules in
rules_cc (e.g. cc_library producing a .a file) deliver a CcInfo
provider (a provider is a kind of struct whose fields contain the
artifacts delivered by a build action). The rules in rules_ocaml (e.g.
ocaml_module) understand CcInfo dependencies and pass them around
using OcamlProvider (a provider specific to the ocaml rules). Bazel
supports a merge operation for CcInfo, and the ocaml rules always
merge their CcInfo deps and pass them on. So every build target
delivers the merge of all its CcInfo deps. The ocaml_binary rule that
links its dependencies into an executable merges its CcInfo deps
(which include merged CcInfo from their deps, recursively) and ends up
with a single CcInfo containing every cc dependency in the dep graph,
in the right order, with no duplicates. Then its simply a matter of
constructing the link command with the appropriate –ccopt options.
More succinctly: you can add a C dep directly to the module that needs
it, and Bazel it pass it up the dependency chain, ensuring that it
ends up on the command line when needed - building archives or
executables. You don’t need to add a C dep to an archive target when
only one of n modules in the archive actually depends on it.

I’ve just started working on rules_jsoo, which I think will nicely
demonstrate the virtues of Bazel integration. The Bazel ecosystem
includes a bunch of tools for working with Javascript; for example
rules_js and rules_nodejs make it easy to control which node toolchain
version to use, integrate npm stuff, etc. Wouldn’t it be nice to be
able to use such tools directly, without writing a bunch of glue code?
Now a key element of Bazel integration is the use of providers. Rules
deliver providers, and since providers act as a kind of rudimentary
type system, I can use the JsInfo provider (defined by rules_js) to
integrate rules_jsoo with the larger Bazel js ecoystem. For example,
the jsoo_library rule takes the OcamlProvider provider delivered by
ocaml_module rules, which contains the .cmo file. So jsoo_library runs
those .cmo files through the jsoo compiler and delivers the resulting
js files in a JsInfo provider. That provider is suitable as input to
the rules in rules_js, which gives us seamless integration. So we can
use the js_binary rule of rules_js to run code produced by
jsoo_library under node. All that’s needed is to list the latter as a
dependency of the former. That’s the plan, anyway. Isn’t that nice?

Gregg Reynolds said and Daniel Bünzli replied
─────────────────────────────────────────────

Ideally somebody learning a new language should not need
to spend any time (at first) dealing with a build language
too.

This doesn’t only apply to learning. It also applies to prototyping,
hypothesis generation and testing.

That’s the reason why I built [`brzo'] which I hope I’ll be able to
release at some point (still needs a good design review and changes to
the OCaml strategy since it assumed we were moving towards a model
that didn’t happen in the end – namely the [library linking proposal],
I’d also like to add more languages to the mix but that could wait).

None of my projects do not start with ~brzo~ing these days and the
hassle free build experience is exhilarating.

Build systems often are “complex and confusing”, but
that’s largely because the problem space itself is complex
and confusing. There’s no getting around that.

Note however that this is largely /accidental/ complexity due to the
fact that compilers work in idiosyncratic ways for what build systems
need in order to do their incremental and parallelization business.

They are still stuck in a world where people would invoke their
compiler manually at the cli level or specify the invocations
themselves in a `Makefile'.

In fact if it were not for the actual tools and the (lack) of
information they give us, build is in fact an excessively simple
problem.

More specific to OCaml, the compiler clis have an insane amount of
quirks and the whole system greatly suffers from an underspecified
linking model. Basically it was not a good idea to let that be defined
by a third party tool, if only so that you can actually talk about
libraries in error messages from the compiler.

[`brzo'] <https://erratique.ch/software/brzo>

[library linking proposal] <https://github.com/ocaml/RFCs/pull/17>

Orsetto: structured data interchange languages (release 1.1.2)
══════════════════════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/ann-orsetto-structured-data-interchange-languages-release-1-1-2/10506/1>

james woodyatt announced
────────────────────────

Announcing the release to OPAM of [Orsetto 1.1.2], an update to a
personal project of mine not sponsored by my employer. Licensed with
BSD 2-Clause.

*Q. What is Orsetto?*

Aspires to do eventually for OCaml more or less what [Serde] has done
for Rust, i.e. to be a curated and self-contained collection of
structured data interchange languages with a cohesive and unified
model of serialization and deserialization.

Two interchange languages are currently supported: [JSON ] and [CBOR
].

*Q. What is new in this release?*

Mostly error corrections, particularly in the CBOR library, produced
by improving test coverage.

The change log for the release is here: [CHANGES.md ]

Highlights:

• Major improvements in test coverage.
• Many corrections for logic errors.
• A few minor usability improvements.

Some things have not changed:

• Still has no Programmer Guide or Tutorial, or really any
introduction at all.
• Still requires *ocamlfind* and builds with *omake*, which is
currently not compatible with OCaml 5.0.
• Still only supports JSON and CBOR.

*Q. It looks incomplete. What are your plans for future development?*

Yes, it’s a personal project, and yes, I’m aware there are no reverse
dependencies on it currently in the public OPAM repository. Still, I’d
welcome opportunities to collaborate with others who share my vision
for the project. As long as it’s just me working on this, development
will continue to be somewhat slow, as I’m prone to over-engineer
things I care about. I have a lot of projects, and this is the only
open source one.

• *Orsetto 1.0.3* is the previous release. It offered parsers and
emitter combinators for JSON and CBOR for OCaml >= 4.06.1
(including 4.13~alpha1). The quality of its JSON support is
adequate, and it scores well on the >[nst/JSONTestSuite] tests. The
quality of its CBOR support is provisional, >and not recommended.
• *Orsetto 1.1.2* is the current release. It adds generalized and
extensible structured data interchange models with specializations
for producing emitters and parsers for JSON and CBOR. The quality
of the CBOR support is much improved, and I’m using it with good
results in other projects. Supported on OCaml >= 4.08.
• *Orsetto 1.2* is the next planned release. It will drop interfaces
marked `@caml.deprecated` in the 1.1 release. It will also drop
support for OCaml < 4.10, and it will stop depending on
**ocamlfind**. I hope to add a PPX for deriving parsers and
emitters from OCaml data type definitions. I might also consider
one or more new interchange languages— suggestions are heartily
encouraged.

[Orsetto 1.1.2] <http://opam.ocaml.org/packages/orsetto/>

[Serde] <https://serde.rs>

[JSON ] <https://json.org>

[CBOR ] <https://cbor.io>

[CHANGES.md ] <https://bitbucket.org/jhw/orsetto/src/r1.1.2/CHANGES.md>

[nst/JSONTestSuite] <https://github.com/nst/JSONTestSuite>

Interest in a Http_sig library?
═══════════════════════════════

Archive:
<https://discuss.ocaml.org/t/interest-in-a-http-sig-library/10518/1>

Kiran Gopinathan announced
──────────────────────────

Heyo all! I’ve been working on an activitypub server for a while now,
and while it’s still not yet complete, recently I’ve reached a point
where I realised that I’ve actually been sitting on some libraries
that the community might benefit from, as the current ecosystem
doesn’t seem to handle these things.

One such component that seemed to be in a state that was suitable to
split off from was a small helper module to implement a particular
http signature scheme that seems to be rather common in the
activitypub scene.

In particular, the scheme I’m referring to is defined here:
<https://datatracker.ietf.org/doc/html/draft-cavage-http-signatures-12>

┌────
│ Signing HTTP Messages
│ draft-cavage-http-signatures-12
│
│ Abstract
│ When communicating over the Internet using the HTTP protocol, it can
│ be desirable for a server or client to authenticate the sender of a
│ particular message. It can also be desirable to ensure that the
│ message was not tampered with during transit. This document
│ describes a way for servers and clients to simultaneously add
│ authentication and message integrity to HTTP messages by using a
│ digital signature.
└────

I’ve written a small library that glues together some components in
the OCaml ecosystem to somewhat handle the signing (I have been mainly
working off an “implement-enough-to-make-the-system-work” process
rather than directly transcribing the specification above):

┌────
│ (** [verify ~signed_string ~signature key] returns true iff
│ [signature] over [signed_string] is valid according to [key]. *)
│ val verify: signed_string:string -> signature:string -> X509.Public_key.t -> bool
│
│ (** [verify_request ~resolve_public_key req] verifies that a dream
│ request has been signed according to the HTTP signature scheme *)
│ val verify_request:
│ resolve_public_key:(string -> (X509.Public_key.t, 'a) Lwt_result.t) ->
│ Dream.request -> (bool, 'a) result Lwt.t
│
│ (** [build_signed_headers ~priv_key ~key_id ~headers ~body_str
│ ~current_time ~method_ ~uri] returns a list of signed headers using
│ [priv_key] according to the HTTP signature scheme. [key_id] should
│ be a string that can be used to look up the public key associated
│ with [priv_key]. *)
│ val build_signed_headers:
│ priv_key:X509.Private_key.t ->
│ key_id:string ->
│ headers:string StringMap.t ->
│ body_str:string ->
│ current_time:Ptime.t -> method_:string -> uri:Uri.t -> (string * string) list
└────

The library is currently published at
<https://github.com/Gopiandcode/http_sig_ocaml> under the LGPL, but I
haven’t released it on opam.

Anyway, I was wondering if anyone else had interest in this kind of
package, and whether it would be a good candidate for submission to
opam - or if there are actually already existing libraries in the
OCaml ecosystem that would actually already do this.

Outreachy summer ’22 closing commemoration session on 23rd Sept
═══════════════════════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/outreachy-summer-22-closing-commemoration-session-on-23rd-sept/10450/5>

Patrick Ferris announced
────────────────────────

Thank you to everyone that could make it to the presentation today.
The presentation is now live:
<https://watch.ocaml.org/videos/watch/dc5bbf5b-3dd9-4c8d-b26a-71e730a67788>
:camel:

In particular a massive congratulations and thank you to
@moazzammoriani and @IIITM-Jay. Thank you also to @sudha for being the
driving force behind making the presentation happen again this round!
See you all for the next round!

Aside: if anybody has any issues with the live video please do reach
out here either publicly or privately, we gave prior warning of our
intentions to record and put the video on watch.ocaml.org, but I
appreciate some people joined a little later/might have some
reservations etc.

findlib-1.9.6
═════════════

Archive:
<https://sympa.inria.fr/sympa/arc/caml-list/2022-09/msg00007.html>

Gerd Stolpmann announced
────────────────────────

findlib-1.9.6 is out, now supporting OCaml-5.00 (as far as we know
it). There are also a few other install-related fixes in it.

For manual, download, manuals, etc. see here:

<http://projects.camlcity.org/projects/findlib.html>

An updated OPAM package will follow soon.

Interesting OCaml Articles
══════════════════════════

Archive:
<https://discuss.ocaml.org/t/interesting-ocaml-articles/1867/100>

Deep in this thread, alan said
──────────────────────────────

An interesting paper that uses OCaml is
<http://gallium.inria.fr/~fpottier/publis/fpottier-elaboration.pdf> by
Francois Pottier, which gives a declarative DSL for implementing type
rules with applicative functors. It has an associated library,
<https://opam.ocaml.org/packages/inferno/>.

Other OCaml News
════════════════

>From the ocaml.org blog
───────────────────────

Here are links from many OCaml blogs aggregated at [the ocaml.org
blog].

• [Tarides Sponsors High School Hackers]

[the ocaml.org blog] <https://ocaml.org/blog/>

[Tarides Sponsors High School Hackers]
<https://tarides.com/blog/2022-09-23-tarides-sponsors-high-school-hackers>

Old CWN
═══════

If you happen to miss a CWN, you can [send me a message] and I’ll mail
it to you, or go take a look at [the archive] or the [RSS feed of the
archives].

If you also wish to receive it every week by mail, you may subscribe
[online].

[Alan Schmitt]

[send me a message] <mailto:alan.schmitt@polytechnique.org>

[the archive] <https://alan.petitepomme.net/cwn/>

[RSS feed of the archives] <https://alan.petitepomme.net/cwn/cwn.rss>

[online] <http://lists.idyll.org/listinfo/caml-news-weekly/>

[Alan Schmitt] <https://alan.petitepomme.net/>

[-- Attachment #2: Type: text/html, Size: 46499 bytes --]