OCaml Weekly News

Here are the meeting notes of the July meeting: https://github.com/ocaml-ppx/ppxlib/wiki/Dev-meeting-18-07-2023

As a heads-up: There won’t be a meeting in August, so the next one will be in September. Have a nice summer/winter/other season everyone! :)

Here’s a paper on algorithms for theoretical particle physics referring to ocaml published today:

Functional directed acyclical graphs for scattering amplitudes in perturbation theory

It looks like this paper builds on previous work implementing a tool called Omega in OCaml, which has been around since 2001:

O’Mega: An Optimizing Matrix Element Generator
Mauro Moretti, Thorsten Ohl, Jürgen Reuter, 2001
https://arxiv.org/abs/hep-ph/0102195

There are several major changes in version 2.0.0 of DkML, a OCaml distribution that uses Visual Studio on Windows. Full DkML 2.0.0 Release Notes are available, and the installer is https://gitlab.com/diskuv-ocaml/distributions/dkml/-/releases/2.0.0/downloads/setup64u.exe.

Consider 2.0.0 a preview release. If you are risk-averse you may want to wait for 2.0.1.

DkML 2.0.0 Release Notes: https://gitlab.com/diskuv-ocaml/distributions/dkml/-/releases#2.0.0

Breaking change: The global environment (where you can run dune, utop, ocamlopt and a few other critical OCaml executables without having to install your own opam switch) has changed significantly. [DkML 2.0.0 Release Notes] describes those changes in detail.

Why do this breaking change?

1. 2.0.0+ installations are faster because the global executables do not have to be compiled.
2. It gets the DkML distribution very close to having a lite “bytecode only” installer that does not need to install (heavy!) Visual Studio, MSYS2 and Git. This upcoming lite installer could be a good fit for educational settings, so feedback from educators is appreciated.

Major non-breaking changes:

• The deprecated fdopen repository is no longer used. Previously DkML installed a smaller and smaller portion of the fdopen repository with each subsequent version. Now DkML 2.x.x relies only on the central opam repository and the DkML opam repository.
• Dune upgraded to 3.8.3. Among other things, the installer no longer installs a C# file watcher proxy for dune build -w; instead Dune uses its own native Windows file watcher.
• The set of pinned packages during a dkml init local switch has gone down from approximately 4000 to 200. Rather than pinning each package in the opam universe to a heuristically-determined version, we pin only packages that we successfully build on Windows.
• The following packages are accessible by just typing utop without installing a switch:
  • refl reflection package
  • graphics package
  • base package

Bug fixes:

• The installer can now restart after a failed installation, without having to use the uninstaller.
• Allow setup to succeed on Windows filesystems that do not support the setting of file attributes
• FlexDLL object files flexdll_initer_msvc64.obj and flexdll_msvc64.obj (or 32-bit variant on 32-bit installs) are installed alongside flexlink.exe so flexlink can be used by itself without setting FLEXDIR environment variable.

Changed packages:

• Many packages are upgraded. Here is a sampling of those upgrades:

  Package	From	To
  dune	3.6.2	3.8.3
  utop	2.10.0	2.13.1
  ptime	0.8.6-msvcsupport	1.1.0
  flexdll	0.42	0.43
  base	v0.15.1	v0.16.1
  yojson	2.0.2	2.1.0
  stdcompat		19+optautoconf
  metapp		0.4.4+win
  ocamlformat	0.24.1	0.25.1
  lsp	1.12.2	1.16.2
  cmdliner	1.1.1	1.2.0
  alcotest	1.6.0	1.7.0
  ocamlfind	1.9.1	1.9.5

I would be hard pressed to mention one priority for OCaml, because I think that many things are important – and important in different ways for different people. I thought that I could give some information on what support I have been involved in, either as a compiler maintainer or as a member of the OCaml Software Foundation, on some of the topics that have been mentioned here.

• Modular implicits: we’ve been trying to find people we could fund to make progress on this, but it is difficult – it requires a lot of type-system expertise, and time. We are funding internships with @yallop in Cambridge to experiment with the system, and I am thinking of getting my hands dirty during the next school year with some module-side implementation work – possibly diving into the PRs of Matthew Ryan.
• Type system for effects: I have decided to setup an informal working group for people interested in working on this (mostly academics, also Jane Street) to exchange information, and in particular to decide a common set of evaluation criteria – with compatibility with existing non-effect-using OCaml code as a top priority. I hope to announce something more structured next September, with the aim of getting a clearer idea of potential design proposals by the end of the academic years.
• Debuggers: the OCaml Foundation funded ocamlearlybird’s author in 2020-2021, hackwaly, before they stopped to work full-time for a startup. (https://discuss.ocaml.org/t/ann-ocamlearlybird-now-an-ocaml-software-foundation-supported-project/6834 , https://discuss.ocaml.org/t/ann-ocamlearlybird-1-0-0-beta1/7180 , etc. ) I learned a week ago that @sim642 is interested in continuing maintenance of the project, and we are looking at funding his work. (ocamlearlybird is a bytecode debugger, similar to the venerable ocamldebug – it is a client for the same debugger protocol. Separately we would ideally have good debugging support with the native compiler. That aspect of the debugging story has been worked on by Mark Shinwell in the past, but there is a disagreement among compiler maintainers about what is the right technical approach to support DWARF information in the compiler – it is a complex format that is not necessarily a good match for OCaml, and adding it to the compiler backend is fairly invasive.)
• A priority that people don’t know about, but in my mind comes before “typed effects”, is to refactor and cleanup the implementation of the OCaml type checker, which has historically been a place of technical debt in the compiler. There has been a fair amount of work on this in the last year, driven by Jacques Garrigue as his group at Nagoya ( in particular Takafumi Saikawa ), with Florian Angeletti and myself on the reviewer side, and in the last few months some very welcome help from the Jane Street group working on language features (in particular Richard Eisenberg, Chris Casinghino, Nick Roberts) who have been lending a hand on refactoring and refactoring-reviewing work. More generally, I have been worried this year about maintenance workforce for the OCaml compiler codebase (https://discuss.ocaml.org/t/maintenance-bottlenecks-in-the-compiler-distribution/11045 ). I think that solving this issue is also a priority, at the level of the compiler distribution. We have made some progress, with notably maintenance contributions from OCamlPro and Jane Street, but I think that the situation still requires careful monitoring.
• Relocatability: I agree with @smorimoto that this is important. I would want all OCaml packages to be relocatable, notably as the right first step to enable caching or even binary distribution of OCaml package build artifacts. There has been work in this direction from David Allsopp in the last few years, which I understand gets closer every year to being in a state that could be submitted as upstream PRs. My strategy so far is to wait for this to make progress.
• Tooling, things that directly come to mind are:
  • development-environment tools whose recommended workflow is to have separate/isolated development environment(s) for each project. The opam client allows this with local switches, but I think the UI could be streamlined a bit to make this the easy default. (For example, not everyone knows how to build a local switch with only the dependencies of the project installed, to start hacking on it.) Ideally we would have caching of package builds across those development environments, and maybe even some distributed caching of build artifacts (assuming relocatability).
  • a “code upgrade” tool to which I can specify program transformations (for example: “rewrite SomeLib.somefun $bar $baz into SomeLib.somefun $bar (Somelib.convert $baz)”, and it applies it to my source code in a diff-friendly way.

My view on this. When people talk about Opam, this covers different aspects.

• the opam files and files repositories. The main repository is (for us OCamlers) ocaml/opam-repository, but the [Coq community also has one](https://github.com/ocaml/opam/wiki/Dev-Meeting-2020.11.06-(opam-and-coq-repos)), and many companies use their private repository. We do not want to break this workflow. The Dune package management proposals aim to stay fully compatible with this workflow and work with any opam packages (using dune or not). There’s also exciting ongoing work for [package signing](https://github.com/ocaml/opam/wiki/Dev-Meeting-2021.06.11-(Conex)) that we do want to see land at one point. We have also built and are operating an extensive CI infrastructure around these repositories – for instance, ocaml-repo-ci is building 100,000 jobs daily on all the Tier1 supported platforms for OCaml. We do not want to rebuild this once more time!
• the opam client(s). The main one is the opam CLI, but many more tools use the opam files metadata. There are a [few tentatives](https://github.com/ocaml/opam/wiki/Dev-Meeting-2020.10.23-(opam2nix)) of generating nix derivations for those files. There’s also [esy](https://github.com/ocaml/opam/wiki/Dev-Meeting-2020.12.04-(esy-drom)) and the package managers that try to close the gap between the OCaml and JavaScript ecosystems. The client is built around a library (opam-lib), but this has never been designed properly. When I wrote opam initially, it was only focused on the CLI. Later with @AltGr, we tried to split it a bit more cleanly, but the API is still painful to use (for instance, every function that needs to load the filesystem needs a value that holds that state that would take dozens of arguments - then, as these functions perform file-system or network effects, you somehow need to keep these values synchronised with the new filesystem state – that’s painful and error prone). This API can somehow be split into various parts:
  1. Reading the opam repository state: parsing opam files, building a dependency graph (that’s the part you mention @gasche)
  2. Resolving constraints: opam has a pluggable interface for constraint solving, and by default, it will use what solver is installed on your system (or some internal heuristics which used to be very naive but seems much better nowadays). Opam needs to serialise and parse solver requests, including solver errors that need to be somehow pretty printed to the user.
  3. If the solver can devise a building plan, parse it and prepare it by downloading (and caching) the build/package sources.
  4. run the build commands for all the packages and install them locally.

Nowadays, most package managers also have the option to snapshot the state of the build plan between 2. and 3. That’s for instance, what tools like opam-monorepo lock or opam lock are doing. But when you do opam install --locked, opam still calls the solver (to check that your lock file is consistent and complete), so doing 3-2-3-4. And opam-monorepo pull does 3. and delegate 4. to dune build (so all your dependencies need to use dune and be co-installable in a dune workspace).

So to come back to your question @gasche: The Dune package management experiment will be using opam-lib to do 1, 2 and 3 (whether it’s the current opam-lib or an improved version that relies a bit less on filesystem state is still in discussions - whatever the result is it will be upstreamed). And it will be using the dune scheduler to do 4 (but still using the opam build instructions).

We are happy to announce the release of OCamlFormat 0.26.0, the auto-formatter for OCaml code.

This release contains changes that might affect your code and overall projects. It could include impact to module arguments, fun passed as a labelled argument, and indentation within parentheses.

The complete list of changes is here. Items marked with a * are likely to change your code.

The OCamlFormat dev team

The current-bench infrastructure and pipeline are used for benchmarking the OCaml compiler and ecosystem projects. It uses the OCurrent pipeline to run predictable, I/O-bound benchmarks for the respective OCaml projects.

We have successfully added benchmarks for Dune, Lockfree, odoc, Merlin, Eio, opam, etc., projects at https://autumn.ocamllabs.io. We would now like to make our infrastructure available to the larger community to increase adoption and continue our work on performance analysis, tooling, and benchmarking.

You can follow the instructions in the current-bench README to create a benchmark for your project and install the GitHub application to onboard your OCaml project to current-bench.

The Tarides Benchmark Tooling team is also making open office hours available for you to engage, ask questions, and work with us on benchmarking OCaml projects. We will be available online, every Monday, between 1030 and 1130 CET on Google Meet as mentioned below:

Tarides Office Hours: OCaml Benchmarking Starting Monday, July 24, 2023 (1030 - 1130 CET) (1400 – 1500 IST)

Google Meet joining info: Video call link: https://meet.google.com/rcu-wjpk-jyh Or dial: ‪(US) +1 440-549-4808‬ PIN: ‪584 331 902‬#

More phone numbers: https://tel.meet/rcu-wjpk-jyh?pin=4479584706118

Please feel free to join our weekly Monday sessions, and we will be happy to interact with the community and enhance our benchmarking efforts for OCaml!

the OCaml.org maintainers are now holding community meetings, on Mondays every two weeks on Zoom.

Here are some dates for the upcoming meetings (scheduled on Google Calendar):

• 24 Jul 2023, Mon 18:00 – 19:00 CEST
• 7 Aug 2023, Mon 18:00 – 19:00 CEST
• 21 Aug 2023, Mon 18:00 – 19:00 CEST
• 4 Sept 2023, Mon 18:00 – 19:00 CEST

Topics may include:

• Proposals you want to champion
• Q&A sessions about the project
• Guidance on contributing

But we’re not limiting topics to just these. Many more topics relevant to OCaml.org and how to best make use of it for the benefit of the OCaml community can also be discussed. If you’d like to add a topic/issue to the discussion for a specific meeting, please contact Sabine on OCaml Discuss or the OCaml.org Discord channel, or open an issue on the ocaml/ocaml.org GitHub Repository

P.S.: https://ocaml.org/news/invitation-to-contribute

Thanks to @mseri, euler is now on opam! And in the meantime it has gained more functions, the most striking of which being Primes.prime_seq : int -> int Seq.t (thank @cuihtlauac for pointing me to the algorithm!) and Arith.smallest_root : int -> int * int (the latter in the next release, 3.0, not on opam yet).

Also, since I still cannot edit the first post, this is a reminder that the project has moved to GitHub: repo, doc. The links in my first message won’t be updated.

I wonder if this is a useful addition to Owl , the ocaml “fat” math library.

Perhaps? I am not very familiar with Owl (ask Owl’s Benevolent Dictators For Life? ;-) ). Euler is an integer arithmetic library whereas, as I understand it, Owl is mainly a floating-point math library for scientific computing (linear algebra, differentiation, statistics and and such). I would guess that prime numbers and friends are rather remote concerns for Owl users, but perhaps Owl is interested in expanding its domains? I did spot a small number of integer arithmetic functions in Owl, most of which are also found in Euler.

I’m pleased to announce that DkML Install API 0.4.0 is available on opam. This library is what I’ve been using to create the DkML Windows distribution installer.

TLDR: To create your own installer for your own software, you define “components” which have dependencies on other components. A component is an OCaml module you create with four essential functions: admin_install and admin_uninstall for escalated privileges, and user_install and user_uninstall for normal user privileges. Each component almost always needs to access assets which can be defined in the install:[] section of an .opam file. The net effect is you can use opam and OCaml to define your own reproducible installer.

There is a walk-through at https://diskuv.github.io/dkml-install-api/doc/packages/Console.html and more general documentation at https://github.com/diskuv/dkml-install-api#readme

The library works well for the DkML Windows distribution, although it is too complicated to recommend for casual use today. Regardless, if someone has a burning desire to create their own installer, and that installer has reasonably complex logic that needs to run on the end-user machine, then take a look at DkML Install API! Expect to see a steady but slow series of changes to the API until it hits 1.0.0.

Limitations: Today it can make an end-to-end installer for Windows. I haven’t needed yet to create macOS and Linux installers, although 95% of the pieces are present. If you are interested in creating the final packaging step for those OS-es (ex. a .deb apt packager for Debian), feel free to contribute a PR!

We are delighted to announce the release of Liquidsoap 2.2.0! It is now available on our release page and should be available via opam shortly!

Liquidsoap is a statically typed scripting general-purpose language with dedicated operators and backend for all thing media, streaming, file generation, automation, HTTP backend and more.

This release comes almost 4 months after the last stable release from the 2.1.x release cycle and 14 months (!) after forking the 2.2.x release branch! It includes some exciting changes in track muxing/demuxing, HLS, sound processing and more. See below for a detailed list!

I’m happy to announce that version 0.6.1 of kcas and kcas_data has been released on opam.

See the Kcas project for more details, but briefly kcas provides a software transactional memory (STM) implementation and kcas_data provides compositional lock-free data structures implemented using Kcas. Together these libraries allow one to implement safe concurrent abstractions compositionally.

Aside from internal improvements, notable features added since the previously announced 0.3.0 version include:

• Doubly linked list Dllist, see example.
• Synchronizing variable MVar, see example.
• Nested conditional transactions, see example.
• Timeouts, see example.
• Support for OCaml 4.14, allowing Kcas to be used to implement systhread and parallelism safe code and help projects requiring OCaml 4 support on their way to OCaml 5.

Stay tuned for more news on Kcas!

I’m happy to announce a new release of ctypes, 0.21.1, available for installation via OPAM.

ctypes is a library for binding to C libraries using pure OCaml, with the primary aim of make writing C extensions as straightforward as possible.

The core of ctypes is a set of combinators for describing the structure of C types – numeric types, arrays, pointers, structs, unions and functions. You can use these combinators to describe the types of the functions that you want to call, then bind directly to those functions – all without writing any C!

The major change in this release is a switch to the dune build system, contributed by Anil Madhavapeddy (@avsm) and Etienne Millon (@emillon). The benefits of this change for users of ctypes are described in the pull request implementing the change; in summary:

With a port to dune, the ctypes library can be embedded in larger dune projects simply by including it in the directory tree of the bigger project.

The release notes have further details of the changes in this release and recent point releases.

We are happy to announce the creation of the ocaml-wasm Github organisation which will work towards WebAssembly (Wasm) support for OCaml. The goal of the organisation is to enable users to build high-performance and secure applications for the Wasm platform using the OCaml programming language and the platform tools.

Wasm is a binary instruction format, designed as a portable compilation target for programming languages, enabling deployment on the Web for client and server applications. It provides a sandboxed execution environment, which also makes it attractive for both blockchains and Cloud deployment. For example, Fastly’s Compute@Edge platform is based on WebAssembly.

Hey everyone,

there’s a new tutorial on “Arrays” at

https://staging.ocaml.org/docs/arrays

Looking for constructive feedback.

Is there any information missing or questions that you would need answered in order for this tutorial to be useful? Anything you would change?