OCaml Weekly News

• Introduce $ dune ocaml doc to open and browse documentation. (#7262, fixes #6831, @EmileTrotignon)
• dune cache trim now accepts binary byte units: KiB, MiB, etc. (#8618, @Alizter)
• Introduce the runtest_alias field to the cram stanza. This allows removing default runtest alias from tests. (@rgrinberg, #8887)
• Display cache location in Dune log (#8974, @nojb)
• Dune can now be built and installed on Haiku (#8795, fix #8551, @Alizter)
• Mark installed directories in dune-package files. This fixes (package) dependencies against packages that contain such directories. (#8953, fixes #8915, @emillon)
• Introduce new experimental odoc rules (#8803, @jonjudlam)

• dune-build-info: when version="" is found in a META file, we now return None as a version string (#9177, @emillon)
• No longer force colors for OCaml 4.03 and 4.04 (#8778, @rgrinberg)
• Dependencies in the copying sandbox are now writeable (#8920, @rgrinberg)
• Rules that only use internal dune actions (write-file, echo, etc.) can now be sandboxed. (#9041, fixes #8854, @rgrinberg)
• Add test_ prefix to default test name in dune init project (#9257, fixes #9131, @9sako6)

• Do not ignore libraries named bigarray when they are defined in conjunction with OCaml 5.0 (#8902, fixes #8901, @rgrinberg)
• Correctly ignore bigarray on recent versions of OCaml (#9076, @rgrinberg)
• Absent packages shouldn’t prevent all rules from being loaded (#8948, fixes #8630, @rgrinberg)
• Correctly determine the stanza of menhir modules when (include_subdirs qualified) is enabled (@rgrinberg, #8949, fixes #7610)
• Re-run actions whenever (expand_aliases_in_sandbox) changes (#8990, @rgrinberg)
• Do not re-run rules when their location changes (#9052, @rgrinberg)
• [coq rules] Be more tolerant when coqc –print-version / –config don’t work properly, and fallback to a reasonable default. This fixes problems when building Coq projects with (stdlib no) and likely other cases. (#8966, fix #8958, @Alizter, reported by Lasse Blaauwbroek)
• Dune will now run at a lower framerate of 15 fps rather than 60 when INSIDE_EMACS. (#8812, @Alizter)

(W4)

Contributed by: @rgrinberg (Tarides), @Leonidas-from-XIV (Tarides), @gridbugs (Tarides), @kit-ty-kate (Tarides), @Alizter

Why: Unify OCaml tooling under a single command line for all development workflows. This addresses one of the most important pain points reported by the community.

What: Prototyping the integration of package management into Dune by using opam as a library. We’re adding a new dune pkg lock command to generate a lock file and extend dune build to support downloading and building dependencies specified in the lock file. Read the Dune RFC for more details.

Activities:

• We reworked the storage for opam repositories. We now have support for multiple repositories, which we store and update in an efficient manner. – ocaml/dune#8950
• We introduced the $ dune pkg outdated command to view all the outdated packages in a lock directory. – ocaml/dune#8773
• We introduced $ dune describe pkg lock to print lock directories. This gives users a nice overview of what’s available in the lock directory’s build plan. – ocaml/dune#8841
• We added support for solver variables in lockdir to make sure opam variables are also available at build and install time – ocaml/dune#8973
• We managed to successfully generate a lock file for cmdliner on Windows! :windows: Next, we’re working on making build work on Windows as well. – ocaml/dune#9048
• And as usual, we fixed a bunch of bugs that prevented your regular packages from building:
  • Require copying sandbox for build rules – ocaml/dune#8923
  • Respect [flags] field in opam packages – ocaml/dune#9047
  • Improve invalid substitute error – ocaml/dune#8922
  • Correctly verify tarball checksums – ocaml/dune#8876
  • Improve locations of conversion errors – ocaml/dune#8828
  • Remove post deps – ocaml/dune#8834
  • Move solver env printing to own command – ocaml/dune#8819
  • Record installed directories in dune-package – ocaml/dune#8953

(W5)

Contributed by: @rjbou (OCamlPro), @kit-ty-kate (Tarides), @dra27 (Tarides), @AltGr (OCamlPro)

Why: Opam and opam-repository currently don’t support Windows natively. This effectively makes OCaml a very niche candidate on Windows, as users either have to (1) not use a package manager or (2) use a fork of opam and the opam-repository. Making opam and the opam-repository compatible with Windows will make OCaml a better choice for Windows users and help us grow the community. More Windows users able to use opam leads to more contributors, more testing, more Windows friendly packages, and more packages in the end.

What: Releasing opam 2.2 with native support for Windows and making the official opam-repository usable on Windows.

Activities:

• Essentially focused on setenv & build-env environment variables update handling on Windows - ocaml/opam#5636

(W25)

Contributed by: @panglesd (Tarides), @EmileTrotignon (Tarides), @julow (Tarides), @jonludlam (Tarides)

Why: The in-package search added in OCaml.org’s central package documentation has been very well received by the community and improves how users navigate and discover OCaml documentation. We’re upstreaming it to odoc to bring it into the local documentation as well and provide more advanced features, like searching by type.

What: We’re adding support in odoc for pluging in a search engine! odoc provides the UI (a search bar) and will generate a search index (that can also be used to be integrated into other search engine like Elasticsearch). We’re also building a default client-side search engine based on Sherlodoc.

Activities:

• We’ve merged the PR adding new search capabilities! :tada: This will ship in the upcoming version of odoc 2.4.0.
• We are currently experimenting with building an odoc search engine based on Sherlodoc --art-w/sherlodoc#4.

(W25)

Contributed by: @panglesd (Tarides), @jonludlam (Tarides), @dbuenzli, @gpetiot (Tarides)

Why: Allow package authors to write rich documentation, making it more useful and improving OCaml developer experience by providing an ecosystem of high-quality documentation for OCaml packages.

What: We’re adding support for assets and new syntax to embed medias (images, audio, and videos).

Activities:

• We’ve decided to rework the asset support design a little to bring it more in line with how other elements are handled, such as modules, types, and values. The intent is to ensure the code is straightforward to maintain in the long term. – ocaml/odoc#1002
• We continued reviewing the PR, adding syntactic support for media. There were some good discussions relating to missing assets, and it’s ready to go in once the assets PR is merged. – ocaml/odoc#1005

(W25)

Contributed by: @jonludlam (Tarides)

Why: Make locally-generated documentation more useful by allowing users to navigate to their dependencies’ documentation from their package docs. Currently users can use odig, which provides a similar workflow. We’re adding support for this in Dune directly.

What: We’re writing new Dune rules for odoc that (1) use the new odoc CLI to enable performance improvement and caching opportunities and (2) generate the documentation of every opam packages in your switch, allowing users to navigate to their dependencies’ documentation from their local docs.

Activities:

• The new Dune rules have been reworked a little following feedback from @alizter and @rgrinberg, improving the following areas: better support for multiple findlib directories; better support for system switches; and a more robust method for translating from findlib paths to local paths. – ocaml/dune#8803

(W19)

Contributed by: @voodoos (Tarides), @trefis (Tarides), @Ekdohibs (OCamlPro), @gasche (INRIA)

Why: Project-wide reference as an editor feature is a great way for developers to navigate their codebase and understand it better. It’s also a feature that users expect to have coming from other ecosystems, so having support for it in Merlin and OCaml LSP will both improve OCaml editor experience and make it on par with other languages.

What: We’re adding a new merlin single occurrences command and support for the LSP textDocument/references request. To do that, we’re extending the compiler’s Shapes to support global occurences, building a tool that generates an index of identifiers in a codebase and adding support for it in Dune, Merlin, and OCaml LSP.

Activities:

• We continued working on the compiler PR for project-wide occurrences, notably adding support for inline records’ labels. The PR is now ready for the next round of reviews. – ocaml/ocaml#12508
• Concurrently, we continued working on the tools involved in providing occurrences. We are still on track for releasing an experimental 4.14-based variant of the compiler to gather feedback on the feature before the end of the year. Our current aim is to provide official project-wide occurrences support in OCaml 5.2. – voodoos/merlin#8, voodoos/ocaml-lsp#2, voodoos/dune#2, ocaml-index#5

Contributed by: @pitag (Tarides), @Engil (Tarides), @3Rafal (Tarides)

Why: Some Merlin queries have been shown to scale poorly in large codebases, making the editor experience subpar, with users reporting that they sometime must wait for a few seconds to get the answer for Merlin. This is obviously a major issue that hurts developer experience, so we’re working on improving Merlin performance when it falls short.

What: We’re building benchmarking and fuzzy-testing CIs to continuously benchmark and test Merlin. We’re addressing the performance bottlenecks identified from profiling Merlin and analysing benchmarking results.

Activities:

• We’ve continued our work on a fuzzy-testing CI for Merlin. Our first approach was to persist the testing data in sync with the Merlin commit history. However, that implied dealing with all kinds of data races when comparing the data between a PR and its base branch, when generating and persisting new data, and when approving changes. To avoid that, in October we experimented with a simpler approach that regenerates the data for every CI run - without compromising on CI run time. The new approach seems promising.
• We’ve also come back to work on Merlin performance improvements. We plan to optimise Merlin’s space-time trade-off by experimenting (on an opt-in basis) with different lifetimes for the cmi-cache and cmt-cache. For that, we’re adding information about Merlin’s memory usage to its telemetry. Furthermore, we’ve looked into refining the typer cache granularity and have analysed its challenges.

The base compiler can be installed as an opam switch with the following commands on opam 2.1 and later:

    opam update
    opam switch create 5.1.1~rc1

The source code for the release candidate is also directly available on:

• GitHub: https://github.com/ocaml/ocaml/archive/5.1.1-rc1.tar.gz
• OCaml archives at Inria: https://caml.inria.fr/pub/distrib/ocaml-5.1/ocaml-5.1.1~rc1.tar.gz

The candidate should be familiar with formal approaches in programming language design, notably type systems, semantics, and logic. More concretely, knowledge of the OCaml programming language is expected, and a knowledge of abstract interpretation or model checking would be highly appreciated.

This postdoc strongly relies on the fact that practical implementation should have strong theoretical foundations and that further refinements of the theory should get inspiration from the practical side.

• Applications will be processed starting the 18th of December. Late applications will be considered until the position is filled.
• You do not need to have defended your PhD thesis to apply, but you will need to have obtained your PhD to start the contract.
  • Candidates can send their application to Guilhem Jaber (guilhem dot jaber at inria dot fr) and Gabriel Radanne (gabriel dot radanne at inria do fr) with a subject containing “[CAVOC post-doc application]“.
• The application should contain a CV, two selected publications and two contacts of reference persons (or reference letters if available).

The CAVOC project aims to develop a sound and precise static analyzer for OCaml, that can catch large classes of bugs represented by uncaught exceptions. The analyzer reasons compositionally on programs, in order to analyze them at the granularity of a function or of a module. It takes into account the abstraction properties provided by the type system and the module system of the language: local values, abstracted definitions of types, parametric polymorphism.

The main goal is to be sound in a strong way: if an OCaml module is considered to be correct by the analyzer, then one will have the guarantee that no OCaml code interacting with this module can trigger uncaught exceptions coming from the code of this module.

To model the behaviour of a module, we rely on game semantics, where programs formed by a module and a client of the module are modeled as calls-and-returns interactions. A module is then represented as a transition system that generates traces representing the interaction with any possible client. This transition system is directly generated from the module’s code (implementation and signature), using an operational semantics. We have implemented such an interactive semantics for a large fragment of OCaml, which we are using to develop our prototype analyzer.

Earlier this year we formed the ocaml-wasm organisation to join efforts between two WebAssembly (Wasm) targeting compiler projects for OCaml:

• wasocaml – a prototype backend for OCaml’s native code compiler based on the Flambda-intermediate representation
• wasm_of_ocaml – a fork of js_of_ocaml that translates OCaml bytecode to Wasm instead of JavaScript

Exciting news: The WebAssembly Garbage Collection (WasmGC) extension which both wasocaml and wasm_of_ocaml depend on is now enabled by default in the latest:

• Chrome 119 release
• Firefox 120 release

wasm_of_ocaml currently requires the following Wasm extensions:

• The GC extension, including functional references and 31-bit integers
• The tail-call extension
• The exception handling extension

As you may have inferred, with the latest release Chrome 119 can now run the output of wasm_of_ocaml out of the box!

For Firefox 120, one needs to enable the Wasm tail call extension first, but this will be remedied in the forthcoming 121 release.

OCaml 5.x code using effect handlers can be compiled in two different ways: one can either enable the CPS transformation from js_of_ocaml or emit code requiring the JavaScript-Promise Integration extension.

Previously, wasm_of_ocaml also required the experimental stringref proposal to convert strings between OCaml and JavaScript. The stringref proposal may however be superseeded by the newer JS-string-builtins proposal, which is being implemented in both Chrome and Firefox. With the JavaScript string situation up in the air, for now wasm_of_ocaml instead uses the TextEncoder~/~TextDecoder API on the JavaScript side to convert JavaScript to or from UTF-8 by writing strings to or reading them from a buffer in Wasm linear memory as an intermediate state.

Since September @vouillon has continued extending and improving wasm_of_ocaml:

• Implemented unboxed float arrays
• Small runtime improvements:
  • Accurate fused multiply–add
  • Improved handling of uncaught exceptions
  • Improved the generated code by reducing the number of casts
• Optimised the JavaScript interface:
  • Generate constant JavaScript strings at compile time
  • Generate JavaScript code for function and method calls
  • Generate JavaScript code for object literals and object property accesses
• Run CI tests after each push:
  • Run js_of_ocaml’s testsuite including effects, and a number of other testsuites testing in particular the~zarith~ JavaScript stubs and the bonsai library
  • Run more Jane Street tests to ensure compatibility
• Updated Eliom to work with wasm_of_ocaml as a stress test. This included:
  • Finding and fixing a number of small bugs in wasm_of_ocaml’s runtime
  • Implementing a few missing features in the runtime, like Json.output and Js.export - Fixed a performance issue with toplevel loops:
    • Our loops included all the code that follows, and the V8 engine used jump offsets to estimate the cost of a loop. So, it could widely overestimate the cost of loops at toplevel and produce optimised top-level code which is run only once. Since this code can be very large, this could take several minutes. Now, the code which is not involved in a loop is moved after the loop if it exceeds a size threshold.

In addition, @vouillon also contributed with a couple of broader Wasm community improvements:

• Making binaryen more effective at cancelling out a boxing followed by an unboxing: WebAssembly/binaryen#5952 and
• Improved the performance of Wasm exceptions in V8, the JavaScript / Wasm engine used by Chrome. After a number of optimisations, like using a cache to find which Wasm code correponds to a code pointer, they are about a order of magnitude faster. As a consequence the Boyer benchmark runs about 5 time faster.

Finally, he prepared and gave two presentations about wasm_of_ocaml:

• A general wasm_of_ocaml talk at the WebAssembly Community Group meeting on October 12. In particular, he presented a number of performance measurements.
• A talk on implementing OCaml effect handlers in Wasm at the WebAssembly’s Stack Subgroup