OCaml Weekly News

Previous Week Up Next Week

Hello

Here is the latest OCaml Weekly News, for the week of October 08 to 15, 2024.

grep_cmt: structural search of OCaml code
Mutaml 0.1
ocaml-activitypub
Ortac/QCheck-STM 0.4.0 Dynamic formal verification beyond one system under test
Openbsd 1.0
Compsort - reorder files in archives for improved compression
Dune dev meeting
Other OCaml News
Old CWN

grep_cmt: structural search of OCaml code

Archive: https://discuss.ocaml.org/t/ann-poc-grep-cmt-structural-search-of-ocaml-code/15411/1

Nicolas Ojeda Bar announced

As mentioned in a previous post:

https://discuss.ocaml.org/t/ann-2nd-editor-tooling-dev-meeting-25th-of-july/14953/5?u=nojb

I had promised to post back here when we had made the source code for the "structural grep" tool that I presented, public.

This is now done:

https://github.com/LexiFi/grep_cmt

We added a [POC] marker to this post, because the code is not really ready for public consumption (it is rough around the edges and may not work in all circumstances). Our hope is to publicize the approach and perhaps motivate interested hackers to take the code and develop it further into a proper tool.

Mutaml 0.1

Archive: https://discuss.ocaml.org/t/ann-mutaml-0-1/12639/2

Jan Midtgaard announced

I'm happy to share news of the Mutaml 0.3 release! :tada: https://github.com/jmid/mutaml/releases/tag/0.3

Together with the recent 0.2 release, this brings Mutaml up to speed with recent ppxlib releases and addresses a few issues reported by brave, early users:

Avoid mutations in attribute parameters #29
Avoid polymorphic equality which is incompatible with Core #30
Add support for ppxlib.0.28 and above #27
Avoid triggering 2 mutations of a pattern incl. a when-clause causing a redundant sub-pattern warning #22, #23

Happy testing! :smiley:

ocaml-activitypub

Archive: https://discuss.ocaml.org/t/ann-ocaml-activitypub/15420/1

Zoggy announced

I'm glad to announce a first release of activitypub* packages, implementing (well, trying to implement some flavor of) both server-to-server and client-to-server activitypub protocols.

Documentation is available from the web site.

The activitypub_server package installs TAPS, an experimental server, handling some common activities. Accounts hosted by this server can at least follow and be followed by mastodon instances, and post and receive activities (Create, Announce, Like, Undo, …).

The library of the activitypub_client package can be used by a client application to post and receive activities to and from a server (though it was only tested with TAPS). See a simple example here.

A GUI client (based on Stk is installed by the activitypub_gui package. It requires a client configuration file as described here. You can drop IRIs/URLs of an actor in the window to open a tab and be able to follow this actor. The GUI also allows to create and post notes with attachments. This client is still very experimental and will be developed more in the future.

EDIT: the package should be available soon in opam.

Ortac/QCheck-STM 0.4.0 Dynamic formal verification beyond one system under test

Archive: https://discuss.ocaml.org/t/ann-ortac-qcheck-stm-0-4-0-dynamic-formal-verification-beyond-one-system-under-test/15427/1

Nicolas Osborne announced

I'm very pleased to announce this exciting new release of ortac-qcheck-stm.0.4.0!

This new release brings some exciting new features, mostly the result of Nikolaus Huber's contributions! Thank you Nik!

Ortac/QCheck-STM is a test generator based on the QCheck-STM model-based testing framework and the Gospel specification language for OCaml.

You can find the project on this repo and install the released packages via opam.

It is also encourage to install ortac-dune to avoid having to write too much dune boilerplate.

In particular, this release extend Ortac/QCheck-STM so that the generated tests will include functions that can take multiple System-Under-Tests as argument and/or that can return one. So now, if we write Gospel specifications for append-like functions, Ortac/QCheck-STM will include them in the generated tests!

Happy testing!

Openbsd 1.0

Archive: https://discuss.ocaml.org/t/ann-openbsd-1-0/15434/1

Sebastien Marie announced

I would like to announce a new (somehow niche) package Openbsd, which provides bindings for some specifics OpenBSD syscalls pledge(2) and unveil(2).

These syscalls lets the kernel OS to know what the running processus is expected to do, and so it is possible to restrict a processus to do only filesystem or only network or only pure computation…

The package is designed to be installable on any platform and provides simple method to check if Pledge or Unveil are supported. This way, it is possible to easily write portable code using the package, as it could be a turned on "no-operation" on Windows or Linux hosts (or provides alternative code path for sandboxing).

Homepage : https://codeberg.org/semarie/ocaml-openbsd/
License : ISC
Documented Interface : lib/openbsd.mli

Examples

let open Openbsd in
if Pledge.supported then
  Pledge.promises "stdio rpath"

let open Openbsd in
if Unveil.supported then (
  Unveil.add "./lib" "r";
  Unveil.add "./logs" "rwc";
  Unveil.lock ())

Compsort - reorder files in archives for improved compression

Archive: https://discuss.ocaml.org/t/ann-compsort-reorder-files-in-archives-for-improved-compression/15436/1

adrien announced

I'm happy to announce the first release of compsort, a tool to reorder files in an archive for better compression. It works by grouping files according to a distance that is computed between every file pair. You can install it with opam install compsort (requires ocaml 5.2 for parallelism).

Website with more details and examples in README.md, plus source: https://gitlab.com/adrien-n/compsort/

The goal is not new but, AFAIK, the approach is. I am very interested in prior or related art!

Results

Compsort achieves improvements that would typically require larger compression windows and therefore more memory. The improvements are only a few percents but in the domain of compression, a few percents is actually a lot.

With xz compression, a Ubuntu initrd on my machine is reduced by more than 11.5%, while the best achievable improvement is 12.7% (the reordering gives 90% of the best result). Similarly, the tree of linux-firmware.git can be compressed 6% better, while the best achievable improvement is 9.4% (the reordering gives 63% of the best result).

Visualizations

In order to better explain what it does, I quite like the visualizations I have so far (there will be better ones in the future), where the value of the pixel at (x,y) indicates how similar are files x and y.

Before:

After reordering:

Files that are very different from others are all packed at the end and there's also an isolated cluster of files together similar but different from everything else. One can also see that the distinct row and column pattern from the first picture has disappeared: it indicated that every 15 or so files in that region were similar and were separated by disimilar files but that they're now grouped.

While there are certainly improvements possible, results are good. It's a case where one might wonder why results are so good considering all the approximations that took place.

[1] Most of the algorithms/libraries I've tried to use rely on having an actual proper distance function which I don't have

Future work

I'll try to improve the distance function. Currently it does some steps of compression algorithms to detect redundancies but maybe reusing a compression library would give better results if it can be made fast enough (lz4 is borderline but it has tiny dictionaries unfortunately).

Clustering could be better as the current algorithm is very basic (it collects files that are 90% similar, then 80% similar, then 70%, …). I tried several algorithms but I don't have a good-enough distance function (for instance the triangular inequality probably doesn't hold) and results were worse.

All this will benefit from better visualizations and I'd like to have interactive plots that can be hovered on with the mouse to get the distance value and full file name.

Oh and code isn't always pretty as it went through a lot of experimental stages and low-level tweaks to improve performance.

Dune dev meeting

Archive: https://discuss.ocaml.org/t/ann-dune-dev-meeting/14994/13

Etienne Marais announced

We will hold our regular Dune dev meeting tomorrow, on Wednesday, October, 16th at 10:00 CET. As usual, the session will be one hour long.

Whether you are a maintainer, a regular contributor, a new joiner or just curious, you are welcome to join: these discussions are opened! The goal of these meetings is to provide a place to discuss the ongoing work together and synchronise between the Dune developers !

Agenda

The agenda is available on the meeting dedicated page. Feel free to ask if you want to add more items in it.

Other OCaml News

From the ocaml.org blog

Here are links from many OCaml blogs aggregated at the ocaml.org blog.

Old CWN

If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe to the caml-list.

Alan Schmitt