OCaml Weekly News
Hello
Here is the latest OCaml Weekly News, for the week of February 20 to 27, 2024.
Table of Contents
- Feedback / Help Wanted: Upcoming OCaml.org Cookbook Feature
- The OCaml community is signed up for Outreachy!
- Upcoming Caqti Release - TLS, Packaging, and minor breakage
- Discussions on the future of the opam repository
- dream-html 3.0.0
- Temporary increase in opam CI response times
- A Versatile OCaml Library for Git Interaction - Seeking Community Feedback
- Ocsigen: summary of recent releases
- fuseau 0.1
- How do we want to present OCaml to the World on OCaml.org?
- MirageOS hack retreat 2024 edition (April 22nd - 28th)
- Recruitement on the Catala project (French)
- Js_of_ocaml 5.7
- Eio-trace 0.1 and Eio 0.15
- Other OCaml News
- Old CWN
Feedback / Help Wanted: Upcoming OCaml.org Cookbook Feature
Archive: https://discuss.ocaml.org/t/feedback-help-wanted-upcoming-ocaml-org-cookbook-feature/14127/1
Sabine Schmaltz announced
Hi all,
we're adding an "OCaml Cookbook" feature to OCaml.org where the community can share recipes (small code snippets) for common tasks. These recipes can use packages from the OCaml ecosystem.
A PR for a prototype is up at https://github.com/ocaml/ocaml.org/pull/1839 and can be explored on
https://staging.ocaml.org/cookbook
I am tempted to take inspiration in spirit primarily from the Rust Cookbook (https://rust-lang-nursery.github.io/rust-cookbook/intro.html), since this is focused entirely on tasks that may be required for shipping applications to production.
However, people also mentioned that they like the format of Go by Example (https://gobyexample.com/sorting) a lot.
The current prototype of the cookbook on staging.ocaml.org/cookbook combines both of these ideas.
Please comment any feedback, ideas, etc. Either here or directly on the PR at https://github.com/ocaml/ocaml.org/pull/1839!
The final version may be simpler - for example, we might find that no recipes need more than a single file, in which case we'll remove the capability to have multiple files on a recipe.
Also, I will definitely need your help in completing the list of tasks and creating all those recipes. You can make suggestions on the PR to add tasks. :heart:
The OCaml community is signed up for Outreachy!
Paul-Elliot announced
Here are the notes from yesterday's meeting! We discussed a lot of things, so they do not go into as much details as what was said. Thanks to all the participants, and to @pitag for organizing, running and taking the notes!
What are the currently proposed projects?
- An accessible interface to read a diff. Mentors : @EmileTrotignon and @Juloo, @panglesd.
- Highlight search terms in sherlodoc results. Mentors : @EmileTrotignon and @Juloo, @panglesd.
- Create a standalone repository as a basis for the exercise section on ocaml.org into a standalone project. Mentors : @cuihtlauac and @shakthimaan
- A diffing tool for cmi artefacts to help maintainers of library detect API changes. @NathanReb, @shonfeder.
What is it like to be an Outreachy mentor?
- Being approachable
- Ping the intern at least once a day
- Expects to spend an hour a day
- Depends on the period. At the beginning, more time-intense. Later, the intern will be more independent.
- Project scoping.: Project "management"
- Preparing for the contribution period
- Find good issues
- Contribution period
- Depending on the issues you found, this can be very time-intense or not.
- Internship
- Check-ins
- Pair-programming
- Meetings. Good idea: Use a pad to explain things via drawings.
- Try out different things
- Try daily stand-ups for a while
- Try to leave the intern alone for a while
- Try to connect the interns
Tips for mentoring
- Have your expectations right from the beginning. A very good internship achieves two things: The project advances a lot, and the intern grows a lot. But also if only one of those two things happens, the internship is very successful.
- Scope out a first very self-contained easy task that you think could be done in one week. Then, expect that the intern will take 1.5 month on it.
- Put a "requirement" on the applicants to start contributing
- Adapt to the intern
- Avoid writing code for the intern
- E.g. in the review, never make a code suggestion, but phrase out the change you have in mind
- Related: Avoid "dragging along" the intern.
Current bottlenecks
Currently, the bottleneck is enough mentors, not finding more projects.
Criteria for a project / funding
- Projects
Any project is a good project for this, assuming you have the mentor capacities. It doesn't matter which part of the community is from or who the maintainer is. The only thing that matters is that the project is open-source and it's in OCaml. A few tips for a good project:
- Be in control of the repos the project is on
- The project needs to be non-controversial among the repo maintainers.
- Don't have too high expectations on what can be done.
Anyone can submit a project.
- Be in control of the repos the project is on
- Funding
Any project as described above is welcome. Any mentor for such a project can get funding. Any intern working on such a project would get funding as well.
- Objective of funding
The objective of the funding is to increase OCaml's outreachy and diversity. It's not to promote projects from a certain part of the ecosystem.
Where do mentors communicate
So far, we've used Slack and have invited mentors who weren't on there. We've discussed that the Outreachy Zulip might be an even better place. We'll try that this round.
Upcoming Caqti Release - TLS, Packaging, and minor breakage
Archive: https://discuss.ocaml.org/t/upcoming-caqti-release-tls-packaging-and-minor-breakage/13926/7
"Petter A. Urkedal announced
Caqti 2.1.1 has now been released to OPAM. The following packages were included:
| Package | 2.1.1 |
|---|---|
| caqti | yes |
| caqti-async | yes |
| caqti-eio | yes |
| caqti-lwt | yes |
| caqti-mirage | yes |
| caqti-tls-async | no, omitted for now |
| caqti-tls-eio | no, omitted for now |
| caqti-tls-lwt | no, omitted for now |
| caqti-tls | yes |
| caqti-driver-mariadb | yes |
| caqti-driver-pgx | yes |
| caqti-driver-postgresql | yes |
| caqti-driver-sqlite3 | yes |
| caqti-dynload | no, unchanged |
| caqti-type-calendar | no, unchanged |
Details can be found in the release notes for version 2.1.0 since 2.1.1 include build and packaging fixes only.
Discussions on the future of the opam repository
dream-html 3.0.0
Yawar Amin announced
I've just released 3.0.1 with a bugfix for ARIA attribute names, please upgrade if you are using ARIA: https://ocaml.org/p/dream-html/3.0.1
Temporary increase in opam CI response times
Anil Madhavapeddy announced
You may all notice a temporary increase over the next few days in the opam repository CI response times when submitting opam packages. This is because Mark Elvers, @samoht and I are attempting to rebalance the cluster that drives these jobs towards using less power, and more renewable power.
Most of the machines are currently hosted at the Cambridge Computer Lab (not renewable currently), and we are moving more work into Scaleway's Paris datacentre (renewable), but the capacity is far lower. The results show a nice sharp drop in resource utilisation, but the job response times are far slower on the lower capacity.
We'll bring the Cambridge machines back into the pool later this week if we can't get Scaleway to be solely enough capacity, but I hope you all agree that it's also important that we minimise resource usage everywhere we can. (and in general, the non-x86 machines will remain at Cambridge as there are no alternative hosting options for POWER or RISC-V currently).
A Versatile OCaml Library for Git Interaction - Seeking Community Feedback
Mathieu Barbin announced
Dear OCaml community,
I would like to talk about a project I've been working on: Vcs, a versatile OCaml library for Git interaction. The library provides a direct-style API for interacting with Git repositories and is designed as an "interface", or "virtual" library
with the actual implementation provided by other component dynamically dispatched at runtime (similar to
Eio vs Eio_main). This design aims high flexibility and adaptability to different use cases.
The Vcs library is designed to be backend-agnostic and concurrency-runtime independent. It's compatible with both Eio and OCaml Stdlib runtimes.
My focus on the backend implementation has been pragmatic as of yet. I've concentrated on implementing a provider called
git-cli that wraps the Git CLI, running it as an external process, interpreting its exit status and parsing its output. This approach has allowed me to focus on the core functionality and design of the Vcs library. I plan to explore the feasibility
of supporting, in a similar fashion, luv and possibly
miou as separate future work (still via the
git-cli provider).
The project is currently in the draft stage in a private repository, and I am gradually working through the process of open-sourcing the dependencies for this project. The API it supports is very incomplete, and I am extending incrementally as I need more.
While the code isn't published yet, I've written a public README that outlines the project's architecture, design principles, and motivation. You can read it here.
I'm reaching out to the community for feedback on the approach. I'm particularly interested in your thoughts on the use of a provider-based parametric library and the granularity of the provider interfaces via Traits, but I would be interested by other feedback you'd feel like sharing. Please feel free to open issues, or asking questions or contacting me if you are interested by the project.
Additionally, I'm in the process of selecting an appropriate license for the project. I think it is fair to respond to that "We as a community cannot give you legal advice on that matter, you should contact with a lawyer". On the other hand, if either you or someone you know have spent a lot of times thinking about the subject of LICENSEs for the OCaml ecosystem, it would be of loss for me not to talk to you. I'll be grateful if you do reach out.
Thank you in advance for your time and feedback. I look forward to engaging with the community on this project.
Best, Mathieu
Ocsigen: summary of recent releases
Vincent Balat announced
Ocsigen Start 6.2.0 was released a few days ago. It simplifies the code of the demo application by removing first-class modules and making the code more standard.
fuseau 0.1
Simon Cruanes announced
Dear all,
It brings me a smile to announce that fuseau 0.1 is in the process of being released on opam. Fuseau ("fuh-zo", french for spindle) is yet another lightweight fiber library for OCaml 5.
This is early days, as emphasized in the 0.1 version number. If you give it a try, expect some bugs, rough edges in the API, and changes in the future. It also doesn't have great docs.
rationale and overview
With that out of the way, I'd like to give a rationale for this new library. The primary goal of Fuseau is to be simple while providing a modern, nice, comfortable API that relies on OCaml 5's effects to avoid monadic hell. As such, fibers, the lightweight threads that are cooperatively scheduled by Fuseau, can natively use all usual control structures (loops, exceptions with proper backtraces, etc.).
Of all the libraries in this crop of new OCaml 5-based schedulers, Fuseau is probably the least ambitious. In particular, it does not:
- attempt to use capabilities for more secure programming
- use domains in any way (it's purely single threaded)
- have a notion of quota or fairness
- insist on having its own IO event loop.
What Fuseau does is provide fibers, cooperative channels, structured concurrency, sleep, cancellation[^1], fiber local storage, and the ability for the scheduler to run alongside an event loop.
The most compelling use for Fuseau right now is via fuseau-lwt, which couples Fuseau's fiber scheduler with
Lwt_engine's event loop (which can leverage libev). Using
fuseau-lwt, it's fairly easy to use Lwt libraries and await them from a Fuseau fiber (in fact, the
biggest example written in Fuseau so far uses ezcurl-lwt or
cohttp-lwt-unix to crawl a web site with multiple concurrent connections, and uses
moonpool to parse the web pages in parallel in the background.
structured concurrency
Fuseau's fibers (type: 'a Fiber.t for a fiber returning 'a) form a cancellation tree. When a fiber spawns a child fiber, it has a few ways of doing that but the main one is:
val spawn : ?name:string -> ?propagate_cancel_to_parent:bool -> (unit -> 'a) -> 'a Fiber.t
which creates a new fiber, registers it as a child of the current fiber, and resumes the current fiber. The idea here is that if the parent is cancelled, the child is
always cancelled; if propagate_cancel_to_parent is
true, then the child failing also cancels the parent.
Whatever happens, a fiber always resolves after all its children have returned. What this means is that, when a fiber's main function returns or raises, Fuseau doesn't immediately resolve the fiber; instead it automatically waits for all alive children. If the function fails, the children are cancelled before the wait.
Only when all children have resolved will the parent fiber's result be properly set. If the fiber's function was a success, but a child (with
~propagate_cancel_to_parent:true) fails, then the result switches from success to failure.
This means Fuseau doesn't have switches or graveyards or nurseries. It only has fibers, and the currently active fiber is the nursery for all newly spawn sub-fibers[^2].
cancellation and racing multiple operations
Currently there is no primitive in Fuseau to take multiple fibers and wait for the first one to finish, cancelling the others. This is common in Lwt and I don't like it, because it means we might end up in situations where both fibers actually completed, and end up losing data in the fiber that lost the race.
Instead Fuseau has a select primitive and a notion of atomic event:
module F = Fuseau F.select [ When (F.Chan.ev_receive c1, fun x -> …); When (F.Chan.ev_send c2 y, ignore); When (F.Sleep.ev_timeout 5., fun _ -> failwith "timeout"); ]
This mechanism decouples polling each atomic event in turn, checking if it's able to fire right now. If, in
When (ev, f), the event ev fires when polled and returns
x, this corresponding branch is taken, and the select tail-calls into
f x. For example if F.Chan.ev_receive c1 fires (receiving an item from a channel), then the callback is passed the received value.
If no branch succeeds, then the fiber suspends and registers to each event's
wait function. This means each event is passed a wakeup callback and registers it somewhere (socket readiness, future completion, channel receiver list, etc.).
This mechanism (on which the paint is very fresh!) is extensible (so you can make your own events) and has, imho, cleaner semantics than complex race-and-cancel-the-loser combinators.
domains
As said above, Fuseau doesn't schedule on multiple cores. Part of it is for simplicity reasons (it makes the scheduler lean and simple); part of it is to facilite interoperability with Lwt.
However, some of the types in Fuseau are thread-safe. In particular:
- it's possible to schedule a fiber from another thread (
spawn_from_anywhere) - many functions to access fibers' state, cancel fibers, or callbacks used to resume suspender fibers, are thread-safe. This means it's not that hard to implement a
awaitfunction on some foreign notion of future.
For CPU bound tasks, there is an optional library fuseau.moonpool that has
Fuseau_moonpool.await_fut : 'a Moonpool.Fut.t -> 'a. This can be run from Fuseau to wait for a moonpool computation to be done.
With additional helpers, it's fairly easy to start a background computation from fuseau, and maybe suspend the current fiber until it's done:
val spawn : on:Moonpool.Runner.t -> (unit -> 'a) -> 'a Moonpool.Fut.t (** An alias to {!Moonpool.Fut.spawn} *) val spawn_and_await : on:Moonpool.Runner.t -> (unit -> 'a) -> 'a
closing thoughts
The design space for concurrency libraries in OCaml 5 is wide, and I find it interesting that already a lot of alternatives have emerged, each with its own focus. A niche that Fuseau
could fill is existing projects that have a lot of Lwt code, and want to migrate it progressively to a (imho) nicer style of concurrency. Another niche is for people or teams that want a reasonably simple fiber library to combine
with their event loop (e.g. luv or libev or maybe event a graphical event loop).
[^1]: cancellation is where most bugs and rough edges are likely to live, unsurprisingly.
[^2]: there are alternatives to spawn that start a fiber under a different parent, or under the root fiber, basically similar to
Lwt.async, when it's needed.
How do we want to present OCaml to the World on OCaml.org?
Archive: https://discuss.ocaml.org/t/how-do-we-want-to-present-ocaml-to-the-world-on-ocaml-org/14163/1
Sabine Schmaltz announced
Hi everyone!
We're reworking the Homepage of ocaml.org and the marketing-related pages, such as "Industrial Users", "Academic Users", and "Why OCaml?"
I put a Google form with some questions for anyone who's interested to give input on this here:
https://forms.gle/ph5ABxLsuycKK4GD8
The aim of all this is to see how we can improve the narrative on the Homepage and to make sure we give people all they need to know. Ideally, the reworked pages cover all the points we want to make and all the resources we want to highlight.
Best
Sabine :sparkles:
MirageOS hack retreat 2024 edition (April 22nd - 28th)
Hannes Mehnert announced
please note registration deadline upcoming (March 1st) - there are still some spots available – looking forward to see you there :palm_tree: :dromedary_camel: :technologist:
Recruitement on the Catala project (French)
Denis Merigoux announced
[As the positions opening are in Paris and hosted at the French National Research Center in Computer Science INRIA, the rest of the post is in French]
Bonjour à toutes et à tous,
Le projet Catala recrute ! catala-lang[point]org
- un.e ingénieur.e compilation https://jobs.inria.fr/public/classic/fr/offres/2024-07190
- une ingénieur.e web https://jobs.inria.fr/public/classic/fr/offres/2024-07191
Les postes sont des CDD de deux ans avec le statut d'ingénieur de recherche INRIA. Télétravail possible mais pas full remote. C'est un contrat de la fonction publique d'État donc 38 h 30 par semaine, 7 semaines de congés + 10 jours de RTT. Le salaire est indexé sur la grille des ingénieurs de recherche INRIA qui est fonction de l'expérience (avec un principe de reconstitution de carrière).
Les gros chantiers pour les 2 prochaines années sont l'évolution et la maintenance du langage, le développement d'outils d'aide au développement mais aussi l'industrialisation du dispositif d'explicabilité des algorithmes publics dont une démo est disponible sur code[point]gouv[point]fr/demos/catala/. Partant des recommandations d'un récent rapport de recherche, nous souhaitons faire plusieurs visualisations de la trace d'explication de la décision à destination du grand public, mais aussi en interne à l'administration aux juristes, développeurs et testeurs. La ou le dev web sur le projet définira potentiellement à quoi ressemblera le prochain avis d'imposition ou détail de calcul des prestations sociales.
Le stack web est pour l'instant écrit en Rescript avec un peu d'interaction avec du JS généré par
js_of_ocaml. On aimerait bien garder Rescript plutôt que de passer à Typescript donc si vous pratiquez Rescript, n'hésitez pas à postuler ! Évidemment, la majorité des choses se passe quand même en OCaml :)
Js_of_ocaml 5.7
Hhugo announced
I’m pleased to announce the release of js_of_ocaml 5.7. It should soon be available in opam.
Js_of_ocaml is a compiler from OCaml bytecode to JavaScript. It makes it possible to run pure OCaml programs in JavaScript environment like browsers and Node.js.
Most significant changes:
- Support for OCaml 5.2.0~alpha1
- No longer rely on IIFE for scoping variable inside loops.
See the Changelog for other changes.
Eio-trace 0.1 and Eio 0.15
Thomas Leonard announced
With Eio 0.15 just released, I'd like to announce also the first release of the eio-trace tool, which can be used to visualise the behaviour of any Eio program (no need for a special instrumented build). For eample, here is one of Eio's tutorial examples:
Fiber.both (fun () -> for x = 1 to 3 do traceln "x = %d" x; Fiber.yield () done) (fun () -> for y = 1 to 3 do traceln "y = %d" y; Fiber.yield () done)
dune build ./examples eio-trace run -- ./_build/default/examples/both/main.exe
will open a window showing:
The upper horizontal bar is the initial fiber, and the brackets show Fiber.both creating a second fiber. The green segments show when each fiber is running. Note that the output from
traceln appears in the trace as well as on the console. In the eio-trace window, scrolling with the mouse or touchpad will zoom in or out of the diagram.
This should make it easier to see what's going on, and to find performance problems. For example, we can see that the first output above took quite a bit longer than the others. You can then use e.g.
magic-trace to get very fine-grained traces at this point (and we can see that e.g. the Linux
write system call takes a lot longer the first time).
eio-trace can trace mutliple domains, and shows the various stages of garbage collection. Here we have two domains running; the yellow regions show when the OCaml code is stopped due to garbage collection or polling the OS:
So far, eio-trace has also uncovered two bugs in OCaml:
#12948 and #12897. For example, the trace below shows a process freezing for around 45ms during
Domain.join, due to a problem waiting for OCaml's tick thread to stop:
Particular thanks are due to @patricoferris and @Lortex for their work on Eio tracing.
There have also been lots of other recent improvements to Eio, including:
- Eio.Executor_pool for submitting jobs to a pool of worker domains (thanks to @asemio_sgrondin).
- Eio.Pool for managing pools of resources (like
Lwt_pool). And note that you can now attach resources to a switch from any domain, which makes pools more useful. - Eio.Lazy is like
Stdlib.Lazybut multi-domain safe. - There are lots more operations in
Eio.Path (
stat,rmtree,mkdirs, etc). - FD-passing over Unix-domain sockets.
- The Eio_posix backend now uses file descriptors for directories (Eio_linux already did this), which avoids races with symlinks and is faster. It also means that Eio now works with Capsicum OS sandboxing.
- run_in_systhread now uses a pool of sys-threads for performance.
For full details, see the release notes, and feel free to join the developer meetings every two weeks.
Other OCaml News
From the ocaml.org blog
Here are links from many OCaml blogs aggregated at the ocaml.org blog.
Old CWN
If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.
If you also wish to receive it every week by mail, you may subscribe to the caml-list.