[Caml-list] Attn: Development Editor, Latest OCaml Weekly News

Alan Schmitt
2024-04-16T12:00:13Z

[-- Attachment #1: Type: text/plain, Size: 19887 bytes --]

Hello

Here is the latest OCaml Weekly News, for the week of April 09 to 16,
2024.

Table of Contents
─────────────────

Melange 2024 Progress Update
Ppxlib maintenance summary
The OCaml community is signed up for Outreachy!
opam 2.2.0~beta2
Gospel 0.3.0
Fred 0.1.0 - Federal Reserve Economic Data API
OCANNL 0.3.1: a from-scratch deep learning (i.e. dense tensor optimization) framework
Other OCaml News
Old CWN

Melange 2024 Progress Update
════════════════════════════

Archive:
<https://discuss.ocaml.org/t/blog-melange-2024-progress-update/14457/1>

Antonio Nuno Monteiro announced
───────────────────────────────

we recently shared what we've been up to around Melange & ecosystem in
a blog post which you can find here:

<https://melange.re/blog/posts/whats-2024-brought-to-melange-so-far>

I hope you find the above informative. Looking forward to your
thoughts.

Ppxlib maintenance summary
══════════════════════════

Archive:
<https://discuss.ocaml.org/t/ppxlib-maintenance-summary/14458/1>

Nathan Rebours announced
────────────────────────

I recently started working on ppxlib again thanks to the OCaml
Software Foundation and wanted to report back to the community all the
work their funding made possible so far along with the plan for the
next steps.

Know that OCSF is only funding me part time on this and that I'm open
to more OCaml freelance work!

Summary of the work
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

◊ Improved error reporting

Ppxlib has an `-embed-error' flag that is most useful to merlin as it
allows it to always get an AST out of the driver and allows it to keep
operating normally when a ppx raises a located exception (as in raised
with `Location.raise_errorf') as it would always get an AST out of the
driver run.

The problem with this mode was that it didn't try to recover from such
exceptions and would stop applying transformations. The error was
properly reported by merlin and it still had a valid AST to work with
but none of the potential errors from subsequent rewriting or code gen
would be reported. This lead to a tedious workflow where the user
would fix one error, save the file, see a new error reported by
merlin, fix it, save, and so on.

There was a series of PRs by @Burnleydev1 long pending review that
were fixing this by collecting all such exceptions while keeping on
the rewriting phases using the last valid AST or node.

I reviewed and fixed those PRs ([#447], [#453] and [#457]) and worked
on a couple fixes and improvements to error reporting related to this
work.

[#447] <https://github.com/ocaml-ppx/ppxlib/pull/447>

[#453] <https://github.com/ocaml-ppx/ppxlib/pull/463>

[#457] <https://github.com/ocaml-ppx/ppxlib/pull/457>

◊ Driver mode for dune

Dune has ongoing internal work to be able to use ppx in
development. Since it cannot depend on ppxlib or any ppx at build
time, their solution relies on using ppxlib and ppx normally in
development but using already preprocessed copies of source files that
require rewriting for bootstrapping, making their opam build ppx-free.

They require a special driver mode that writes to the output file only
if any actual rewriting happened.

I worked on a first prototype of this using a pre-existing hook called
for each generated AST node.

◊ 5.2 compat on trunk-support

The main task since I started working on ppxlib was the 5.2
support. As OCaml 5.2 is coming out soon and ppxlib is a central piece
in the OCaml ecosystem, it's important that ppxlib has a compatible
version available for testing out the new compiler. Without it, a lot
of opam packages can't be built with the alpha and beta releases of
the compiler because of their ppx dependencies.

ppxlib has a `trunk-support' branch that is meant to be kept up to
date for such occasions. While it already contained the AST migration
from/to the 5.2 version, it was out of sync with the main branch of
ppxlib.

I rebased the 5.2 relevant parts of the branch on top of our main
branch to be able to cut a first preview version with 5.2 compat and
fixed a couple of bugs and quirks in the code base that prevented the
test suite from running properly with the 5.2 support.

After the first release of the preview version we discovered a series
of bugs with the help of @kit-ty-kate, @octachron and @anmonteiro. The
most important among those being:
• A bug in the round trip migration of the new `Pexp_function' node
from 5.2 that was causing compilation errors when the function's
return type was coerced.
• The new syntax for `ocaml.ppx.context' was causing driver crashes
when reading some binary ASTs. I wrote a migration for those
attributes that fixed the issue.
• The driver was silently relying on the compiler to re-open files to
display source quotation when reporting located exceptions.
Since this was removed from the compiler in 5.2, I fixed the driver to
properly set `Location.input_lexbuf' and re-enable source-quotation.

◊ ppx_deriving maintenance

`ppx_deriving' is quite a central piece of the ppx ecosystem,
especially for the set of standard derivers it provides
(pretty-printers, equality and comparison functions, etc.).

The project was lacking maintainers with enough time to review an
important PR migrating those standard derivers to ppxlib. This is
important because it makes those quite broadly used derivers better
integrated with ppxlib features and improves both performances and
error reporting for their users as they are now applied as part of the
main ppxlib driver AST traversal.

I reviewed the PR and cleaned up the repo a bit to attempt a release,
something that has not happened for 3 years for this package.

The initial release failed for two reasons:
• `ppx_deriving.show''s deriver accepts an argument that specifies how
the implementation should behave without impacting the
function signature. In the ppxlib port we did not register this
argument for the signature deriver since it had no impact on the
generated code there. It turns out at least one opam package used the
argument in an `.mli' file so we added it for compatibility as
`ppx_deriving' duplicated the set of arguments for implementation and
interfaces
• `ppx_deriving' used to automatically register extensions for each
derivers that can be used to inline the derived function for a
given type in an expression. We preserved this in the ppxlib port but
it caused a conflict with `ppx_let'. This conflict should be resolved
on `ppx_let''s side as they were declaring an extension named `map'
without any possible prefix such as `ppx_let.map' which is the
recommended way. Using a prefix allows the user to disambiguate if
several ppx declare an extension with the same base name.

We had to cancel the release to fix those issues before attempting
again.

◊ General maintenance

There was also some regular maintenance such as improving our homemade
expect test runner to be able to better run our tests across all
supported compiler versions, reviewing all pending PRs, upgrading
ocamlformat and cutting a release of ppxlib with the latest features.

Next steps
╌╌╌╌╌╌╌╌╌╌

Along with the general maintenance of the project there are two things
that would greatly reduce the maintenance burden on ppxlib and would
improve the state of the ecosystem for the community that we would
like to work on.

◊ Upstreaming Astlib to the compiler

This has been in the works for quite a long time but the previous
maintainers haven't had the chance to see it through.

The plan is to upstream a small part of ppxlib into the compiler to
ease the release process for new compilers.

This small library should contain the minimal subset of stable API
ppxlib needs to properly function and, most importantly, the AST
migrations from the current version of the compiler to the previous
version and back. The idea is that trunk would then provide the
migration to the latest released version at all time and ppxlib would
be able to use those if it does not natively supports them yet. That
would make testing the compiler on opam work without requiring a
special release of ppxlib and would also ease the migration writing
process as they would be written at the time of the AST change by the
person who modified it and therefore that is most qualified to do so.

Indeed the migration writing process is time consuming at the moment
because it is done by ppxlib maintainers when the next ocaml releasing
is closing in and requires us to dive into changes we are not
necessarily familiar with.

During the compiler release process, ppxlib would incorporate those
new migration to the entire set of migrations it supports, allowing it
to be compatible with the "new" trunk until the next release.

The core of this work is to formally write down this process and start
the discussion with the compiler team. Once we agree on a plan, I
don't expect there is a lot of code to write for this as all of it
already lives inside ppxlib.

◊ Refining the release process for ppxlib with an AST bump

Part of the ppxlib contract is that all ppx-es written with ppxlib
must use the same version of the AST chosen by ppxlib. This allows for
better performances and semantics of rewriting. This version is
usually the latest supported version as it is required for ppx-es to
support all the new language features. It sometimes happen that the
internal AST version lags behind if no new features would be
"unlocked" by upgrading the AST, as it has been the case since 4.14.

Every now and then ppxlib has to bump its internal AST though,
potentially breaking its API and therefore a few of its reverse
dependencies.

In the past few years, we decided to build the entire set of reverse
dependencies every time were releasing such a change and to send PRs
to fix revdeps to help keep the ppx ecosystem sane and avoid putting
to much pressure on individual ppx-es maintainers.

We know that we will have to do this for the 5.3 release as it will be
adding effects syntax.

The current workflow relies on building a "duniverse" i.e. some sort
of large dune-project containing ppxlib and a clone of all its reverse
dependencies. This proves quite a challenge as it is often hard to get
everything to build together. An ideal solution would be to rely more
on `opam-ci' for this but in its current state it is not very
reliable.

I'd like to spend some time on improving the process of building
revdeps of ppxlib and submitting PRs and experiment to prepare for the
next bump which will include the 5.2 changes to `Pexp_function' that
we expect to potentially break quite a lot of reverse dependencies.

The OCaml community is signed up for Outreachy!
═══════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/the-ocaml-community-is-signed-up-for-outreachy/13892/18>

Nathan Rebours announced
────────────────────────

I'm a bit late to the party but still wanted to let you know about the
project we submitted with @shonfeder and @dinakajoy.

[ocaml-api-watch] is a fresh project that aims at providing a suite of
tools to help OCaml library maintainers and users deal with changes in
the public API of their libraries or the ones they use. This includes
libraries and CLI tools to detect potentially unwanted breaking
changes before releasing a new version or to determine the version of
a library that introduced a new function.

The goal of the internship is to develop a library and tool pair that
detects changes in the public API of a library, build an internal
representation of them and displays them in a human readable, git
diff-like format.

The application period went really well and we have several strong
candidates. We've been extremely happy to work with all of them and
are looking forward to the internship.

[ocaml-api-watch] <https://github.com/NathanReb/ocaml-api-watch>

opam 2.2.0~beta2
════════════════

Archive: <https://discuss.ocaml.org/t/ann-opam-2-2-0-beta2/14461/1>

Kate announced
──────────────

We're very excited to announce this second beta for opam 2.2.0.

What’s new in this beta?
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

• *Windows support*: this beta introduces a bunch of changes necessary
to be able to make the default opam-repository support Windows out
of the box. We will write a dedicated blog post very soon on this,
once we have finalised the PR/branch that you can test.
• *opam-repository scalability*: The current draft resolution
resulting from the discussion in [ocaml/opam-repository#23789]
includes the removal of packages from opam-repository. Currently
opam can misbehave (in particular on macOS) when exposed to file
deletions in repositories due to the use of the system `patch`
command. To fix this, as a stop gap, after many trials and errors,
opam now warns when GNU patch is not detected on your system. These
changes will make their way to the upcoming opam 2.1.6, in a few
weeks.
• Many *regression fixes*, *performance* and general *improvements*

:open_book: You can read our [blog post] for more information about
these changes and more, and for even more details you can take a look
at the [release note] or the [changelog].

[ocaml/opam-repository#23789]
<https://github.com/ocaml/opam-repository/issues/23789>

[blog post] <https://opam.ocaml.org/blog/opam-2-2-0-beta2/>

[release note] <https://github.com/ocaml/opam/releases/tag/2.2.0-beta2>

[changelog] <https://github.com/ocaml/opam/blob/2.2.0-beta2/CHANGES>

How to upgrade
╌╌╌╌╌╌╌╌╌╌╌╌╌╌

• For Windows we will write a dedicated blog post to show how to
install and use opam on Windows very soon. Stay tuned!
• On Unix-like systems, to upgrade, simply run:
┌────
│ bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh) --version 2.2.0~beta2"
└────

We're on the home stretch for the final release of opam 2.2.0, so feel
free to report any issue you encounter on our [bug-tracker].

Happy hacking, *<> <> The opam team <> <>* :camel:

[bug-tracker] <https://github.com/ocaml/opam/issues>

Gospel 0.3.0
════════════

Archive: <https://discuss.ocaml.org/t/ann-gospel-0-3-0/14480/1>

Nicolas Osborne announced
─────────────────────────

Hi! We are very happy to announce the release of `gospel.0.3.0'

Gospel is a tool-agnostic behavioural specification language for
OCaml. It allows you to write strongly typed contract-based
specifications for your OCaml libraries (for a reasonable subset of
OCaml). Gospel’s syntax has been designed to be easy to learn for an
OCaml programmer. You can access the documentation [here]

Apart from some bug fixes, this release brings two main improvements:

• Make the type-checker save type information in a `.gospel' file
• Make the `with' keyword necessary when declaring type invariants

Beware that `ortac.0.1.0' is not compatible with this version, please
use Ortac development version from the git repository until the next
Ortac release.

[here] <https://ocaml-gospel.github.io/gospel/>

Fred 0.1.0 - Federal Reserve Economic Data API
══════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/ann-fred-0-1-0-federal-reserve-economic-data-api/14489/1>

Geoffrey Borough announced
──────────────────────────

Hi folks howdy! I just release the first version of Fred, a library
for the Federal Reserve Economic Data API binding. I made this to
facilitate one of my personal project but I hope others would find
this library useful in some way.

Source code: <https://github.com/gborough/fred>

Documentation: <https://gborough.github.io/fred/fred/fred/index.html>

Opam repo publish on the way

OCANNL 0.3.1: a from-scratch deep learning (i.e. dense tensor optimization) framework
═════════════════════════════════════════════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/ann-ocannl-0-3-1-a-from-scratch-deep-learning-i-e-dense-tensor-optimization-framework/14492/1>

Lukasz Stafiniak announced
──────────────────────────

Hi! I'm happy to announce release 0.3.1 of OCANNL, a tensor
optimization framework with:

• concise notation via PPXs,
• powerful shape inference,
• backpropagation (first-order automatic differentiation),
• low-level backends – currently only one, CPU via `gccjit`, but Cuda
is on the horizon.

OCANNL is sponsored by [Ahrefs]. ([Ahrefs website].)

[ahrefs/ocannl: OCANNL: OCaml Compiles Algorithms for Neural Networks
Learning (github.com)]

I am not submitting it to Opam yet as OCANNL is insufficiently
documented at the moment. I welcome your feedback if you decide to
take a look!

[Ahrefs] <https://ocaml.org/success-stories/peta-byte-scale-web-crawler>

[Ahrefs website] <https://ahrefs.com/>

[ahrefs/ocannl: OCANNL: OCaml Compiles Algorithms for Neural Networks
Learning (github.com)] <https://github.com/ahrefs/ocannl>

Other OCaml News
════════════════

From the ocaml.org blog
───────────────────────

Here are links from many OCaml blogs aggregated at [the ocaml.org
blog].

• [Multicore Testing Tools: DSCheck Pt 2]

[the ocaml.org blog] <https://ocaml.org/blog/>

[Multicore Testing Tools: DSCheck Pt 2]
<https://tarides.com/blog/2024-04-10-multicore-testing-tools-dscheck-pt-2>

Old CWN
═══════

If you happen to miss a CWN, you can [send me a message] and I'll mail
it to you, or go take a look at [the archive] or the [RSS feed of the
archives].

If you also wish to receive it every week by mail, you may subscribe
to the [caml-list].

[Alan Schmitt]

[send me a message] <mailto:alan.schmitt@polytechnique.org>

[the archive] <https://alan.petitepomme.net/cwn/>

[RSS feed of the archives] <https://alan.petitepomme.net/cwn/cwn.rss>

[caml-list] <https://sympa.inria.fr/sympa/info/caml-list>

[Alan Schmitt] <https://alan.petitepomme.net/>

[-- Attachment #2: Type: text/html, Size: 31912 bytes --]