OCaml Weekly News

Previous Week<= /a> Up Next Week

Hello

Here is the latest OCaml Weekly News, for the week of April 09 to 16, 2024.

Melange 2024 Progress Update

Archive: https://discuss.ocaml.org/t/blog-melange-2024-progress-upda= te/14457/1

Antonio Nuno Monteiro announced

we recently shared what we've been up to around Melange & ecosystem in = a blog post which you can find here:

https://melange.re/blog/posts/whats-2024-brought-to-melange-so-far

I hope you find the above informative. Looking forward to your thoughts.

Ppxlib maintenance summary

Archive: https://discuss.ocaml.org/t/ppxlib-maintenance-summary/14458/1

Nathan Rebours announced

I recently started working on ppxlib again thanks to the OCaml Software Fou= ndation and wanted to report back to the community all the work their funding made possible so far along with the plan for the next st= eps.

Know that OCSF is only funding me part time on this and that I'm open to mo= re OCaml freelance work!

Summary of the work

Improved error reporting

Ppxlib has an -embed-error flag that is most useful to merlin = as it allows it to always get an AST out of the driver and allows it to keep operating normally when a ppx raises a located exception (as in rai= sed with Location.raise_errorf) as it would always get an AST out of the driver run.

The problem with this mode was that it didn't try to recover from such exce= ptions and would stop applying transformations. The error was properly reported by merlin and it still had a valid AST to work with b= ut none of the potential errors from subsequent rewriting or code gen would be reported. This lead to a tedious workflow where the user would fix one error, save th= e file, see a new error reported by merlin, fix it, save, and so on.

There was a series of PRs by @Burnleydev1 long pending review that were fix= ing this by collecting all such exceptions while keeping on the rewriting phases using the last valid AST or node.

I reviewed and fixed those PRs (#447, #453 and #457) and worked on= a couple fixes and improvements to error reporting related to this work.
Driver mode for dune

Dune has ongoing internal work to be able to use ppx in development. Since = it cannot depend on ppxlib or any ppx at build time, their solution relies on using ppxlib and ppx normally in development but using a= lready preprocessed copies of source files that require rewriting for bootstrapping, making their opam build ppx-free.

They require a special driver mode that writes to the output file only if a= ny actual rewriting happened.

I worked on a first prototype of this using a pre-existing hook called for = each generated AST node.
5.2 compat on trunk-support
The main task since I started working on ppxlib was the 5.2 support. As OCa= ml 5.2 is coming out soon and ppxlib is a central piece in the OCaml ecosystem, it's important that ppxlib has a compatible version av= ailable for testing out the new compiler. Without it, a lot of opam packages can't be built with the alpha and beta releases of the= compiler because of their ppx dependencies.

ppxlib has a trunk-support branch that is meant to be kept up = to date for such occasions. While it already contained the AST migration from/to the 5.2 version, it was out of sync with the main branch = of ppxlib.

I rebased the 5.2 relevant parts of the branch on top of our main branch to= be able to cut a first preview version with 5.2 compat and fixed a couple of bugs and quirks in the code base that prevented the t= est suite from running properly with the 5.2 support.

After the first release of the preview version we discovered a series of bu= gs with the help of @kit-ty-kate, @octachron and @anmonteiro. The most important among those being:
- A bug in the round trip migration of the new Pexp_function= node from 5.2 that was causing compilation errors when the function's
return type was coerced.
- The new syntax for ocaml.ppx.context was causing driver cr= ashes when reading some binary ASTs. I wrote a migration for those
attributes that fixed the issue.
- The driver was silently relying on the compiler to re-open files to dis= play source quotation when reporting located exceptions.
Since this was removed from the compiler in 5.2, I fixed the driver to prop= erly set Location.input_lexbuf and re-enable source-quotation.
ppx_deriving maintenance
ppx_deriving is quite a central piece of the ppx ecosystem, es= pecially for the set of standard derivers it provides (pretty-printers, equality and comparison functions, etc.).

The project was lacking maintainers with enough time to review an important= PR migrating those standard derivers to ppxlib. This is important because it makes those quite broadly used derivers better integra= ted with ppxlib features and improves both performances and error reporting for their users as they are now applied as part of the = main ppxlib driver AST traversal.

I reviewed the PR and cleaned up the repo a bit to attempt a release, somet= hing that has not happened for 3 years for this package.

The initial release failed for two reasons:
- ppx_deriving.show's deriver accepts an argument that speci= fies how the implementation should behave without impacting the
function signature. In the ppxlib port we did not register this argument fo= r the signature deriver since it had no impact on the generated code there. It turns out at least one opam package used the argum= ent in an .mli file so we added it for compatibility as ppx_deriving duplicated the set of arguments for implementatio= n and interfaces
- ppx_deriving used to automatically register extensions for= each derivers that can be used to inline the derived function for a
given type in an expression. We preserved this in the ppxlib port but it ca= used a conflict with ppx_let. This conflict should be resolved on ppx_let's side as they were declaring an extension= named map without any possible prefix such as ppx_let.m= ap which is the recommended way. Using a prefix allows the user to disambiguate if s= everal ppx declare an extension with the same base name.

We had to cancel the release to fix those issues before attempting again.
General maintenance

There was also some regular maintenance such as improving our homemade expe= ct test runner to be able to better run our tests across all supported compiler versions, reviewing all pending PRs, upgrading ocaml= format and cutting a release of ppxlib with the latest features.

Next steps

Along with the general maintenance of the project there are two things that= would greatly reduce the maintenance burden on ppxlib and would improve the state of the ecosystem for the community that we would li= ke to work on.

Upstreaming Astlib to the compiler

This has been in the works for quite a long time but the previous maintaine= rs haven't had the chance to see it through.

The plan is to upstream a small part of ppxlib into the compiler to ease th= e release process for new compilers.

This small library should contain the minimal subset of stable API ppxlib n= eeds to properly function and, most importantly, the AST migrations from the current version of the compiler to the previous version= and back. The idea is that trunk would then provide the migration to the latest released version at all time and ppxlib would be ab= le to use those if it does not natively supports them yet. That would make testing the compiler on opam work without requiring a speci= al release of ppxlib and would also ease the migration writing process as they would be written at the time of the AST change by t= he person who modified it and therefore that is most qualified to do so.

Indeed the migration writing process is time consuming at the moment becaus= e it is done by ppxlib maintainers when the next ocaml releasing is closing in and requires us to dive into changes we are not nec= essarily familiar with.

During the compiler release process, ppxlib would incorporate those new mig= ration to the entire set of migrations it supports, allowing it to be compatible with the "new" trunk until the next release.

The core of this work is to formally write down this process and start the = discussion with the compiler team. Once we agree on a plan, I don't expect there is a lot of code to write for this as all of it = already lives inside ppxlib.
Refining the release process for ppxlib with a= n AST bump

Part of the ppxlib contract is that all ppx-es written with ppxlib must use= the same version of the AST chosen by ppxlib. This allows for better performances and semantics of rewriting. This version is usually= the latest supported version as it is required for ppx-es to support all the new language features. It sometimes happen that the inte= rnal AST version lags behind if no new features would be "unlocked" by upgrading the AST, as it has been the case since 4.1= 4.

Every now and then ppxlib has to bump its internal AST though, potentially = breaking its API and therefore a few of its reverse dependencies.

In the past few years, we decided to build the entire set of reverse depend= encies every time were releasing such a change and to send PRs to fix revdeps to help keep the ppx ecosystem sane and avoid putting to= much pressure on individual ppx-es maintainers.

We know that we will have to do this for the 5.3 release as it will be addi= ng effects syntax.

The current workflow relies on building a "duniverse" i.e. some sort of lar= ge dune-project containing ppxlib and a clone of all its reverse dependencies. This proves quite a challenge as it is often hard to = get everything to build together. An ideal solution would be to rely more on opam-ci for this bu= t in its current state it is not very reliable.

I'd like to spend some time on improving the process of building revdeps of= ppxlib and submitting PRs and experiment to prepare for the next bump which will include the 5.2 changes to Pexp_function that we expect to potentially break quite a lot of reverse dependencies.



The OCaml community is signed up for Outreachy!


Archive: https://discuss.ocaml.org/t/the-ocaml-communi=
ty-is-signed-up-for-outreachy/13892/18



Nathan Rebours announced


I'm a bit late to the party but still wanted to let you know about the proj=
ect we submitted with @shonfeder and @dinakajoy.



ocaml-api-watch is a fresh project that aims at providing a suite of tools to help
OCaml library maintainers and users deal with changes in the public API of =
their libraries or the ones they use. This includes
libraries and CLI tools to detect potentially unwanted breaking changes bef=
ore releasing a new version or to determine the version of
a library that introduced a new function.



The goal of the internship is to develop a library and tool pair that detec=
ts changes in the public API of a library, build an
internal representation of them and displays them in a human readable, git =
diff-like format.



The application period went really well and we have several strong candidat=
es. We've been extremely happy to work with all of them
and are looking forward to the internship.





opam 2.2.0~beta2


Archive: https://discuss.ocaml.org/t/ann-opam-2-2-0-beta2/14461/1



Kate announced


We're very excited to announce this second beta for opam 2.2.0.



What=E2=80=99s new in this beta?


Windows support: this beta introduces a bunch of changes necessa=
ry to be able to make the default opam-repository support Windows out of th=
e box. We will write a dedicated blog post very soon on this, once we have =
finalised the PR/branch that you can test.
opam-repository scalability: The current draft resolution result=
ing from the discussion in ocaml/opam-repository#23789 includes the removal of p=
ackages from opam-repository. Currently opam can misbehave (in particular o=
n macOS) when exposed to file deletions in repositories due to the use of t=
he system `patch` command. To fix this, as a stop gap, after many trials an=
d errors, opam now warns when GNU patch is not detected on your system. The=
se changes will make their way to the upcoming opam 2.1.6, in a few weeks.<=
/li>
Many regression fixes, performance and general improve=
ments



:open_book: You can read our blog post for more information about these changes and
more, and for even more details you can take a look at the release note or
the chan=
gelog.




How to upgrade


For Windows we will write a dedicated blog post to show how to install =
and use opam on Windows very soon. Stay tuned!

On Unix-like systems, to upgrade, simply run:

bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/ma=
ster/shell/install.sh) --version 2.2.0~beta2"




We're on the home stretch for the final release of opam 2.2.0, so feel free=
 to report any issue you encounter on our
bug-tracker.



Happy hacking,
<> <> The opam team <> <> :camel:






Gospel 0.3.0


Archive: h=
ttps://discuss.ocaml.org/t/ann-gospel-0-3-0/14480/1



Nicolas Osborne announced


Hi! We are very happy to announce the release of gospel.0.3.0



Gospel is a tool-agnostic behavioural specification language for OCaml. It =
allows you to write strongly typed contract-based
specifications for your OCaml libraries (for a reasonable subset of OCaml).=
 Gospel=E2=80=99s syntax has been designed to be easy to learn for
an OCaml programmer. You can access the documentation here



Apart from some bug fixes, this release brings two main improvements:



Make the type-checker save type information in a .gospel f=
ile
Make the with keyword necessary when declaring type invari=
ants



Beware that ortac.0.1.0 is not compatible with this version, p=
lease use Ortac development version from the git repository until the
next Ortac release.





Fred 0.1.0 - Federal Reserve Economic Data API


Archive: https://discuss.ocaml.org/t/ann-fred-0-1-0-f=
ederal-reserve-economic-data-api/14489/1



Geoffrey Borough announced


Hi folks howdy! I just release the first version of Fred, a library for the=
 Federal Reserve Economic Data API binding. I made this to
facilitate one of my personal project but I hope others would find this lib=
rary useful in some way.



Source code: https://github.co=
m/gborough/fred



Documentation: https://gborough.github.io/fred/fred/fred/index.html



Opam repo publish on the way





OCANNL 0.3.1: a from-scratch deep learning (i.e. dense tensor =
optimization) framework


Archive: https:/=
/discuss.ocaml.org/t/ann-ocannl-0-3-1-a-from-scratch-deep-learning-i-e-dens=
e-tensor-optimization-framework/14492/1



Lukasz Stafiniak announced


Hi! I'm happy to announce release 0.3.1 of OCANNL, a tensor optimization fr=
amework with:



concise notation via PPXs,
powerful shape inference,
backpropagation (first-order automatic differentiation),
low-level backends – currently only one, CPU via `gccjit`, but C=
uda is on the horizon.



OCANNL is sponsored by Ahrefs. (Ahrefs
website.)



ahrefs/ocannl: OCANNL: OCaml C=
ompiles Algorithms for Neural Networks Learning (github.com)



I am not submitting it to Opam yet as OCANNL is insufficiently documented a=
t the moment. I welcome your feedback if you decide to
take a look!





Other OCaml News



From the ocaml.org blog


Here are links from many OCaml blogs aggregated at the ocaml.org blog.



Multicore Testing Tools: DSCheck Pt 2





Old CWN


If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a loo=
k at the archive or the <=
a href=3D"https://alan.petitepomme.net/cwn/cwn.rss">RSS feed of the archive=
s.



If you also wish to receive it every week by mail, you may subscribe to the=
 caml-list.




Alan Schmitt

OCaml Weekly News

Table of Contents

Melange 2024 Progress Update

Antonio Nuno Monteiro announced

Ppxlib maintenance summary

Nathan Rebours announced

Summary of the work

Next steps

The OCaml community is signed up for Outreachy!

Nathan Rebours announced

opam 2.2.0~beta2

Kate announced

What=E2=80=99s new in this beta?

How to upgrade

Gospel 0.3.0

Nicolas Osborne announced

Fred 0.1.0 - Federal Reserve Economic Data API

Geoffrey Borough announced

OCANNL 0.3.1: a from-scratch deep learning (i.e. dense tensor = optimization) framework

Lukasz Stafiniak announced

Other OCaml News

From the ocaml.org blog

Old CWN