OCaml Weekly News

What?

When?

What’s the Fmt?

Call for presentations?

Ppxlib and compiler releases: How it works today

• Ppxlib internal AST
  

  Before ppxlib there was ocaml-migrate-parsetree. OMP had the advantage of providing a stable API for ppx authors. Each ppx would select a fixed version of the AST and be implemented as a full AST rewrite, i.e. a structure -> structure or signature -> signature function.

  This had the advantage of making ppx-es forward compatible as omp maintainers would add support for new compiler releases in the form of a new module containing the AST types for this version and migration functions to convert to/from the types matching the previous compiler version.

  OMP also came with a ppx driver, i.e. a program in charge of applying a set of ppx-es on a given AST or source file and spit out the final preprocessed AST for the compiler. The driver was responsible for migrating the AST from the compiler's version to the one used by a ppx. Because each ppx could require a different AST version, it also potentially had to migrate the AST transformed by a ppx before it passing it on to the next one.

  This had a few disadvantages though:

  1. poor performance as the AST was traversed and migrated (i.e. copied) several times through the course of a single driver run.
  2. transformations semantic issues: the order in which ppx-es were applied was uncertain or rather tied to the set of ppx-es version used. That meant that updating one ppx could change its "turn" and result in a different AST returned by the driver. This also did not allow ppx-es to interact together reliably.

  ppxlib aimed at fixing those issues by forcing ppx-es to agree on the AST version to use. ppxlib provides its own, fixed AST version that ppx-es have to use. Its driver handles the migration to/from the current compiler and provides a smooth API to write transformations as rewriting rules. The driver then handles the AST rewrite by recursively applying those rules in the right places in a single AST traversal.

• Support for new compilers
  

  Support for new compilers comes in two stages.

  • Build and preprocess old code with new compiler
    

    This is the most basic support, that is making sure that one can still build and preprocess its code using the newest compiler, provided they don't use any of the new language features.

    To do this, we add the new AST types and migration functions, just as OMP used to do. This does not allow new features in the code because those cannot be represented with the old AST types and the migration would fail (This was also an existing limitation of OMP).

    This is usually released early on, when the compiler is still in beta and is a non breaking change, all reverse dependencies still build with this new version.

  • Support new language features
    

    To support new language features, we bump the AST used by ppxlib. This means new features don't have to be migrated anymore and are therefore supported.

    This does change types that are exposed as part of ppxlib's API and can cause breakage in reverse dependencies, depending on which part of the AST were modified and which part each individual ppx uses explicitly.

    We provide tools that can help make ppx code more robust as they allow matching over and producing AST nodes without explicitly referencing the types themselves: metaquot, Ast_builder or Ast_pattern for instance. That's not always enough though and eventually, those ppx-es have to be updated to be compatible with the latest version of ppxlib.

    As was the case for the 5.2 AST bump, when we release such a ppxlib version, we send PRs to help maintainers of our opam reverse dependencies update and carefully add upper bounds to the versions that aren't compatible anymore.

    This worked pretty well for a few years as the AST was relatively stable and the parts that were modified were not directly used by a lot of ppx-es.

    A problem with this approach is that even though we can help maintainers go through the update, we cannot release packages in their stead which means that unmaintained ppx-es aren't compatible anymore no matter how much effort we put into easing the upgrade. It is also often the case that not all ppx-es have a compatible release straight away and this results in a transition period with the opam universe split mentioned in the introduction.

Proposed approach for 5.3 onward

• Limitations
  

  Encoding new features into extension points is not always easy, only specific parts of the AST can be replaced by an extension point. To keep things under control and prevent ppx-es from generating inconsistent nodes, all new features will always be migrated into an extension point. That means that if the impacted node cannot directly be encoded that way, we will encode the first suitable parent node. In some scenarios, that can climb up the AST types quite significantly, potentially all the way to the structure_item~/~signature_item. This means that new features won't be equal when it comes to how easy it is to use them in conjunction with some ppx-es. It's important to keep in mind that this is still a net improvement as it was previously not possible to use them together at all.

  Similarly, providing a nice API to allow building and destructing encoded new features will vastly depend on the features themselves and how entangled they are with new AST types. We will likely not always expose such builder/destructor pairs and might only add some of them if the demand is high enough.

  It is also part of the reason why we will probably still bump our AST at some points in time even if much less frequently than we have in the past. When that eventually happens, we will be able to maintain the previous major versions for quite a while as this will just be a matter of adding our newest migrations there as well.

• Effect syntax example
  

  OCaml 5.3 introduced the following syntax:

  match f () with
  | v -> Complete v
  | effect (Xchg msg), k ->
    ...
  

  This special effect pattern is represented in the 5.3 AST with the Ppat_effect variant:

  | Ppat_effect of pattern * pattern
  

  We cannot represent this in the 5.2 AST and previously, any attempt at migrating such a node down would have failed. With this new approach we instead migrate it to something along those lines:

  [%ppxlib.migration.ppat_effect? (Xchg msg, k)]
  

  and the upward migration knows to translate this to the right Ppat_effect node. This migration needs to work without context outside the extension so that any ppx that would unknowingly copy such a node elsewhere in the AST would not cause an uninterpreted extension error later on during the compilation.

  If this is passed down to an existing ppx as part of its payload and it tries to interpret it, it should fail as it won't know what to do with such an extension.

  Note that ppx authors should never rely on the actual extension point encoding, we reserve ourselves the right to change that encoding as part of minor or patch releases of ppxlib. Such nodes should be left untouched or dealt with using the stable API described below.

  Now if a ppx author needs to add explicit support for effects they will be able to use something like:

  val ppat_effect : loc: location -> pattern -> pattern -> pattern
  

  from Ast_builder to generate such a node. Of course if your ppx generates an effect pattern with an older compiler, this will lead to a compile error as the extension won't be translated unless migrated back up. Authors will have to be mindful of this and properly document when/how they'll generate newer nodes and eventually restrict their ppx to the right range of compilers.

  These will likely come with a "destruct" version in Ast_pattern. For the effect pattern it should look like:

  val ppat_effect : (pattern * pattern, 'a, 'b) t -> (pattern, 'a, 'b) t
  

• Bivariant type parameter example
  

  This example is probably a bit of a stretch as it is a very niche syntax change and is highly unlikely to actually be used in the wild, but it makes a good example of a feature that is hard to encode.

  In OCaml 5.4, a new variant was added to the Asttypes.variance type: Bivariant. The variance type is used in the AST to describe how a type parameter behaves relative to the type itself. This can be manually annotated for each parameter when writing a type declaration or a class.

  The Bivariant case is a bit of a special one as a parameter can only be Bivariant (i.e. covariant AND contravariant) with the type if it does not actually appear in the concrete type definition, that is in cases such as:

  type 'a t = A
  

  For reasons that we won't expand upon here, 5.4 introduced the following syntax to allow one to explicit annotate a parameter as bivariant:

  type +-'a t
  

  The problem is that the variance cannot be replaced directly by an extension point, see the type type_declaration for instance:

  and type_declaration =
      {
       ptype_name: string loc;
       ptype_params: (core_type * (variance * injectivity)) list;
                                   ^^^^^^^^
        (** [('a1,...'an) t] *)
       ptype_cstrs: (core_type * core_type * Location.t) list;
        (** [... constraint T1=T1'  ... constraint Tn=Tn'] *)
       ptype_kind: type_kind;
       ptype_private: private_flag;  (** for [= private ...] *)
       ptype_manifest: core_type option;  (** represents [= T] *)
       ptype_attributes: attributes;  (** [... [\@\@id1] [\@\@id2]] *)
       ptype_loc: Location.t;
      }
  

  In this example we have to encode the entire parent node of the type declaration as an extension point.

  This means that it spreads in quite a few places, type_declaration can be found in structure_items, signature_items and inside some module_type nodes as well.

  Given there's very little to no use for this syntax, we won't be providing any function to build or destruct such nodes initially.

Features