[Caml-list] [ANN] Jane Street's ppx rewriters

[Caml-list] [ANN] Jane Street's ppx rewriters

[Jeremie Dimino]

I am pleased to announce the initial release of the Jane Street's ppx
rewirters. They are now all in opam, and the next release of Core will
be using them.

They are using a specific framework that we developed to ensure our
rewriters are robust against misuses of new syntaxes. However all
packages export regular ppx executables or ppx_deriving plugins, than
can be used the standard way with ocamlfind.

This is the list of ppx rewriters that we released so far:

- ppx_assert (replaces pa_test)
- ppx_bench
- ppx_bin_prot
- ppx_compare
- ppx_csv_conv
- ppx_custom_printf
- ppx_enumerate
- ppx_fail
- ppx_fields_conv
- ppx_here
- ppx_inline_test (replaces pa_ounit)
- ppx_pipebang
- ppx_sexp_conv
- ppx_sexp_value (replaces pa_structural_sexp)
- ppx_typerep_conv
- ppx_variants_conv
- ppx_xml_conv

In addition we are releasing a few libraries:

# ppx_core

This is our PPX standard library. Amongst other things it contains:

- Various open recursion classes to traverse the OCaml AST:
  Ast_traverse.{map,fold,fold_map,...}. This work extends the
  Ast_mapper.mapper record of the OCaml compiler libraries. However it
  uses names that are closer to the ones in parsetree.mli, so that is
  is easier to infer them by just knowing the parsetree. We found that
  was helpful when writing ppx related code.
- A framework for attributes and extension points. It deals with
  namespacing and make it easier to describe the expected payload.
  When used in combination with ppx_driver, it provides helpful error
  messages for users.
- Helpers for building and matching the OCaml AST. The building part
  is similar to the Ast_helper module of OCaml but with a different
  naming convention and no use of a global variable: a functor is provided
  to factorize the [~loc] argument.

# ppx_driver

This is what we use to handle all of our code rewriting. Instead of
running one process per code transformation, we crunch them all into
one executable. This has several advantages:

- probably speed, although this hasn't been benchmarked
- a simple way to see the transformed code: no need to build a complex
  command line, just run "./ppx file.ml"
- some helpers to debug code transformations

But more importantly, since the driver has knowledge of all
transformations and all supported attributes and extensions, it can:

- check that no attribute accidentally ends up being dead code
- give helpful error messages for unused attributes

For instance:

    # type t = int [@@derivin sexp]
    Attribute `derivin' was not used
    Hint: Did you mean deriving?

    # type t = int [@deriving sexp]
    Attribute `deriving' was not used
    Hint: `deriving' is available for type declarations, type extensions
    and extension constructors but is used here in the context of a core type.
    Did you put it at the wrong level?"

    # type t = { x : int [@default 42] } [@@deriving sexp_of];;
    Attribute `default' was not used

    # let%test ("name" [@foo]) = f x = 42;;
    Attributes not allowed here

This is important to us as we don't want people to waste time because
of a misspelled/misplaced attribute. Also when developing rewriters,
we found that it was quite easy to accidentally drop attributes, and
sometimes it is hard to avoid. ppx_driver notifies the user when this
happens.

# ppx_type_conv

This is the base library for all our type-driven code generators. It
is similar to ppx_deriving, but with similar requirements as
ppx_driver. Unless used with ppx_driver, ppx_type_conv will just use
ppx_deriving.

# ppx_optcomp

This is what we use for when we need conditional compilation. It's has
a cpp-like syntax, which is nice for this kind of things, and works at
the lexer level, which make it possible to use it in pattern matchings
for instance.

It is used by the lexer fronted of ppx_driver. So if used as a -pp
instead of -ppx, a ppx driver will have optcomp enabled.

-- 
Jeremie Dimino, for the Core team

Re: [Caml-list] [ANN] Jane Street's ppx rewriters

[Gabriel Scherer]

Do you plan provide any backward-compatibility guarantees for
ppx-core, or do you plan the helpers there to mirror changes to
upstream parsetree?

(I think there would be a place in the ppx ecosystem for a library
that allows users to write rewriters on their current OCaml version,
and know that they will still compile and be useable to preprocess
OCaml code for future versions. This may require an API with explicit
versioning.)

On Mon, Oct 5, 2015 at 3:40 PM, Jeremie Dimino <jdimino@janestreet.com> wrote:
> I am pleased to announce the initial release of the Jane Street's ppx
> rewirters. They are now all in opam, and the next release of Core will
> be using them.
>
> They are using a specific framework that we developed to ensure our
> rewriters are robust against misuses of new syntaxes. However all
> packages export regular ppx executables or ppx_deriving plugins, than
> can be used the standard way with ocamlfind.
>
> This is the list of ppx rewriters that we released so far:
>
> - ppx_assert (replaces pa_test)
> - ppx_bench
> - ppx_bin_prot
> - ppx_compare
> - ppx_csv_conv
> - ppx_custom_printf
> - ppx_enumerate
> - ppx_fail
> - ppx_fields_conv
> - ppx_here
> - ppx_inline_test (replaces pa_ounit)
> - ppx_pipebang
> - ppx_sexp_conv
> - ppx_sexp_value (replaces pa_structural_sexp)
> - ppx_typerep_conv
> - ppx_variants_conv
> - ppx_xml_conv
>
> In addition we are releasing a few libraries:
>
> # ppx_core
>
> This is our PPX standard library. Amongst other things it contains:
>
> - Various open recursion classes to traverse the OCaml AST:
>   Ast_traverse.{map,fold,fold_map,...}. This work extends the
>   Ast_mapper.mapper record of the OCaml compiler libraries. However it
>   uses names that are closer to the ones in parsetree.mli, so that is
>   is easier to infer them by just knowing the parsetree. We found that
>   was helpful when writing ppx related code.
> - A framework for attributes and extension points. It deals with
>   namespacing and make it easier to describe the expected payload.
>   When used in combination with ppx_driver, it provides helpful error
>   messages for users.
> - Helpers for building and matching the OCaml AST. The building part
>   is similar to the Ast_helper module of OCaml but with a different
>   naming convention and no use of a global variable: a functor is provided
>   to factorize the [~loc] argument.
>
> # ppx_driver
>
> This is what we use to handle all of our code rewriting. Instead of
> running one process per code transformation, we crunch them all into
> one executable. This has several advantages:
>
> - probably speed, although this hasn't been benchmarked
> - a simple way to see the transformed code: no need to build a complex
>   command line, just run "./ppx file.ml"
> - some helpers to debug code transformations
>
> But more importantly, since the driver has knowledge of all
> transformations and all supported attributes and extensions, it can:
>
> - check that no attribute accidentally ends up being dead code
> - give helpful error messages for unused attributes
>
> For instance:
>
>     # type t = int [@@derivin sexp]
>     Attribute `derivin' was not used
>     Hint: Did you mean deriving?
>
>     # type t = int [@deriving sexp]
>     Attribute `deriving' was not used
>     Hint: `deriving' is available for type declarations, type extensions
>     and extension constructors but is used here in the context of a core type.
>     Did you put it at the wrong level?"
>
>     # type t = { x : int [@default 42] } [@@deriving sexp_of];;
>     Attribute `default' was not used
>
>     # let%test ("name" [@foo]) = f x = 42;;
>     Attributes not allowed here
>
> This is important to us as we don't want people to waste time because
> of a misspelled/misplaced attribute. Also when developing rewriters,
> we found that it was quite easy to accidentally drop attributes, and
> sometimes it is hard to avoid. ppx_driver notifies the user when this
> happens.
>
> # ppx_type_conv
>
> This is the base library for all our type-driven code generators. It
> is similar to ppx_deriving, but with similar requirements as
> ppx_driver. Unless used with ppx_driver, ppx_type_conv will just use
> ppx_deriving.
>
> # ppx_optcomp
>
> This is what we use for when we need conditional compilation. It's has
> a cpp-like syntax, which is nice for this kind of things, and works at
> the lexer level, which make it possible to use it in pattern matchings
> for instance.
>
> It is used by the lexer fronted of ppx_driver. So if used as a -pp
> instead of -ppx, a ppx driver will have optcomp enabled.
>
> --
> Jeremie Dimino, for the Core team
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] [ANN] Jane Street's ppx rewriters

[Jeremie Dimino]

On Mon, Oct 5, 2015 at 2:48 PM, Gabriel Scherer
<gabriel.scherer@gmail.com> wrote:
> Do you plan provide any backward-compatibility guarantees for
> ppx-core, or do you plan the helpers there to mirror changes to
> upstream parsetree?

Good question. Currently most of the helpers are auto-generated from
the signature of Parsetree. Using names that closely matches the ones
in parsetree.mli. These will surely change as Parsetree changes. Then
we have additional helpers with different names for common use, these
will probably be backward compatible. Going forward we'll add more
such helpers to improve compatibility between versions of the
parsetree.

You can see the interface for these helpers here:

  https://github.com/janestreet/ppx_core/blob/master/src/ast_builder.ml
  https://github.com/janestreet/ppx_core/blob/master/src/ast_builder_intf.ml

And for patterns:

  https://github.com/janestreet/ppx_core/blob/master/src/ast_pattern.ml
  https://github.com/janestreet/ppx_core/blob/master/src/ast_pattern_intf.ml

> (I think there would be a place in the ppx ecosystem for a library
> that allows users to write rewriters on their current OCaml version,
> and know that they will still compile and be useable to preprocess
> OCaml code for future versions. This may require an API with explicit
> versioning.)

Sure. But having backward-compatible helpers for building the AST
is not enough as your patterns will still break. We have [Ast_pattern]
which is often enough for matching the payload of attributes and
extensions, but it doesn't conveniently replace all pattern matchings.

-- 
Jeremie