Re: [Caml-list] Camlp4/p5 type reflection [was: OCaml maintenance status / community fork (again)]

[Gabriel Scherer <gabriel.scherer@gmail.com>]

Gerd, you are summing up in a few paragraphs what I tried to say in a few pages.

There are other parts of Camlp4 that I would also welcome:
- the OCaml quotation parsers that reads quoted OCaml expression (and
patterns) and translate them to their ASTs (as an OCaml expression);
this makes generating OCaml code easy
- controlled extension points such as 'type-conv' and 'deriving',
hopefully generalized to most AST nodes; this goes in the direction of
Alain's annotation proposal

> There could be still camlp4 or p5 as a separate add-on,
> but it will be a second-class citizen, and it would be clear that it is
> a bit behind with the newest syntactical features.

It's not a big problem if the second-class extensions have trouble
processing and producing pieces of OCaml AST using the newest
syntactical features (eg. the OCaml quotations don't have sugar for
them). But it's a problem if they simply can't be run on files using
those features, forcing their users to keep older versions of OCaml.
It's not a *new* problem as it already appears at version transitions,
but it also won't go away as long as people use a preprocessor that
insists on understanding the whole AST, whether those tools are first-
or second-class.

On Sun, Dec 11, 2011 at 12:23 PM, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
> Many people are still frustrated with the camlp4/p5 situation. IMHO, we
> should give up on camlp4 inside the distribution, and only implement a
> few of its features in the regular parser:
>
> - Antiquotation syntax (i.e. << >> expressions) because this makes it
>  very easy to incorporate foreign syntax elements. Of course, we
>  would also need the ability to recursively invoke the ocaml parser
>  for parts of the antiquoted expression (this would also open the
>  door for transformations of the AST).
>
> - A simple macro language (ifdef at least), for the cases antiquotations
>  would be too complex.
>
> That means we would give up on arbitrary syntax extensions in the core
> distribution. There could be still camlp4 or p5 as a separate add-on,
> but it will be a second-class citizen, and it would be clear that it is
> a bit behind with the newest syntactical features. Also, it could be
> developed outside the core team.
>
> Gerd
>
>
> Am Sonntag, den 11.12.2011, 11:29 +0100 schrieb Gabriel Scherer:
>> > And Xavier's mail suggests that camlp4 is a maintenance burden for the OCaml team.
>> > Why is it such a bad idea to drop camlp4 out of the distribution, and
>> > just let camlp5 live?
>>
>> First of all, I don't have a strong opinion here: I just voiced
>> doubt. My reasoning for going so goes along two lines of argument:
>>
>> 1. I'm not exactly sure how Camlp4 being in or out of the distribution
>>    will change the maintainance burden. The main maintainance
>>    difficulty with camlp{4,5} is that it needs to evolve its own
>>    parser in parallel with the 'official' one (that's by design), with
>>    changes to the language syntax. You need to change Camlp{4,5} when
>>    Ocaml 3.N+1 introduces a new syntax, or it won't be usable on OCaml
>>    3.N+1 code.
>>
>>    There are already users relying on Camlp{4,5}, and those user
>>    generally wish to use the new, exciting features of the next
>>    version. If they can't, they will complex, regardless of whether
>>    their preprocessor is in or out the distribution. That means that
>>    when the OCaml team is about to release a new version with
>>    syntactic changes, they have to worry about the preprocessor
>>    anyway, or make users unhappy.
>>
>>    So there is a "preprocessor burden" on the OCaml team, independently
>>    of where the code is maintained and located. If the change means
>>    "make it easier to distribute camlp4 fixes without bumping OCaml's
>>    version number", why not. If it means "now we won't care about
>>    Camlp4 state before releasing a new versions", this may mean
>>    a degradation in the life of Camlp4 users. I doubt that's the idea.
>>
>>    Being in or out the distribution also wouldn't change much, I think,
>>    the possibility of external contributions. In my experience Nicolas
>>    Pouillard and now Xavier Clerc have a good track record of
>>    integrating external contributions (I have sent one or two bugfixes
>>    on the tracker), and I'm confident they would be able to work with
>>    Jérémie Dimino if he wished to contribute to camlp4's evolution more
>>    frequently.
>>
>> 2. I suppose -- purely personal guess -- the intention being the
>>    consortium's suggestion is to try to move away slowly from
>>    Camlp{4,5}, towards alternatives such as Alain Frisch's
>>    "annotations" proposal. As I said previously, I would personally
>>    welcome such a move, but I don't see said alternatives released
>>    yet. I haven't had the opportunity to play with alternate tools,
>>    have an idea of how a transition would work out, see if the
>>    documentation is reasonably complete, etc.
>>
>>    It would make more sense, in my opinion, to downplay camlp{4,5}
>>    *once* we have played with alternatives and are confident that they
>>    are mature enough to make a transition.
>>
>>    Please also remember that the consortium members represent
>>    relatively large, well-educated, experienced players in the OCaml
>>    community. It probably wouldn't bother them much if, say,
>>    ocamlbuild, ocamldoc, or ocamldbg where taken out of the official
>>    distribution. They have important tooling in place and would adapt
>>    relatively easily. The end user or OCaml beginner may not adapt to
>>    such changes that easily. Now this is a matter where distributions,
>>    such as Debian, can be of great help, by providing "complete"
>>    packages regardless of what is or isn't in the official
>>    distribution. However, I have handled just enough users reports
>>    wondering why they didn't have "camlp4o" available, or
>>    graphics.cma, or whatever, to know that this can also be a barrier
>>    to use and adoption.
>>
>>
>> Again, no strong opinion. I will welcome any change that strenghten
>> the OCaml language. If you think distributing camlp4 out of the
>> distribution would ease the live of OCaml developpers and maintainers,
>> at no cost to the users, nor complicating the distribution side, then
>> all is good. I just feel that it may be a bit too soon.
>>
>>
>> On Sun, Dec 11, 2011 at 10:04 AM, Stéphane Glondu <steph@glondu.net> wrote:
>> > Le 11/12/2011 00:34, Gabriel Scherer a écrit :
>> >> The original Camlp4 tool was mostly developped by Daniel de
>> >> Rauglaudre.
>> >> [...]
>> >> (I'm thinking of
>> >> eg. Martin Jambon, which had extensive Camlp4 extensions, and the Coq
>> >> team which has user-defined notations using Camlp4 and, huh, I really
>> >> don't want to know the details); basically they didn't upgrade to
>> >> 3.10 -- instead of porting the extensions, as was originally hoped.
>> >> [...]
>> >> Daniel, which apparently did not agree with some of the changes made,
>> >> relatively suddenly restarted developpment of "his" branch of Camlp4,
>> >> taken from the old sources, before refactoring. This was done as
>> >> a separate project, outside the OCaml distribution (apparently Daniel
>> >> and the OCaml team prefer not to work together).
>> >> [...]
>> >> Camlp4 is a piece of devil beauty. It does incredibly clever things,
>> >> and is incredibly complex inside: Daniel is clearly a remarkable
>> >> hacker, but his code is not easy to understand. I know that
>> >> maintaining the whole thing is very hard; and that is the reason why
>> >> Camlp4 tends to have problems to bump from one version to another,
>> >> when non-neglectible syntaxic changes are made to the language.
>> >> [...]
>> >> (And I'm not sure it's a good idea to move Camlp4 out of the
>> >> distribution as long as we don't have a viable alternative proposal
>> >> and some users have started moving to it. Camlp4 will still need to be
>> >> supported by someone anyway, and it needs to evolve in lockstep with
>> >> OCaml language changes.)
>> >
>> > Coq has been adapted to work with both camlp4 and camlp5. In my
>> > experience, Daniel has been much more responsive on camlp5 matters than
>> > the OCaml team on camlp4 matters. And Xavier's mail suggests that camlp4
>> > is a maintenance burden for the OCaml team.
>> >
>> > Why is it such a bad idea to drop camlp4 out of the distribution, and
>> > just let camlp5 live? My feeling is that people using camlp5 care about
>> > it, but people using camlp4 do so just because it's in the official
>> > distribution and wouldn't care switching to camlp5 (or stop using this
>> > kind of syntax extension) if camlp4 was officially deprecated in favor
>> > of camlp5. At worst, someone will start maintaining camlp4 independently.
>> >
>> > Daniel already maintains and develops camlp5. I guess Jérémie would also
>> > volunteer to join him (but he already offered to maintain camlp4
>> > independently). In Debian, there are currently 49 packages depending on
>> > camlp4, and 7 depending on camlp5. That also increases incentive /
>> > possible help for maintaining it.
>> >
>> >
>> > Cheers,
>> >
>> > --
>> > Stéphane
>> >
>>
>>
>
>