Re: On language extensions (was Re: [Caml-list] global record)

Re: On language extensions (was Re: [Caml-list] global record)

[Eric Breck]

> From: Daniel_Bünzli <daniel.buenzli@e...>
> Subject: On language extensions (was Re: [Caml-list] global record)
>
> Contrary to lisp's macros, each syntax extension is syntactically a
> new language. Hence not only do we need to learn a new semantics but
> also a new syntax. If I have to read someone else's code I want to
> read caml code and not caml code augmented with the author's
> syntactic obsessions.
>
> In code readability, there is a trade-off between succinctness and
> syntactic regularity. Camlp4 extensions increase the former but
> decrease the latter. For me camlp4's usage should be limited to real
> domain specific languages (e.g. like coq does) and research, it
> should not be used to increase ocaml's succinctness.

Succinctness is not precisely what I was after so much as locality -  
in this case, having each option declared in a single location - as  
opposed to the type definition in one place, the default value in  
another, the Arg.spec in another, code to print the chosen value of  
each option in another, and, optionally, the de-ref-ication in  
another.  Such locality is a basic principle of software engineering,  
and in this case I don't really know how to achieve it with only a  
library (and not a syntax extension).

I'm not advocating extending the language willy-nilly, but I feel  
that any new library already requires some amount of learning to use  
its particular API, and if a small, local extension to the language  
vastly shrinks (and localizes) code required to interface to the  
library, it seems reasonable to me (IMHO).

To put it another way, I do not agree with the adage that "syntactic  
sugar causes cancer of the semicolon." :)

As far as readability, I tried to choose syntax that would make clear  
to a casual reader what was going on, but clearly I'm blinded by  
having stared at this for awhile now.  As Nicolas Pouillard points  
out, you can always unsugar the code.

Anyway, I've been using variants of this library for half a year or  
so, and I've found it very convenient, so I thought I would offer it  
to others.

> This may be seen as a matter of personal taste: it is true that I
> lean towards syntactic regularity, i.e. less syntactic constructs.
> However there is another argument against using extensions: code
> maintenance. You have no guarantee that (1) an extension will not be
> broken by the next version of the language and (2) that the author
> will continue to maintain it.

I'm not sure why either 1 or 2 is more true for a syntax extension  
than for an arbitrary library, assuming the syntax extension doesn't  
try to make large-scale changes to the grammar.

> P.S. Even for domain specific languages many things can be done in
> pure ocaml by embedding the dsl in ocaml using meta-programming
> techniques.

I'm not sure what you mean here.  I understand meta-programming to  
mean (roughly) generating code programatically.  A solution along  
those lines to my problem (locality of command-line arguments) would  
be something like this: specify the options in some file, write some  
piece of code to process this and produce an ocaml file, which is  
then compiled and linked with the main program.  I actually  
originally did something along these lines, but then decided that  
since this approach also requires defining and processing a new  
syntax (of the option file), I would use camlp4 and integrate that  
syntax into my main program.  That is, camlp4 *is* meta-programming,  
it's just that it (a) integrates the DSL into the main program, and  
(b) hides the intermediate file.

But perhaps I'm misunderstanding your point - if you could clarify  
what you mean, I'd like to know what alternate approaches are.

Finally, I have the sense (perhaps I'm completely wrong) that the  
post I'm responding to was motivated in part by a thought that  
command-line option parsing is somehow not worthy of a domain- 
specific language / syntax extension.  I suppose it is something of a  
"syntactic obsession" with me to find a clear and concise way of  
specifying such options, since I write mostly small, command-line  
programs to which I want to be able to quickly add variant  
functionality.  Clearly, the benefit to any particular program is  
small - when you write enough of them, though, the lifetime savings  
in tearing-my-hair-out is, I think, a win.

best,

-E r i c

(PS: sorry if this doesn't get threaded appropriately, I read the  
digest of the list).

Re: On language extensions (was Re: [Caml-list] global record)

[skaller]

On Wed, 2006-07-19 at 21:12 -0400, Eric Breck wrote:
> > From: Daniel_Bünzli <daniel.buenzli@e...>

>  If I have to read someone else's code I want to
> > read caml code and not caml code augmented with the author's
> > syntactic obsessions.

> Succinctness is not precisely what I was after so much as locality -  

>  Such locality is a basic principle of software engineering,  
> and in this case I don't really know how to achieve it with only a  
> library (and not a syntax extension).

In my opinion, the difficulty here lies in the Ocaml 
programming model.

Precisely as you say above regarding locality .. the syntax
extensions should be localised. In particular, they should
be embedded in the file that uses them -- along with
documentation -- and not specified on the command line.

For example

syntax MyExtensions;;

which could itself be implemented in ocamlp4 perhaps,
and then *that* extension converted to a builtin part
of the language.

Felix implements syntax extensions with preprocessor
statements (which can be provided in a #included file).

Of course the syntax extension should contain documentation!

Having said that, I do agree excessive extensions make
code hard to read .. and probably defeat other tools
such as ocamldoc (which even if applied to expanded 
output may report on the expanded source when one would
prefer reporting on the unexpanded input).

This form of code generation is weak compared with
systems like MetaOcaml.

Similarly, Ocaml should have 

  import List;;
  .. List.fold_left ..

so that you are required to import a separately compiled
module before using it. As a sop to existing code, 

  open List;;

could imply import. grep for either 'open' or 'import'
is then enough to establish dependencies.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: On language extensions (was Re: [Caml-list] global record)

[Stefano Zacchiroli]

On Thu, Jul 20, 2006 at 03:16:18PM +1000, skaller wrote:
> On Wed, 2006-07-19 at 21:12 -0400, Eric Breck wrote:
> >  Such locality is a basic principle of software engineering,  
> > and in this case I don't really know how to achieve it with only a  
> > library (and not a syntax extension).
> 
> In my opinion, the difficulty here lies in the Ocaml 
> programming model.
> 
> Precisely as you say above regarding locality .. the syntax
> extensions should be localised. In particular, they should
> be embedded in the file that uses them -- along with
> documentation -- and not specified on the command line.
> 
> For example
> 
> syntax MyExtensions;;

Fully agreed. Me myself try to enforce locality of syntax extensions to
avoid possible clashes. Unfortunately handling that in large project is
really painful, since you end up in having to differentiate "clusters"
of files which need to be built with different set of camlp4 flags, and
you need to do that in Makefile.

In my mind this is asking for trouble, and the usual solution I pursue
is to avoid using syntax extensions ... a pity.

Cheers.

-- 
Stefano Zacchiroli -*- Computer Science PhD student @ Uny Bologna, Italy
zack@{cs.unibo.it,debian.org,bononia.it} -%- http://www.bononia.it/zack/
If there's any real truth it's that the entire multidimensional infinity
of the Universe is almost certainly being run by a bunch of maniacs. -!-

Re: On language extensions (was Re: [Caml-list] global record)

[Jean-Marie Gaillourdet]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

On 20.07.2006, at 08:29, Stefano Zacchiroli wrote:
> Fully agreed. Me myself try to enforce locality of syntax  
> extensions to
> avoid possible clashes. Unfortunately handling that in large  
> project is
> really painful, since you end up in having to differentiate "clusters"
> of files which need to be built with different set of camlp4 flags,  
> and
> you need to do that in Makefile.

The OCamlMakefile in godi supports syntax extension declarations in  
the first line of ocaml source. Those declaration are wrapped in  
OCaml comments. Then make takes care that every single source file is  
compiled with the appropriate syntax extensions.

- --
   Jean-Marie

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFEv0WONIUNP/I5YOgRAlGIAJ9Fr+vKiTbXCA0AMwvWOOG+wv7j+ACfQOEO
O00L8arLwjNmvXRxp76r4Ic=
=EkZi
-----END PGP SIGNATURE-----

Re: On language extensions (was Re: [Caml-list] global record)

[Bünzli Daniel]

Le 20 juil. 06 à 03:12, Eric Breck a écrit :

> Succinctness is not precisely what I was after so much as locality  
> - in this case, having each option declared in a single location -  
> as opposed to the type definition in one place, the default value  
> in another, the Arg.spec in another, code to print the chosen value  
> of each option in another, and, optionally, the de-ref-ication in  
> another.  Such locality is a basic principle of software  
> engineering, and in this case I don't really know how to achieve it  
> with only a library (and not a syntax extension).

The typechecker (indirectly) gives you that locality, it tells you  
exactly where changes are needed and frankly all this is in the same  
module, maybe even on the same page. If I have to maintain your code  
I'll understand quicker hand made code than your syntax extension  
(especially since I already know how the Arg module works).

> I'm not advocating extending the language willy-nilly, but I feel  
> that any new library already requires some amount of learning to  
> use its particular API, and if a small, local extension to the  
> language vastly shrinks (and localizes) code required to interface  
> to the library, it seems reasonable to me (IMHO).

Api is not new syntax. As a said before when you learn an api you  
only need to understand new semantics.

> As far as readability, I tried to choose syntax that would make  
> clear to a casual reader what was going on, but clearly I'm blinded  
> by having stared at this for awhile now.  As Nicolas Pouillard  
> points out, you can always unsugar the code.

For small examples this can be a solution. But I usually prefer not  
to read machine generated code. Most programmer (I hope) reread their  
code and try to make it readable and elegant, a machine won't.

>> You have no guarantee that (1) an extension will not be
>> broken by the next version of the language and (2) that the author
>> will continue to maintain it.
>
> I'm not sure why either 1 or 2 is more true for a syntax extension  
> than for an arbitrary library, assuming the syntax extension  
> doesn't try to make large-scale changes to the grammar.

I agree on this point for 2 but not on 1. Incompatible language  
changes are rare (inexistant ?) with ocaml.

>> P.S. Even for domain specific languages many things can be done in
>> pure ocaml by embedding the dsl in ocaml using meta-programming
>> techniques.
>
> I'm not sure what you mean here.  I understand meta-programming to  
> mean (roughly) generating code programatically.

I was not specifically talking about your problem here, though I  
could maybe be applied. It was more on using caml to write code that  
should be written in another language (e.g. regexps, sql, etc.),  
instead of writing a horrible language extension that will allow you  
to mix both languages or use caml strings, see [1] for more information.

I'm sorry to say that but I regard (maybe out of ignorance) camlp4 as  
very low level and brittle tool (not to say hack). An obvious proof  
of this to me is the composition problem which is according to  
previous messages is not solved by camlp4.

Best,

Daniel

[1]
@article{967844,
  author = {Conal Elliott and Sigbj\&\#248;rn Finne and Oege De Moor},
  title = {Compiling embedded languages},
  journal = {J. Funct. Program.},
  volume = {13},
  number = {3},
  year = {2003},
  issn = {0956-7968},
  pages = {455--481},
  doi = {http://dx.doi.org/10.1017/S0956796802004574},
  publisher = {Cambridge University Press},
  address = {New York, NY, USA},
  }

Camlp4 mysteries (was Re: On language extensions (was Re: [Caml-list] global record))

[Martin Jambon]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2298 bytes --]

On Thu, 20 Jul 2006, Bünzli Daniel wrote:

> I'm sorry to say that but I regard (maybe out of ignorance) camlp4 as very 
> low level and brittle tool (not to say hack). An obvious proof of this to me 
> is the composition problem which is according to previous messages is not 
> solved by camlp4.

In theory the system of quotations allows you to use as many quotation 
expanders as you want. Quotations are well delimited and do not interfere 
with each other, but look a bit ugly and are restricted to exprs and 
patts.

Otherwise it's possible to define well-disciplined syntax extensions.
For example, if each new syntax construct (new rule) is forced to start 
with a unique, registered keyword and end with "end", then 
different syntax extensions that follow this rule should play well 
together. 
Deleting or rewriting existing rules would of course be forbidden.
And tools like Declare_once [1] should become builtins.

So if you take my favorite syntax extension (micmatch), you would need to 
create a new keyword, let's say "mm":

   match "Hello World!" with
       / "Hello"~ blank+ (alnum+ as user) / -> Some user
     | _ -> None

could become

   mm match "Hello World!" with
      mm "Hello"~ blank+ (alnum+ as user) end -> Some user
    | _ -> None
   end


To the OCaml development team:

Would such a convention make sense?
Could there be special restricted EXTEND statement that only accepts that 
kind of rules, and registers the keywords to avoid unexpected clashes 
between extensions that try to use the same leading keyword?

It would be really nice to have official guidelines on how to develop 
clean syntax extensions, if not automatic enforcement.

That's all I can say, I don't have time to spend playing with the 
new Camlp4 or trying to understand what MetaOCaml is about.
Any official word about the metaprogramming issues in OCaml would be 
appreciated!
I am sorry to say that, but there is way too much mystery floating around 
Camlp4. A web page that presents a roadmap for the development of OCaml 
language and tools including Camlp4 would be much appreciated.


Martin


[1] Declare_once: look for it at http://www.yl.is.s.u-tokyo.ac.jp/~oiwa/en_US.ISO-8859-1/pub/caml/regexp-pp-1.0.0/README.match-regexp

--
Martin Jambon, PhD
http://martin.jambon.free.fr

Re: Camlp4 mysteries (was Re: On language extensions (was Re: [Caml-list] global record))

[Alain Frisch]

Martin Jambon wrote:
> Otherwise it's possible to define well-disciplined syntax extensions.
> For example, if each new syntax construct (new rule) is forced to start
> with a unique, registered keyword and end with "end", then different
> syntax extensions that follow this rule should play well together.

Except that any new keyword can potentially break existing code. You'd
need some other syntactical convention.

> It would be really nice to have official guidelines on how to develop
> clean syntax extensions, if not automatic enforcement.

Do you have concrete examples of extensions that don't play well together?

-- Alain

Re: Camlp4 mysteries (was Re: On language extensions (was Re: [Caml-list] global record))

[Martin Jambon]

On Fri, 21 Jul 2006, Alain Frisch wrote:

> Martin Jambon wrote:
>> Otherwise it's possible to define well-disciplined syntax extensions.
>> For example, if each new syntax construct (new rule) is forced to start
>> with a unique, registered keyword and end with "end", then different
>> syntax extensions that follow this rule should play well together.
>
> Except that any new keyword can potentially break existing code. You'd
> need some other syntactical convention.

Sure, but for future code which uses future syntax extensions, it will be 
OK.

For the current extensions, it is possible to offer alternatives (via a 
-std option or something like that). If I understand correctly, we will have to 
adapt quite a few things for our syntax extensions to support the new 
Camlp4, so while we are at it, adding an alternate standardized syntax 
won't be a problem if it doesn't require too much thinking.

(In addition to that, like Skaller suggested, something like "open syntax 
Mysyntax" would make syntax extensions available like library modules, 
which simplifies makefiles and doesn't force every module to use them. But 
that's about integrating Camlp4 tightly in the OCaml language, which is 
another story)


>> It would be really nice to have official guidelines on how to develop
>> clean syntax extensions, if not automatic enforcement.
>
> Do you have concrete examples of extensions that don't play well together?

Yes, mine :-)

In micmatch I delete and redefine the constructs with bindings: let, let 
in, match, try, function.

The resulting syntax gives the illusion of a nice integration, but this 
is a redefinition of the existing constructs, so if another extension 
does the same, only the one which is loaded last will be usable.

In practice I haven't had any problem like that, but it's a 
possibility.
So this is why someone should decide of a standard, and we will follow it.


Martin

--
Martin Jambon, PhD
http://martin.jambon.free.fr

Re: Camlp4 mysteries (was Re: On language extensions (was Re: [Caml-list] global record))

[skaller]

On Fri, 2006-07-21 at 01:38 +0200, Alain Frisch wrote:
> Martin Jambon wrote:
> > Otherwise it's possible to define well-disciplined syntax extensions.
> > For example, if each new syntax construct (new rule) is forced to start
> > with a unique, registered keyword and end with "end", then different
> > syntax extensions that follow this rule should play well together.
> 
> Except that any new keyword can potentially break existing code. You'd
> need some other syntactical convention.

If you follow the rules I specified, this is not possible.

My rule is: extensions must be enabled in every file that
use them:

#syntax extension_name

Then the extension only affects files deliberately written
using the extension, so there won't be any code breakage
unless the extension itself is upgraded, or, ocaml itself
is upgraded.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: Camlp4 mysteries (was Re: On language extensions (was Re: [Caml-list] global record))

[Martin Jambon]

On Fri, 21 Jul 2006, Alain Frisch wrote:

> Martin Jambon wrote:
>> Otherwise it's possible to define well-disciplined syntax extensions.
>> For example, if each new syntax construct (new rule) is forced to start
>> with a unique, registered keyword and end with "end", then different
>> syntax extensions that follow this rule should play well together.
>
> Except that any new keyword can potentially break existing code. You'd
> need some other syntactical convention.

Do you mean keyword as opposed to LIDENT "mylident"?
Then something like "<:mylident<" but simpler would be better.
Maybe ``mylident ... end


Martin

--
Martin Jambon, PhD
http://martin.jambon.free.fr

Re: [Caml-list] global record

[Eric Breck]

[-- Attachment #1: Type: text/plain, Size: 3028 bytes --]


> Date: Wed, 19 Jul 2006 15:07:29 +0100
> From: Richard Jones <rich@annexia.org>
> Subject: Re: [Caml-list] global record
> To: Andreas Biegert <andreas.biegert@googlemail.com>
> Cc: caml-list@yquem.inria.fr

>This is a bit ugly, but we use it in our Adwords API toolkit:
>
>-------------------------------------------------- stdargs.mli
>val username : string
>val password : string
>val client : string option
>val token : string
>val update : bool
>val verbose : bool
>val args : string list
>
>-------------------------------------------------- stdargs.ml
>let username = ref ""
>let password = ref ""
>let client = ref ""
>let token = ref ""
>let update = ref false
>let verbose = ref false
>let args = ref []
>
>(* followed by some code which tries to read default values
> * from $HOME/.adwordsapidata -- code omitted
> *)
>
>(* parse the command line parameters *)
>let argspec = [
>  "--username", Arg.Set_string username, "Adwords account username.";
>  "--password", Arg.Set_string password, "Adwords account password.";
>  "--client", Arg.Set_string client, "Adwords account client (optional).";
>  "--token", Arg.Set_string token, "Adwords account token.";
>  "--update", Arg.Set update, "Perform updates.";
>  "--verbose", Arg.Set verbose, "Be verbose.";
>]
>
>let anon_fn str = args := str :: !args
>let usage =
>  Sys.executable_name ^ " [--options]"
>
>let () = Arg.parse argspec anon_fn usage
>
>let username = !username
>let password = !password
>let client = if !client = "" then None else Some !client
>let token = !token
>let update = !update
>let verbose = !verbose
>let args = List.rev !args
>
>---
>
>Then the code just stuff like:
>
>  open Stdargs
>
>  if verbose then printf "this is verbose mode\n"

<gauche_self_promotion>

I was also writing code that looked like that, and I got tired of the
duplication, and also the non-locality - each new option requires a
modification in several places.  So the pa_arg syntax extension I just
announced makes this a lot easier.  Except for the reading of values from the
dotfile, all the above code is generated from this declaration (not tested):

open Parseopt

type option myopts = {
username = "" : help = "Adwords account username"; string;
password = "" : help = "Adwords account password";  string;
client: help = "Adwords account client (optional)"; string;
token = "": help = "Adwords account token"; string;
update = false: help = "Perform updates"; action = set; bool;
verbose = false: help = "Be verbose"; action = set; bool;
}
let myopts = {myopts with keyspecs = [Long2]} in (** specify you want field
foo => --foo *)

let options, args = parse_argv myopts in

...

if options.verbose then print "this is verbose mode!\n"

...

You also get a detailed usage and help message and a function to convert the
returned options to a string, plus further customizability.

http://www.cs.cornell.edu/~ebreck/pa_arg

-E r i c

</gauche_self_promotion>

[-- Attachment #2: Type: text/html, Size: 4376 bytes --]

On language extensions (was Re: [Caml-list] global record)

[Daniel Bünzli]

Hello,

I would like to issue a warning about everybody rolling its own  
syntax extension to suit its taste.

Contrary to lisp's macros, each syntax extension is syntactically a  
new language. Hence not only do we need to learn a new semantics but  
also a new syntax. If I have to read someone else's code I want to  
read caml code and not caml code augmented with the author's  
syntactic obsessions.

In code readability, there is a trade-off between succinctness and  
syntactic regularity. Camlp4 extensions increase the former but  
decrease the latter. For me camlp4's usage should be limited to real  
domain specific languages (e.g. like coq does) and research, it  
should not be used to increase ocaml's succinctness.

This may be seen as a matter of personal taste: it is true that I  
lean towards syntactic regularity, i.e. less syntactic constructs.  
However there is another argument against using extensions: code  
maintenance. You have no guarantee that (1) an extension will not be  
broken by the next version of the language and (2) that the author  
will continue to maintain it.

And by the way, are we guaranteed that two arbitrary camlp4  
extensions will compose or may the order of application matter ?

Cheers,

Daniel

P.S. Even for domain specific languages many things can be done in  
pure ocaml by embedding the dsl in ocaml using meta-programming  
techniques.

Re: On language extensions (was Re: [Caml-list] global record)

[Richard Jones]

On Wed, Jul 19, 2006 at 09:41:46PM +0200, Daniel Bünzli wrote:
> And by the way, are we guaranteed that two arbitrary camlp4  
> extensions will compose or may the order of application matter ?

I'm probably wrong, but I think the only way to compose two camlp4
extensions is to combine the source code to both together and compile
a new extension.  Is there some other way?

Rich.

-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Team Notepad - intranets and extranets for business - http://team-notepad.com

Re: On language extensions (was Re: [Caml-list] global record)

[Stefano Zacchiroli]

On Wed, Jul 19, 2006 at 08:57:44PM +0100, Richard Jones wrote:
> I'm probably wrong, but I think the only way to compose two camlp4
> extensions is to combine the source code to both together and compile
> a new extension.  Is there some other way?

Camlp4 extensions are regular .cmo/.cma files which, upon loading,
dynamically extends a grammar. You can "compose" multiple extensions by
passing all of them to your preferred camlp4 pre-processor.

Merely loading multiple extensions of course does not grant you obtain a
grammar able to parse anything at all ...

Cheers.

-- 
Stefano Zacchiroli -*- Computer Science PhD student @ Uny Bologna, Italy
zack@{cs.unibo.it,debian.org,bononia.it} -%- http://www.bononia.it/zack/
If there's any real truth it's that the entire multidimensional infinity
of the Universe is almost certainly being run by a bunch of maniacs. -!-

Re: On language extensions (was Re: [Caml-list] global record)

[Nicolas Pouillard]

On 7/19/06, Daniel Bünzli <daniel.buenzli@epfl.ch> wrote:
> Hello,
>
> I would like to issue a warning about everybody rolling its own
> syntax extension to suit its taste.
>
> Contrary to lisp's macros, each syntax extension is syntactically a
> new language. Hence not only do we need to learn a new semantics but
> also a new syntax. If I have to read someone else's code I want to
> read caml code and not caml code augmented with the author's
> syntactic obsessions.
>
> In code readability, there is a trade-off between succinctness and
> syntactic regularity. Camlp4 extensions increase the former but
> decrease the latter. For me camlp4's usage should be limited to real
> domain specific languages (e.g. like coq does) and research, it
> should not be used to increase ocaml's succinctness.
>

[...]

Camlp4 is "just" a pre-processor, you always can unsugar your code and
take back a plain ocaml code. This can be done to get rid of a no more
maintained extension, or to understand some code that use such an
extension.

$ camlp4o pa_foo.cmo pa_bar.cmo pr_o.cmo input.ml -o output.ml

You can now read output.ml as a plain ocaml file.

Thus, IMHO, people can add syntax extensions since we have an easy way
to get rid of them.

-- 
Nicolas Pouillard

Re: On language extensions (was Re: [Caml-list] global record)

[Martin Jambon]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 299 bytes --]

On Wed, 19 Jul 2006, Daniel Bünzli wrote:

> Hello,
>
> I would like to issue a warning about everybody rolling its own syntax 
> extension to suit its taste.

Well, it's good to suit your own taste when you write scripts for
yourself :-)


Martin

--
Martin Jambon, PhD
http://martin.jambon.free.fr