Re: [Caml-list] Is it possible to extend OCaml lexer rules via Camlp4?

[Jun Furuse <jun.furuse@gmail.com>]

Gabriel,

Thanks for the info. But what I want cannot be achieved by the lex filter.

I want to have pcre regexp literals in the same syntax as Perl i.e.
/hello\sworld\\n/. Currently what we do in OCaml is Pcre.regexp
"hello\\sworld\\\\n", where the backslash char must be escaped in a
OCaml string literal. This is lousy for scripting in OCaml.

To have the same or similar syntax as Perl, the lexer must be really
modified. Currently I am using a modified CamlP4 where I can replace
its lexer function, but it is an adhoc way, and I am seeking any
healthier way without such a modification.

Jun

On Thu, Nov 3, 2011 at 7:52 AM, Gabriel Scherer
<gabriel.scherer@gmail.com> wrote:
>> I have tried to override whole the syntax as follows, but it seems
>> that it changes nothing...:
>
> Camlp4 is designed around mutable state. That you Make module produces
> a new grammar doesn't make it the current grammar used by the
> preprocessor. What need to happen is that the current state of the
> lexer/parser is *mutated* by your Make module (whose evaluation is
> then delayed and controlled by Camlp4 itself thanks to registration
> with Register). This is what EXTEND does at the grammar level.
>
> If it suits your need, you can define your modification as a filter
> that will post-process the output of the original lexer. The Token
> module expose a define_filter function to imperatively update the set
> of such active stream transformers.
>
> This idea was suggested to me by Jérémie Dimino, and works well. It is
> used for example in pa_comprehension to define "[?" and "?]" as new
> OCaml keywords (asking for "["; "?" at the camlp4 grammar level would
> allow spaces in between):
>  https://github.com/ocaml-batteries-team/batteries-included/blob/master/src/syntax/pa_comprehension/pa_comprehension.ml
>
> Below is the relevant code:
>
>  module Make (Syntax : Sig.Camlp4Syntax) = struct
>    open Sig;
>    include Syntax;
>
>
>    (* "[?" and "?]" are not recognized as delimiters by the Camlp4
>      lexer; This token parser will spot "["; "?" and "?"; "]" token
>      and insert "[?" and "?]" instead.
>     Thanks to Jérémie Dimino for the idea. *)
>    value rec delim_filter older_filter stream =
>      let rec filter = parser
>      [ [: `(KEYWORD "[", loc); rest :] ->
>          match rest with parser
>          [ [: `(KEYWORD "?", _) :] -> [: `(KEYWORD "[?", loc); filter rest :]
>          | [: :] -> [: `(KEYWORD "[", loc); filter rest :] ]
>      | [: `(KEYWORD "?", loc); rest :] ->
>          match rest with parser
>          [ [: `(KEYWORD "]", loc) :] -> [: `(KEYWORD "?]", loc); filter rest :]
>          | [: :] -> [: `(KEYWORD "?", loc); filter rest :] ]
>      | [: `other; rest :] -> [: `other; filter rest :] ] in
>      older_filter (filter stream);
>
>    value _ = Token.Filter.define_filter (Gram.get_filter ()) delim_filter;
>
>    (* REST OF THE CAMLP4 EXTENSION ... *)
>  end;
>
>
> On Wed, Nov 2, 2011 at 9:34 PM, Jun Furuse <jun.furuse@gmail.com> wrote:
>> Hi,
>>
>> Is it possible for Camlp4 to implement an OCaml syntax extension (i.e.
>> pa_*) which modifies the lexer of OCaml syntax?
>>
>> I have tried to override whole the syntax as follows, but it seems
>> that it changes nothing...:
>>
>> -----------------------------------------------------------
>> open Camlp4
>>
>> module Id : Sig.Id = struct
>>  let name = "pa_extlex"
>>  let version = "1.0"
>> end
>>
>> module XLexer = Xlexer.Make(PreCast.Token)        (* XLexer
>> reimplements OCaml lexer with some extra rules *)
>> module XGram = PreCast.MakeGram(XLexer)
>>
>> module Make (Syntax : Sig.Camlp4Syntax) = struct
>>  let _ = prerr_endline "Creating OCaml syntax with lexer extension"
>>  module M1 = OCamlInitSyntax.Make(PreCast.Ast)(XGram)(PreCast.Quotation)
>>  module M2 = Camlp4OCamlRevisedParser.Make(M1)
>>  module M3 = Camlp4OCamlParser.Make(M2)
>>  include M3
>> end
>>
>> let module M = Register.OCamlSyntaxExtension(Id)(Make) in ()
>> -----------------------------------
>>
>> Jun
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa-roc.inria.fr/wws/info/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>
>>
>