Dynamically switching lexer

Dynamically switching lexer

[Till Varoquaux]

I'm currently doing a toy DSEL,
Camlp4 pops to mind as a very nice way to embed it unfortunately the
standard lexer it comes with is not all that great for my language.
Although I do know how to use another lexer it comes with a catch:
antiquotation need to be properly lexed to string to be passed to the
standard ocaml parser as strings (I use
let expr_of_string = Gram.parse_string Syntax.expr_eoi
this seems simpler than what is shown in tutorials. Am I missing
something?) off course the easy way to delimit such strings
(respecting nested comments, escaped strings...) would be to use the
former lexer, It would need to be switched between both lexers, is
this all possibe?.
Cheers,
Till

Re: [Caml-list] Dynamically switching lexer

[Nicolas Pouillard]

Excerpts from till.varoquaux's message of Wed Jan 09 23:18:57 +0100 2008:
> I'm currently doing a toy DSEL,
> Camlp4 pops to mind as a very nice way to embed it unfortunately the
> standard lexer it comes with is not all that great for my language.
> Although I do know how to use another lexer it comes with a catch:
> antiquotation need to be properly lexed to string to be passed to the
> standard ocaml parser as strings (I use
> let expr_of_string = Gram.parse_string Syntax.expr_eoi
> this seems simpler than what is shown in tutorials. Am I missing
> something?) off course the easy way to delimit such strings
> (respecting nested comments, escaped strings...) would be to use the
> former lexer, It would need to be switched between both lexers, is
> this all possibe?.
> Cheers,
> Till
> 

You cannot easily change the camlp4 lexer dynamically, but when you get a quotation you get a string that you can lex with a different lexer.

Their is only two restrictions:

- ">>" is forbidden since it will end the quotation
- if you start a new quotation inside, you must close it: '<' (':' ident)? ('@' locname)? '<' ... ">>"

Here is the intersting part of the lexer

---------------------------
and quotation c = parse
  | '<' (':' ident)? ('@' locname)? '<'    {                          store c ;
                                                    with_curr_loc quotation c ;
                                                            parse quotation c }
  | ">>"                                                            { store c }
  | eof                                  { err Unterminated_quotation (loc c) }
  | newline                                     { update_loc c None 1 false 0 ;
                                                      store_parse quotation c }
  | _                                               { store_parse quotation c }
---------------------------


-- 
Nicolas Pouillard aka Ertai