Re: [Caml-list] New release of Menhir (20141215)

["Frédéric Bour" <frederic.bour@lakaban.net>]

Hi,

I tried various methods with Merlin, with good results.
That's quite close to what you suggest: we add an empty non-terminal and 
change the behavior when it is on stack.
Something like:

lexer_switch:
| (* empty *) { () }

inline:
| …
| LINK lexer_switch OPEN RAW END OPEN inline* END
| …

Lexing loop:

if has_lexer_switch parser then
   feed parser (Lexer.raw_token buf)
else
   feed parser (Lexer.token buf)

Of course this require introspection (Merlin's internal version exposes 
a lot more informations, but that's for debugging and experimentation 
purposes, we hope to clean that).
You can still use side-effects to emulate the trick:

lexer_switch:
| (* empty *) { in_raw_lexer := true }

lexer_leave:
| (* empty *) { in_raw_lexer := false }

… but be careful :).

Cheers,
Fred

On 26/12/2014 12:13, Dario Teixeira wrote:
> Hi,
>
>> Hmm, maybe. The new API will probably allow you to inspect the stack (which is
>> basically a list of pairs of a state and a semantic value) and to inspect a
>> state (which can be viewed as a set of LR(1) items). I don't know whether that
>> would offer you a simple way of deciding when to switch from one lexer to
>> another...
> I suspect it will *at least* be an improvement over the current situation.
> Consider the following rule:
>
> inline:
> | ...
> | LINK OPEN RAW END OPEN inline* END
> | ...
>
> Suppose that by default I'm using a 'general' lexer.  However, upon encountering
> that first OPEN token, I must switch to a 'raw' lexer and then switch back to the
> 'general' lexer upon encountering the first END token.  This lexer dance won't
> happen with the second OPEN token, though.
>
> Anyway, as long as I know which state Menhir is in, choosing the right lexer
> should be an easy task.  It may require a large lookup table on my part to map
> state to lexer, but at least that's a lot less hairy than the current approach.
>
> Best regards,
> Dario Teixeira
>