Re: lexer, parser

[Christian Lindig <lindig@ips.cs.tu-bs.de>]

Quoting John Skaller (skaller@maxtal.com.au):
> I need to maintain state, but the component also need
> to be re-entrant.

Quoting Jean-Marc EBER (Jean-Marc.Eber@socgen.com):
> You should certainly be in contact with Christian Lindig, who has
> done interesting work on OCamlLex.
> 
> lindig@ips.cs.tu-bs.de
> http://www.cs.tu-bs.de/softech/people/lindig/index.html

Hi,

Like Jean-Marc Eber pointed out have I suggested two extension for
OCamlLex that makes maintaining state easier:

http://www.cs.tu-bs.de/softech/people/lindig/software/lex-patch.html

A patched version of OCamlLex allows for additional parameters to
rules:

    rule scan x y = parse 
    ...

Parameters like x and y make it easier to accumulate results across
many calls of the same or different rules in a lexer.  They help to
avoid global variables which make a scanner no longer re-entrant.

Using the patched version of OCamlLex is fairly uncritical because
only the generated code if different from the original version. 

Maintaining state across different invocations of a lexer requires
some more effort:

http://www.cs.tu-bs.de/softech/people/lindig/software/lexing.html

I have proposed a new Lexing module with a new `lexbuf' type.  The
lexbuf type is used by scanners to maintain state but currently does
not hold user provided state.  The proposed module permits to store
user state in lexbuf as well. 

Unfortunately is it not possible to simply exchange the new module for
the old.  The whole OCaml system must be recompiled with the new
module.  However, all old code compiles with the new system.

I still would like to see both proposals added to the OCaml system
since both are backward compatible at the source code level.

-- Christian

-- 
 Christian Lindig       Technische Universitaet Braunschweig, Germany
                        http://www.cs.tu-bs.de/softech/people/lindig
                        lindig@ips.cs.tu-bs.de phone:+49-531-391-3276