Re: [Caml-list] updating position in an ocamllex lexer

[Leo White <leo@lpw25.net>]

I think the problem with your attempt to manually update the current
position is that you are adjusting pos_cnum, which is the number of
characters since the beginning of the whole file, not the number of
characters since the beginning of the line. Instead you should be
adjusting pos_bol. For example, if you look at the source for
Lexing.new_line:

  let new_line lexbuf =
    let lcp = lexbuf.lex_curr_p in
    lexbuf.lex_curr_p <- { lcp with
      pos_lnum = lcp.pos_lnum + 1;
      pos_bol = lcp.pos_cnum;
    }

you can see that it adds one to the line number and sets pos_bol
to be the current position.

Regards,

Leo

On Tue, 28 Jul 2015, at 11:56 AM, Sébastien Hinderer wrote:
> Dear all,
> 
> I have to maintain a lexer which did not, so far, compute token
> positions during the lexing step. The positions were computed
> later, which was not very eficient. So I am now trying to compute
> positions during the lexing and have a problem to figure out what
> to do for one token:
> 
>   | ['\n'] [' ' '\t' '\r' '\011' '\012' ]* { ... }
> 
> When this token is met and contains just "\n", calling new_line on the
> lexbuf is enough to keep further position information accurate.
> 
> However, if the recognized token is, say, "\n\t", then it seems that the
> column for further tokens will be incorrect.
> 
> I assumed that I had to manually update the current position and added
> code like so:
> 
>   let s = Lexing.lexeme lexbuf in
>   let l = String.length s in
>   let t = TCommentNewline (tokinfo lexbuf) in
>   (* Adjust the column manually *)
>   Lexing.new_line lexbuf;
>   let lcp = lexbuf.lex_curr_p in
>   lexbuf.lex_curr_p <- { lcp with
>     pos_cnum = lcp.pos_bol + l - 1;
>   };
>   t
> 
> But that does not seem to work.
> 
> Does somebody know how such tokens should be handled, please?
> 
> Many thanks in advance for any help,
> 
> Sébastien.
> 
> -- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs