From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <Sebastien.Hinderer@inria.fr>
Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104])
	by sympa.inria.fr (Postfix) with ESMTPS id 052877EEF8
	for <caml-list@sympa.inria.fr>; Tue, 28 Jul 2015 17:52:03 +0200 (CEST)
X-IronPort-AV: E=Sophos;i="5.15,564,1432591200"; 
   d="scan'208";a="141407569"
Received: from hp-sebastien.rsr.lip6.fr (HELO pl-59055.rocqadm.inria.fr) ([132.227.76.32])
  by mail3-relais-sop.national.inria.fr with ESMTP/TLS/AES128-SHA; 28 Jul 2015 17:52:02 +0200
Received: from shindere by pl-59055.rocqadm.inria.fr with local (Exim 4.84)
	(envelope-from <Sebastien.Hinderer@inria.fr>)
	id 1ZK7F8-0007Ex-Ny
	for caml-list@inria.fr; Tue, 28 Jul 2015 17:56:38 +0200
Date: Tue, 28 Jul 2015 17:56:38 +0200
From: =?utf-8?Q?S=C3=A9bastien?= Hinderer <Sebastien.Hinderer@inria.fr>
To: caml-list@inria.fr
Message-ID: <20150728155638.GA27761@pl-59055.rocqadm.inria.fr>
Mail-Followup-To: caml-list@inria.fr
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
Subject: [Caml-list] updating position in an ocamllex lexer

Dear all,

I have to maintain a lexer which did not, so far, compute token
positions during the lexing step. The positions were computed
later, which was not very eficient. So I am now trying to compute
positions during the lexing and have a problem to figure out what
to do for one token:

  | ['\n'] [' ' '\t' '\r' '\011' '\012' ]* { ... }

When this token is met and contains just "\n", calling new_line on the
lexbuf is enough to keep further position information accurate.

However, if the recognized token is, say, "\n\t", then it seems that the
column for further tokens will be incorrect.

I assumed that I had to manually update the current position and added
code like so:

  let s = Lexing.lexeme lexbuf in
  let l = String.length s in
  let t = TCommentNewline (tokinfo lexbuf) in
  (* Adjust the column manually *)
  Lexing.new_line lexbuf;
  let lcp = lexbuf.lex_curr_p in
  lexbuf.lex_curr_p <- { lcp with
    pos_cnum = lcp.pos_bol + l - 1;
  };
  t

But that does not seem to work.

Does somebody know how such tokens should be handled, please?

Many thanks in advance for any help,

Sébastien.