Re: ocamllex speed [was Re: [Caml-list] mboxlib reloaded ;-)]

[skaller <skaller@users.sourceforge.net>]

On Mon, 2007-09-24 at 20:22 +0200, Bruno De Fraine wrote:

> { }
> rule translate = parse
> | "current_directory" { print_endline (Sys.getcwd ()); translate  
> lexbuf }
> | _ { translate lexbuf }
> | eof { () }
> {
>      for i = 1 to (Array.length Sys.argv - 1); do
>          translate (Lexing.from_channel (open_in Sys.argv.(i)))
>      done ;;
> }
> 
>  What causes this difference? Perhaps there  
> is a high overhead in calling the translate function for every input  
> character in such big input files, but I don't know how this can be  
> avoided.

Easily. Rewrite the lexer to 

| [a-zA-z_]+ { if lexeme = "current_directory" ... }
| _ { translate lexbuf }

Then it will eat whole words instead of characters,
which should make it about 5 times faster.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net