Re: [Caml-list] Efficient scanning of large strings from files

["Jérémie Dimino" <jeremie@dimino.org>]

Le Fri, 16 Mar 2012 14:03:38 +0100,
Philippe Veber <philippe.veber@gmail.com> a écrit :

> Say that you'd like to search a regexp on a file with lines so long
> that you'd rather not load them entirely at once. If you can bound
> the size of a match by k << length of a line, then you know that you
> can only keep a small portion of the line in memory to search the
> regexp. Typically you'd like to access substrings of size k from left
> to right. I guess such a thing should involve buffered inputs and
> avoid copying strings as much as possible. My question is as follows:
> has anybody written a library to access these substrings gracefully
> and with decent performance? Cheers,

You can use a non-backtracking regexp library to find offsets of the
substrings, then seek in the file to extract them. You can use for
example the libre library from Jérôme Vouillon [1]. It only accept
strings as input but it would be really easy to make it work on input
channels (just replace "s.[pos]" by "input_char ic").

  [1] http://sourceforge.net/projects/libre/
      https://github.com/avsm/ocaml-re.git

Cheers,

-- 
Jérémie