[Caml-list] Bigarray.map_file with large files?

[Caml-list] Bigarray.map_file with large files?

[Nicolas Ojeda Bar]

Dear list,

Is it correct that one cannot map large files (> 2GB in 32-bit systems) using Bigarray.map_file?
Is there any way around this?

Thank you very much,
Nicolas

Re: [Caml-list] Bigarray.map_file with large files?

[Goswin von Brederlow]

On Tue, Nov 05, 2013 at 07:10:24PM +0000, Nicolas Ojeda Bar wrote:
> Dear list,
> 
> Is it correct that one cannot map large files (> 2GB in 32-bit systems) using Bigarray.map_file?
> Is there any way around this?
> 
> Thank you very much,
> Nicolas

You can map large files just fine. The Bigarray.Array1.map_file takes
a "?pos:int64" argument.

BUT you can only have 2GB memory total on a 32-bit system with a 1:1
(obsolete) split, 3GB with 3:1 split and 3.99GB with a 32-bit thread
on a 64-bit kernel. So you can only map a 1.9G/2.9G/3.9G chunk (or
multiple smaller ones) at a time. So this doesn't help you that much I
bet.


You have to map and unmap the file in chunks and juggle them manually.
There simply is not enough address space available in 32-bit.

You could write (or google for) a wrapper class around Bigarray that
mmaps/munmaps chunks of files on demand. You would have to use an
Int64 as index variable.

MfG
	Goswin