Re: [Caml-list] ocamlc.opt tries to allocate 34 GB of RAM ... and fails

[Richard Jones <rich@annexia.org>]

On Thu, May 08, 2008 at 09:40:12AM -0400, Markus Mottl wrote:
> On Thu, May 8, 2008 at 6:14 AM, Richard Jones <rich@annexia.org> wrote:
> >   Growing heap to 4320k bytes
> >   Growing page table to 34359705221 entries
> >   No room for growing page table
> >   Fatal error: out of memory.
> 
> Yeah, that's an indication that you've hit the mentioned problem
> (addresses of allocated memory too far apart).

Yes, we came a similar conclusion.

However there is a MAP_32BIT flag that you can add to mmap which makes
it return low memory addresses.  There are two problems with this,
namely (a) the flag is only available on x86-64 (and we have the same
problem on ppc64), and (b) it limits the whole heap to 1 or 2 GB.
Nevertheless this is the solution we have gone with for now.

  http://lxr.linux.no/linux/include/asm-x86_64/mman.h#L6
  https://bugzilla.redhat.com/attachment.cgi?id=304848
  https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9

Hopefully OCaml 3.11 will arrive before Fedora 10 is released, in
which case we'll just upgrade.

> >  Unfortunately I have no idea how to solve this, certainly it seems
> >  there is no simple fix ...  (I tried to set vm.overcommit_memory
> >  policy, but that doesn't work because the page table is initialized
> >  right after allocation).
> 
> If my mmap "fix" (not guaranteed to work with all kernels) is in the
> executing runtime, then, I'm afraid, there is little you can do other
> than switching to a CVS-version of the upcoming OCaml-release.  Xavier
> has implemented a completely new page table representation there,
> which should solve this problem once and for all.

Backporting all that hash table code into OCaml 3.10 didn't look like
fun to me :-(

Rich.

-- 
Richard Jones
Red Hat