From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=AWL autolearn=disabled version=3.1.3 Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by yquem.inria.fr (Postfix) with ESMTP id E99D2BBCA for ; Thu, 8 May 2008 15:55:40 +0200 (CEST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AhACANeiIkjAXQIniGdsb2JhbACSAwEBAQ8gmV4 X-IronPort-AV: E=Sophos;i="4.27,455,1204498800"; d="scan'208";a="12382883" Received: from concorde.inria.fr ([192.93.2.39]) by mail3-smtp-sop.national.inria.fr with ESMTP; 08 May 2008 15:55:40 +0200 Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by concorde.inria.fr (8.13.6/8.13.6) with ESMTP id m48DteZO028699 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=OK) for ; Thu, 8 May 2008 15:55:40 +0200 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AhACABOjIkhQRFuwiGdsb2JhbACSAwEBAQ8gmWA X-IronPort-AV: E=Sophos;i="4.27,455,1204498800"; d="scan'208";a="10512595" Received: from furbychan.cocan.org ([80.68.91.176]) by mail2-smtp-roc.national.inria.fr with ESMTP; 08 May 2008 15:55:39 +0200 Received: from rich by furbychan.cocan.org with local (Exim 4.63) (envelope-from ) id 1Ju6ax-0007Ft-3E; Thu, 08 May 2008 14:55:39 +0100 Date: Thu, 8 May 2008 14:55:38 +0100 To: Markus Mottl Cc: caml-list@inria.fr Subject: Re: [Caml-list] ocamlc.opt tries to allocate 34 GB of RAM ... and fails Message-ID: <20080508135538.GF20312@annexia.org> References: <20080507170149.GA21077@annexia.org> <04bfd9a7e0fb697f30fc7376d28b8fa5@localhost> <20080507183424.GA24624@annexia.org> <20080507184957.GA25416@annexia.org> <20080508091449.GA16556@annexia.org> <20080508101413.GA18974@annexia.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.13 (2006-08-11) From: Richard Jones X-Miltered: at concorde with ID 4823065C.000 by Joe's j-chkmail (http://j-chkmail . ensmp . fr)! X-Spam: no; 0.00; ocamlc:01 markus:01 mottl:01 bug:01 ocaml:01 initialized:01 runtime:01 cvs-version:01 hash:01 ocaml:01 wrote:01 wrote:01 heap:01 heap:01 caml-list:01 On Thu, May 08, 2008 at 09:40:12AM -0400, Markus Mottl wrote: > On Thu, May 8, 2008 at 6:14 AM, Richard Jones wrote: > > Growing heap to 4320k bytes > > Growing page table to 34359705221 entries > > No room for growing page table > > Fatal error: out of memory. > > Yeah, that's an indication that you've hit the mentioned problem > (addresses of allocated memory too far apart). Yes, we came a similar conclusion. However there is a MAP_32BIT flag that you can add to mmap which makes it return low memory addresses. There are two problems with this, namely (a) the flag is only available on x86-64 (and we have the same problem on ppc64), and (b) it limits the whole heap to 1 or 2 GB. Nevertheless this is the solution we have gone with for now. http://lxr.linux.no/linux/include/asm-x86_64/mman.h#L6 https://bugzilla.redhat.com/attachment.cgi?id=304848 https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9 Hopefully OCaml 3.11 will arrive before Fedora 10 is released, in which case we'll just upgrade. > > Unfortunately I have no idea how to solve this, certainly it seems > > there is no simple fix ... (I tried to set vm.overcommit_memory > > policy, but that doesn't work because the page table is initialized > > right after allocation). > > If my mmap "fix" (not guaranteed to work with all kernels) is in the > executing runtime, then, I'm afraid, there is little you can do other > than switching to a CVS-version of the upcoming OCaml-release. Xavier > has implemented a completely new page table representation there, > which should solve this problem once and for all. Backporting all that hash table code into OCaml 3.10 didn't look like fun to me :-( Rich. -- Richard Jones Red Hat