From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=disabled version=3.1.3 Received: from mail4-relais-sop.national.inria.fr (mail4-relais-sop.national.inria.fr [192.134.164.105]) by yquem.inria.fr (Postfix) with ESMTP id 2C5B7BC69 for ; Wed, 14 Nov 2007 17:45:49 +0100 (CET) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ah4FAPa0OkdSX+nf/2dsb2JhbACBXI50 X-IronPort-AV: E=Sophos;i="4.21,417,1188770400"; d="scan'208";a="19293260" Received: from concorde.inria.fr ([192.93.2.39]) by mail4-smtp-sop.national.inria.fr with ESMTP; 14 Nov 2007 17:45:49 +0100 Received: from mail4-relais-sop.national.inria.fr (mail4-relais-sop.national.inria.fr [192.134.164.105]) by concorde.inria.fr (8.13.6/8.13.6) with ESMTP id lAEGjkmZ006432 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=OK) for ; Wed, 14 Nov 2007 17:45:48 +0100 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ah4FAPa0OkdSX+nf/2dsb2JhbACBXI50 X-IronPort-AV: E=Sophos;i="4.21,417,1188770400"; d="scan'208";a="19293258" Received: from 5.xs4all.nl (HELO capsaicin.mamane.lu) ([82.95.233.223]) by mail4-smtp-sop.national.inria.fr with ESMTP; 14 Nov 2007 17:45:45 +0100 Received: from master by capsaicin.mamane.lu with local (Exim 4.68) (envelope-from ) id 1IsLN2-0004Yo-BL; Wed, 14 Nov 2007 17:45:44 +0100 Date: Wed, 14 Nov 2007 17:45:44 +0100 From: Lionel Elie Mamane To: Stefan Monnier Cc: caml-list@inria.fr Subject: Re: [Caml-list] Re: OCaml runtime using too much memory in 64-bit Linux Message-ID: <20071114164544.GB16398@capsaicin.mamane.lu> Mail-Followup-To: Lionel Elie Mamane , Stefan Monnier , caml-list@inria.fr References: <4731F5D1.2070405@janestcapital.com> <1194459622.15728.94.camel@localhost.localdomain> <47320E10.1050307@janestcapital.com> <200711140520.46016.romain.beauxis@gmail.com> <8ef825670711140403m464c01detff5537abb2211946@mail.gmail.com> <473AF04C.7030107@inria.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: GNU/Linux X-Request-PGP: http://www.mamane.lu/openpgp/rsa_v4_4096.asc User-Agent: Mutt/1.5.16 (2007-06-11) X-Miltered: at concorde with ID 473B263A.000 by Joe's j-chkmail (http://j-chkmail . ensmp . fr)! X-Spam: no; 0.00; ocaml:01 runtime:01 allocating:01 initialized:01 byte:01 wrote:01 heap:01 caml-list:01 behaves:02 megabytes:02 kernel:02 sys:03 bytes:03 bytes:03 gnu:03 On Wed, Nov 14, 2007 at 11:22:46AM -0500, Stefan Monnier wrote: >> and uses a page table for this purpose, with a dense representation >> (an array of bytes). If the major heap areas are closely spaced, this > [...] >> For 32-bit platforms, this isn't much of a problem since the maximum >> size of the page table is 1 megabytes. For 64-bit platforms, the sky > How about allocating this array of bytes via mmap and then leave it > uninitialized (relying on POSIX's guarantee that it's already > initialized to zeros)? > This way you can easily have a 4GB "dense" table which doesn't use much > RAM since most of the 4GB will be mapped (via copy-on-write) to the same > "zero page". I think this will fail on a GNU/Linux 2.6 system with /proc/sys/vm/overcommit_memory set to 2, or any other system that behaves as Linux with overcommit to 2. (Meaning, it actually reserves place from the swap+ram pool, so that any mmapped/sbrk'd memory can actually be used. It permits the kernel to guarantee that even if all programs actually use all the memory they allocate, it will be able to serve them - albeit slowly (swap use)). In particular, the addressing space of a 64 bit machine is, well... 64 bits, by definition. For 4kiB = 2^12 B pages, one thus needs a table of size 2^(64-12) = 2^52 bytes, that is 4 EB. That is, on any machine with less than that of memory (and overcommit to 2), the program will not run. Even at one bit (and not byte) per page, that is still 16PB... Big pages don't get you out of the problem. 4MB pages only buy you a factor 1024, that is 4PB and 16GB. -- Lionel