From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id 3DDE07ED69 for ; Tue, 28 May 2019 19:17:00 +0200 (CEST) Authentication-Results: mail3-smtp-sop.national.inria.fr; spf=None smtp.pra=xavier.leroy@college-de-france.fr; spf=Pass smtp.mailfrom=xavier.leroy@college-de-france.fr; spf=None smtp.helo=postmaster@smtpout10.partage.renater.fr Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of xavier.leroy@college-de-france.fr) identity=pra; client-ip=194.254.240.31; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="xavier.leroy@college-de-france.fr"; x-sender="xavier.leroy@college-de-france.fr"; x-conformance=sidf_compatible Received-SPF: Pass (mail3-smtp-sop.national.inria.fr: domain of xavier.leroy@college-de-france.fr designates 194.254.240.31 as permitted sender) identity=mailfrom; client-ip=194.254.240.31; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="xavier.leroy@college-de-france.fr"; x-sender="xavier.leroy@college-de-france.fr"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@smtpout10.partage.renater.fr) identity=helo; client-ip=194.254.240.31; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="xavier.leroy@college-de-france.fr"; x-sender="postmaster@smtpout10.partage.renater.fr"; x-conformance=sidf_compatible X-IronPort-AV: E=Sophos;i="5.60,523,1549926000"; d="scan'208,217";a="307485133" X-MGA-submission: =?us-ascii?q?MDG/EV7Q82RY1QdPjZ2vNyk0/rObgKNUMbctXt?= =?us-ascii?q?A1awSpsJk7I4itAilkrWeTjvSFfeENeuWR6MjVAJb+wJFL3Ii26Kgk1C?= =?us-ascii?q?J6l6F72pZ2PkRCY58+tABrtOy7P3srS5T1d2UHZ8xQUADda6RxOxjHYt?= =?us-ascii?q?MxJkEHJreJIFSnnLBEtJvT1Q=3D=3D?= Received: from smtpout01-ext4.partage.renater.fr (HELO smtpout10.partage.renater.fr) ([194.254.240.31]) by mail3-smtp-sop.national.inria.fr with ESMTP; 28 May 2019 19:16:59 +0200 Received: from zmtaout01.partage.renater.fr (zmtaout01.partage.renater.fr [194.254.240.30]) by smtpout10.partage.renater.fr (Postfix) with ESMTP id C30BB6017A for ; Tue, 28 May 2019 19:16:58 +0200 (CEST) Received: from zmtaout01.partage.renater.fr (localhost [127.0.0.1]) by zmtaout01.partage.renater.fr (Postfix) with ESMTPS id BE0901A003C for ; Tue, 28 May 2019 19:16:58 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by zmtaout01.partage.renater.fr (Postfix) with ESMTP id B00D71A0101 for ; Tue, 28 May 2019 19:16:58 +0200 (CEST) X-Virus-Scanned: amavisd-new at zmtaout01.partage.renater.fr Received: from zmtaout01.partage.renater.fr ([127.0.0.1]) by localhost (zmtaout01.partage.renater.fr [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id fMPBv7KajuW6 for ; Tue, 28 May 2019 19:16:58 +0200 (CEST) Received: from 209.85.222.47 (lb01.partage.renater.fr [194.254.242.1]) by zmtaout01.partage.renater.fr (Postfix) with ESMTPA id 6D38D1A003C for ; Tue, 28 May 2019 19:16:58 +0200 (CEST) Received: by mail-ua1-f47.google.com with SMTP id n7so8243497uap.12 for ; Tue, 28 May 2019 10:16:58 -0700 (PDT) X-Gm-Message-State: APjAAAXvAjzcu/AbXVQztz2RV8JBKG63jg7b/JsSQoV0ZfleQDTAMfAy DLS+x21HqKuFUoHY3bY0k+UF78WjPBFbrJx7K74= X-Google-Smtp-Source: APXvYqyz5bL7jrhQoazwqQ9mNjqJAcChU12Fa70F6h3Wiep0Jyg4laTboNhWnuPSS6QXo1DJjm5xi8SZeOhCvyK5eMw= X-Received: by 2002:ab0:7842:: with SMTP id y2mr9888604uaq.80.1559063817642; Tue, 28 May 2019 10:16:57 -0700 (PDT) MIME-Version: 1.0 References: <8d058e9d-5293-46f1-9941-7c33fd232e04@www.fastmail.com> In-Reply-To: <8d058e9d-5293-46f1-9941-7c33fd232e04@www.fastmail.com> From: Xavier Leroy Date: Tue, 28 May 2019 19:16:31 +0200 X-Gmail-Original-Message-ID: Message-ID: To: Jon French Cc: caml users Content-Type: multipart/alternative; boundary="000000000000086b2e0589f5d5b7" Subject: Re: [Caml-list] Only bytecode version of executable allocating huge amounts --000000000000086b2e0589f5d5b7 Content-Type: text/plain; charset="UTF-8" On Tue, May 28, 2019 at 2:27 PM Jon French wrote: > Hi all, > > I'm wondering if anyone on this list might have encountered this issue > before, since it's got me pretty stumped. > > We have an OCaml project ( https://github.com/rems-project/sail ) which > tries to allocate huge amounts of memory, but only when compiled to > bytecode. The native version works perfectly fine. > This is unusual indeed. Do you have a repro case that you could share publicly? If so, please submit an issue at https://github.com/ocaml/ocaml/issues and attach the repro. The first thing I would do is to increase the verbosity of the GC: set the OCAMLRUNPARAM environment variable to the value "v=255". The resulting GC traces can give an idea of what's going on. Kind regards, - Xavier Leroy > > strace for an example run ends in: > > openat(AT_FDCWD, "/home/ojno/work/sail2/lib/vector_dec.sail", > O_RDONLY|O_CLOEXEC) = 3 > lseek(3, 0, SEEK_CUR) = 0 > read(3, "$ifndef _VECTOR_DEC\n$define _VEC"..., 65536) = 7031 > mmap(NULL, 1965819695104, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, > -1, 0) = -1 ENOMEM (Cannot allocate memory) > brk(0x5729f32c4000) = 0x55603f2f4000 > mmap(NULL, 1965819826176, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, > -1, 0) = -1 ENOMEM (Cannot allocate memory) > write(2, "Fatal error: out of memory.\n", 28Fatal error: out of memory. > ) = 28 > exit_group(2) = ? > +++ exited with 2 +++ > > > which naturally fails since I don't have 2 TB of memory available. > > The exact amount of the allocation varies from input to input and even run > to run, but is always that approximate size. There's no characteristic > pattern of heap or stack running out of control if I turn on memory > debugging info in $OCAMLRUNPARAM, just that one huge allocation. When run > in ocamldebug, it also doesn't appear to be failing at a location which is > sensible to be allocating such amounts of memory -- and the location also > varies with the input. And on tiny inputs it doesn't seem to try and make > the allocation at all. > > Are we actually seeing a compiler/runtime bug here? Is there something I'm > missing? > > Thanks, > > Jon French > Computer Laboratory > University of Cambridge > --000000000000086b2e0589f5d5b7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Tue, May 28, 2019 at 2:27 PM Jon French <jf451@cam.ac.uk> wrote:
Hi all,

I'm wondering if anyone on this list might have encountered thi= s issue before, since it's got me pretty stumped.

We have an OCaml project ( https://github.com/rems-project/sail ) which= tries to allocate huge amounts of memory, but only when compiled to byteco= de. The native version works perfectly fine.

This is unusual indeed.

Do you h= ave a repro case that you could share publicly?=C2=A0 If so, please submit = an issue at https://githu= b.com/ocaml/ocaml/issues and attach the repro.

The first thing I would do is to increase the verbosity of the GC: set the= OCAMLRUNPARAM environment variable to the value "v=3D255".=C2=A0= The resulting GC traces can give an idea of what's going on.

Kind regards,

- Xavier Leroy


=C2=A0

strace for an= example run ends in:

openat(AT_FDCWD, "/home/ojno/work/sail2/lib/vector_dec.sail", O= _RDONLY|O_CLOEXEC) =3D 3
lseek(3, 0, SEEK_CUR)=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 =3D 0
read(3, "$ifndef _VECTOR_DEC\n$= define _VEC"..., 65536) =3D 7031
mmap(NULL, 196581969510= 4, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =3D -1 ENOMEM (C= annot allocate memory)
brk(0x5729f32c4000)=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D 0x55603f2f4000
mmap(NULL, 196= 5819826176, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =3D -1 = ENOMEM (Cannot allocate memory)
write(2, "Fatal error: o= ut of memory.\n", 28Fatal error: out of memory.
) =3D 28=
exit_group(2)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D ?
+++ exited with 2 = +++

which naturally fails since I= don't have 2 TB of memory available.

The= exact amount of the allocation varies from input to input and even run to = run, but is always that approximate size. There's no characteristic pat= tern of heap or stack running out of control if I turn on memory debugging = info in $OCAMLRUNPARAM, just that one huge allocation. When run in ocamldeb= ug, it also doesn't appear to be failing at a location which is sensibl= e to be allocating such amounts of memory -- and the location also varies w= ith the input. And on tiny inputs it doesn't seem to try and make the a= llocation at all.

Are we actually seeing a com= piler/runtime bug here? Is there something I'm missing?
<= br>
Thanks,

Jon French
Computer Laboratory
University of Cambridge
<= /blockquote> --000000000000086b2e0589f5d5b7--