From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 77F4B7ED69 for ; Wed, 29 May 2019 17:22:30 +0200 (CEST) Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=jf451@cam.ac.uk; spf=Pass smtp.mailfrom=jf451@cam.ac.uk; spf=Pass smtp.helo=postmaster@ppsw-41.csi.cam.ac.uk Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of jf451@cam.ac.uk) identity=pra; client-ip=131.111.8.141; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jf451@cam.ac.uk"; x-sender="jf451@cam.ac.uk"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of jf451@cam.ac.uk designates 131.111.8.141 as permitted sender) identity=mailfrom; client-ip=131.111.8.141; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jf451@cam.ac.uk"; x-sender="jf451@cam.ac.uk"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of postmaster@ppsw-41.csi.cam.ac.uk designates 131.111.8.141 as permitted sender) identity=helo; client-ip=131.111.8.141; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jf451@cam.ac.uk"; x-sender="postmaster@ppsw-41.csi.cam.ac.uk"; x-conformance=sidf_compatible; x-record-type="v=spf1" IronPort-PHdr: =?us-ascii?q?9a23=3AnCK03h2yVjGuX0masmDT+DRfVm0co7zxezQtwd8Z?= =?us-ascii?q?sesWIvzxwZ3uMQTl6Ol3ixeRBMOHsqsC0rCJ+PC/EUU7or+5+EgYd5JNUxJXwe?= =?us-ascii?q?43pCcHRPC/NEvgMfTxZDY7FskRHHVs/nW8LFQHUJ2mPw6arXK99yMdFQviPgRp?= =?us-ascii?q?OOv1BpTSj8Oq3Oyu5pHfeQpFiCegbb9oMRm6swfcusYVjIZgN6081gbHrnxUdu?= =?us-ascii?q?pM2GhmP0iTnxHy5sex+J5s7SFdsO8/+sBDTKv3Yb02QaRXAzo6PW814tbrtQTY?= =?us-ascii?q?QguU+nQcSGQWnQFWDAXD8Rr3Q43+sir+tup6xSmaIcj7Rq06VDi+86tmTgLjhT?= =?us-ascii?q?wZPDAl7m7Yls1wjLpaoB2/oRx/35XUa5yROPZnY6/RYc8WSW9HU81MVSJOH5m8?= =?us-ascii?q?YpMIAeUPMulWrIvyp1UAoxW+GweiGf/gxyRSiXPqx6A3yf4sHBvE0QEmAtkAsG?= =?us-ascii?q?7UrNLwNKoKX+y7yK7IzTPeZP1Wwzfy9o7IfQwhof2CQLl9dsjRyUcgGg7Fk1md?= =?us-ascii?q?spDqMCmQ1ugXqWeU8/BsVf+si2M+rQx6vzahxsApiobTh4IVzEjJ9Sp4wIYpJd?= =?us-ascii?q?24VVV0bcS4H5tXsiGWL4R2QsI+Q2FopSY10acKtYSncygNzZQqwQPUZf+fc4WQ?= =?us-ascii?q?/x7uWvudLS1liH54Zb6znRW//VK9xuDzS8W4yFdHoytfntXRq3wBygbf58edRv?= =?us-ascii?q?dj4Eus2zCC3B3J5O5eO0A7j6/bJoYhwrEukpoTtlzOHjfumEXtgq6ab0op9vWy?= =?us-ascii?q?5+v7ebXmp4WQOJNuhQH7KKghgNCwDf4lMggNR2Sb+OK826P//UDhXblHgOA6nr?= =?us-ascii?q?PEvJzHOMgXvK20DxVI3oss9hqzFzKm384ZnXkDIlJFYhWHj43xNlHMLvD1Avey?= =?us-ascii?q?j0m3nTh33f/GO6ftDY/RIXTZjbfhfq5x61RAxwor0dBf+5VUB6kdL/3pX0/xsM?= =?us-ascii?q?XUDhs4Mwyv3+bqE85914MbWWKXGKCVKqLSsVmS5uIuOeaAfoEVuCzlJ/gg/fHu?= =?us-ascii?q?jHs5mVEHfamu2JsacHe4HvZgI0WeZnbsh8kOEXwPvgoiVOzlkkCCUSJTZ3aqQa?= =?us-ascii?q?08/Co7CIWgDYjZQoCtgaCB3SeiEpBVaW1LCVGBHWnoeoiLVPoAcT+eL8t9njAa?= =?us-ascii?q?V7WsSoss2BC3uA/4xbpqIerZ9jAAuJ/7yNd6/ejTmQso+jNoFcidzmKNQnp6nm?= =?us-ascii?q?wSXD82wKV/rlZ8yleHy6R3n/tYFdlV6vhUTAo6MYPcz/dmC9/sQALPY9aJSVe4?= =?us-ascii?q?Tdi+HT1iBu42lpUgakxnGt6vxjTOlwSnGKQckaDBTMg6+6jG3nP8YcJw/HjLz7?= =?us-ascii?q?IoiUUORcBGMGm+nKk5/A/WUcqB2WSHnqDiWqMA2zDG9Gaf1iDG6EBGXyZxXKjI?= =?us-ascii?q?G3cFaR2Fg87+4xaIbbioQZo9Pw1KyYTKfqlENoCwpV5PQbHqM5LDYDTiyC+LGR?= =?us-ascii?q?+Uy+bUP8LRcGIH0XCYURBcylFBzTO9LQE7QxyZjSfbBT1qG0joZhq1o+J3rTWy?= =?us-ascii?q?RQkpzFPTNhAz5/+O4hcQwMekZbYT07YD4nxzsy1vAxPhhpTdENvGrANkOqxXJ8?= =?us-ascii?q?4+sg4eiTDp8jdlN5nlFJhMw0YEel0u7Ujn0lN+AcNdkppyoQ=3D=3D?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0BjAgBPo+5ch40Ib4NlHAEBAQQBAQcEA?= =?us-ascii?q?QGBZYJ6UwEwKIQTgV+HHIpbAQEGgTV+iFKRBgEIAQMBDCMMAQGDCYE3AoJ1HAY?= =?us-ascii?q?BBDQTAQMBAQQBAQIBAQMBEwEBAQgNCQgpIwyCOikBgmcBAgMjHQEBNwEPCxgqA?= =?us-ascii?q?gIsKwYBEoMiAYF2FA+ob3GBL4J5AQEFgTIBgQmCfiGBTgMGgTSCFoRnhWWBB4E?= =?us-ascii?q?RJ4I2By4+gmECgg6CXYJYgTEBAQGmRmkBBgIBAoINX4VZjGEhgh+KaolKjHWFW?= =?us-ascii?q?4ErjyOBZoF4gQE/LhpWgU6CDxqDVod6glo+AQExjkcBAQ?= X-IPAS-Result: =?us-ascii?q?A0BjAgBPo+5ch40Ib4NlHAEBAQQBAQcEAQGBZYJ6UwEwKIQ?= =?us-ascii?q?TgV+HHIpbAQEGgTV+iFKRBgEIAQMBDCMMAQGDCYE3AoJ1HAYBBDQTAQMBAQQBA?= =?us-ascii?q?QIBAQMBEwEBAQgNCQgpIwyCOikBgmcBAgMjHQEBNwEPCxgqAgIsKwYBEoMiAYF?= =?us-ascii?q?2FA+ob3GBL4J5AQEFgTIBgQmCfiGBTgMGgTSCFoRnhWWBB4ERJ4I2By4+gmECg?= =?us-ascii?q?g6CXYJYgTEBAQGmRmkBBgIBAoINX4VZjGEhgh+KaolKjHWFW4ErjyOBZoF4gQE?= =?us-ascii?q?/LhpWgU6CDxqDVod6glo+AQExjkcBAQ?= X-IronPort-AV: E=Sophos;i="5.60,527,1549926000"; d="scan'208,217";a="385268575" X-MGA-submission: =?us-ascii?q?MDEkXopC2iXhbKPGX49nrbun0Hm3Mgy9KhDoQS?= =?us-ascii?q?OEhG3CCXvIT97JXgAVjHsfgz5ZRWgoCgtLJ3Mhn9L+LJG4S/qM/K97Ft?= =?us-ascii?q?jGsshlyudlqPGEsk2Fi0hRsx4LYxGCsaWKoePePtpVOp+KZmpHQy0Czg?= =?us-ascii?q?cYAdeVH3PaPs8/J45LsO10Yw=3D=3D?= Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 May 2019 17:22:18 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cam.ac.uk; s=20180806.ppsw; h=Content-Type:Subject:Cc:To:From:Date:References: In-Reply-To:Message-Id:Mime-Version:Sender:Reply-To:Content-Transfer-Encoding :Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=CWDwk+FmGtiEgm4uoL4nAG6szwCrC8TYGmvcaAxuBHE=; b=hvqVC0cXQWz9IS3Hf8MwPGkhgM 0w42COQN0AhfYNSJmHMOs5fXOxLsBDoCTPalUFLFB6hnieQtxHewSsy4CcvPb1/bIB+QNjpEp0Ehr qTn28RzXgrysl+vPZM3o/JM8W19shs9J8mTEEXFl8V8OodagA+RVGUwIX1scwqoHMMF0=; X-Cam-AntiVirus: no malware found X-Cam-ScannerInfo: http://help.uis.cam.ac.uk/email-scanner-virus Received: from auth1-smtp.messagingengine.com ([66.111.4.227]:38875) by ppsw-41.csi.cam.ac.uk (smtp.hermes.cam.ac.uk [131.111.8.159]:587) with esmtpsa (LOGIN:jf451) (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) id 1hW0PA-0017GN-QY (Exim 4.91) (return-path ); Wed, 29 May 2019 16:22:16 +0100 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailauth.nyi.internal (Postfix) with ESMTP id 5C35721FF5; Wed, 29 May 2019 11:22:14 -0400 (EDT) Received: from imap2 ([10.202.2.52]) by compute6.internal (MEProxy); Wed, 29 May 2019 11:22:14 -0400 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddvjedgkeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvffutgesrgdtreerreerjeenucfhrhhomhepfdflohhn ucfhrhgvnhgthhdfuceojhhfgeehudestggrmhdrrggtrdhukheqnecuffhomhgrihhnpe hgihhthhhusgdrtghomhenucfrrghrrghmpehmrghilhhfrhhomhepohhjnhhoodhmvghs mhhtphgruhhthhhpvghrshhonhgrlhhithihqdekgeekiedvheeggedqudeltddvjeeile eiqdhjfheghedupeeptggrmhdrrggtrdhukhesfhgrshhtmhgrihhlrdgtohhmnecuvehl uhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id AD64DE00A1; Wed, 29 May 2019 11:22:13 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.6-555-g49357e1-fmstable-20190528v2 Mime-Version: 1.0 Message-Id: In-Reply-To: References: <8d058e9d-5293-46f1-9941-7c33fd232e04@www.fastmail.com> Date: Wed, 29 May 2019 16:22:13 +0100 From: "Jon French" To: "Fabrice Le Fessant" , "Ivan Gotovchits" Cc: caml-list Content-Type: multipart/alternative; boundary=14104797783642638de8b435ec21f658 Subject: Re: [Caml-list] Only bytecode version of executable allocating huge amounts --14104797783642638de8b435ec21f658 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable Thanks all for your replies. I've opened a github issue here: https://githu= b.com/ocaml/ocaml/issues/8699 On Tue, 28 May 2019, at 18:58, Fabrice Le Fessant wrote: > A common difference between bytecode and native programs is that the nati= ve compiler computes the liveness of variables in the stack, so that they a= re not scanned anymore afterwards (or coalesced with other variables). For = example, if you have a data structure that is allocated before a computatio= n, but not used in the computation, the structure might be kept alive in th= e bytecode, but garbage collected in native code. >=20 > Le mar. 28 mai 2019 =C3=A0 19:23, Ivan Gotovchits a =C3=A9= crit : >> My (common) suspect is Zarith. It uses a different backend when compiled= to bytecode, so I would start my investigation from this point :)=20 >>=20 >> Hope it helps, >> Ivan >>=20 >> On Tue, May 28, 2019 at 8:27 AM Jon French wrote: >>> __ >>> Hi all, >>>=20 >>> I'm wondering if anyone on this list might have encountered this issue = before, since it's got me pretty stumped. >>>=20 >>> We have an OCaml project ( https://github.com/rems-project/sail ) which= tries to allocate huge amounts of memory, but only when compiled to byteco= de. The native version works perfectly fine. >>>=20 >>> strace for an example run ends in: >>>=20 >>>> openat(AT_FDCWD, "/home/ojno/work/sail2/lib/vector_dec.sail", O_RDONLY= |O_CLOEXEC) =3D 3 >>>> lseek(3, 0, SEEK_CUR) =3D 0 >>>> read(3, "$ifndef _VECTOR_DEC\n$define _VEC"..., 65536) =3D 7031 >>>> mmap(NULL, 1965819695104, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYM= OUS, -1, 0) =3D -1 ENOMEM (Cannot allocate memory) >>>> brk(0x5729f32c4000) =3D 0x55603f2f4000 >>>> mmap(NULL, 1965819826176, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYM= OUS, -1, 0) =3D -1 ENOMEM (Cannot allocate memory) >>>> write(2, "Fatal error: out of memory.\n", 28Fatal error: out of memory. >>>> ) =3D 28 >>>> exit_group(2) =3D ? >>>> +++ exited with 2 +++ >>>=20 >>> which naturally fails since I don't have 2 TB of memory available.=20 >>>=20 >>> The exact amount of the allocation varies from input to input and even = run to run, but is always that approximate size. There's no characteristic = pattern of heap or stack running out of control if I turn on memory debuggi= ng info in $OCAMLRUNPARAM, just that one huge allocation. When run in ocaml= debug, it also doesn't appear to be failing at a location which is sensible= to be allocating such amounts of memory -- and the location also varies wi= th the input. And on tiny inputs it doesn't seem to try and make the alloca= tion at all. >>>=20 >>> Are we actually seeing a compiler/runtime bug here? Is there something = I'm missing? >>>=20 >>> Thanks, >>>=20 >>> Jon French >>> Computer Laboratory >>> University of Cambridge > --=20 > Fabrice LE FESSANT > CEO, OCamlPro SAS --14104797783642638de8b435ec21f658 Content-Type: text/html;charset=utf-8 Content-Transfer-Encoding: quoted-printable
Thanks all for your r= eplies. I've opened a github issue here: https://github.com/ocaml/ocaml/issues/8699

On Tue, 28 May 2019, at 18:58, Fabrice Le Fessant wro= te:
A c= ommon difference between bytecode and native programs is that the native co= mpiler computes the liveness of variables in the stack, so that they are no= t scanned anymore afterwards (or coalesced with other variables). For examp= le, if you have a data structure that is allocated before a computation, bu= t not used in the computation, the structure might be kept alive in the byt= ecode, but garbage collected in native code.

<= /div>
Le mar. 28 mai 2019 =C3=A0 19:23, Ivan Gotovchits <ivg@ieee.org> a =C3=A9crit :
=
My (common) suspect is Zarith. It uses a different backend when compil= ed to bytecode, so I would start my investigation from this point :) <= br>

Hope it helps,
Ivan

On Tue, May 28, 2019 at 8:27 AM Jon French <jf451@cam.ac.uk> wrote:

<= div>
Hi all,

I'm wondering if anyone on th= is list might have encountered this issue before, since it's got me pretty = stumped.

We have an OCaml project ( https://github.com/rems-project/sail= ) which tries to allocate huge amounts of memory, but only when compil= ed to bytecode. The native version works perfectly fine.

=
strace for an example run ends in:

openat(AT_FDCWD, "/home/ojno/work/sail2/lib/vecto= r_dec.sail", O_RDONLY|O_CLOEXEC) =3D 3
lseek(3, 0, SEEK_CUR)&= nbsp;           &nbs= p;      =3D 0
read(3, "$ifndef _VECT= OR_DEC\n$define _VEC"..., 65536) =3D 7031
mmap(NULL, 19658196= 95104, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =3D -1 ENOME= M (Cannot allocate memory)
brk(0x5729f32c4000)  &nb= sp;            =       =3D 0x55603f2f4000
mmap(NULL, = 1965819826176, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =3D = -1 ENOMEM (Cannot allocate memory)
write(2, "Fatal error: out= of memory.\n", 28Fatal error: out of memory.
) =3D 28
exit_group(2)         =             &nb= sp;     =3D ?
+++ exited with 2 +++

which naturally fails since I don't ha= ve 2 TB of memory available.

The exact amount= of the allocation varies from input to input and even run to run, but is a= lways that approximate size. There's no characteristic pattern of heap or s= tack running out of control if I turn on memory debugging info in $OCAMLRUN= PARAM, just that one huge allocation. When run in ocamldebug, it also doesn= 't appear to be failing at a location which is sensible to be allocating su= ch amounts of memory -- and the location also varies with the input. And on= tiny inputs it doesn't seem to try and make the allocation at all.

Are we actually seeing a compiler/runtime bug here? I= s there something I'm missing?

Thanks,

Jon French
Computer Laboratory
University of Cambridge
--
Fabrice LE FESSANT
CEO, OCamlPro SAS
<= /blockquote>

= --14104797783642638de8b435ec21f658--