From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 3D03C7F98F for ; Thu, 10 Aug 2017 15:17:32 +0200 (CEST) Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=kennethadammiller@gmail.com; spf=Pass smtp.mailfrom=kennethadammiller@gmail.com; spf=None smtp.helo=postmaster@mail-lf0-f45.google.com Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of kennethadammiller@gmail.com) identity=pra; client-ip=209.85.215.45; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="kennethadammiller@gmail.com"; x-sender="kennethadammiller@gmail.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of kennethadammiller@gmail.com designates 209.85.215.45 as permitted sender) identity=mailfrom; client-ip=209.85.215.45; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="kennethadammiller@gmail.com"; x-sender="kennethadammiller@gmail.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mail-lf0-f45.google.com) identity=helo; client-ip=209.85.215.45; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="kennethadammiller@gmail.com"; x-sender="postmaster@mail-lf0-f45.google.com"; x-conformance=sidf_compatible IronPort-PHdr: =?us-ascii?q?9a23=3Akq9Whhz3hCKY9zfXCy+O+j09IxM/srCxBDY+r6Qd?= =?us-ascii?q?0eoeIJqq85mqBkHD//Il1AaPBtqLra8cw8Pt8IneGkU4qa6bt34DdJEeHzQksu?= =?us-ascii?q?4x2zIaPcieFEfgJ+TrZSFpVO5LVVti4m3peRMNQJW2TxTor3az9T8fHAnkfUow?= =?us-ascii?q?f7ytW92as8Pi8+21s6LTYhlFzG65frNzMBierwzXu9IKm4ZvNuA6zR6f8VVSfO?= =?us-ascii?q?ED5m5uI1+Pn17V6s61tLti9yBdobp19MNGV6jmf600RLldDTAiPnod68jitB2F?= =?us-ascii?q?RgyKsChPGl4KmwZFVlCWpCrxWY38526j7rJw?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0B4AQDGW4xZhi3XVdFcHAEBBAEBCgEBF?= =?us-ascii?q?wEBBAEBCgEBgkQ+giUHhSWYa4FukGKHRYVHAoRzB0IVAQEBAQEBAQEBAQESAQE?= =?us-ascii?q?BCAsLCCgvgjMFAR4BBYI7AQEBAQIBIx0BGx0BAwwGBQsDCioCAiEBAREBBQEcB?= =?us-ascii?q?hOKFgEDDQigZD+MCoIEBQEcgwkFg2AKGScNVoNPAQEBAQEBBAEBAQEBARoCBhK?= =?us-ascii?q?DFoIChlaCV4UvgmEBBJ9hPI9EhHSSUowviBcUH4EVNYEsMiEkXhqEbg8QDIIDJ?= =?us-ascii?q?DaIUYFTAQEB?= X-IPAS-Result: =?us-ascii?q?A0B4AQDGW4xZhi3XVdFcHAEBBAEBCgEBFwEBBAEBCgEBgkQ?= =?us-ascii?q?+giUHhSWYa4FukGKHRYVHAoRzB0IVAQEBAQEBAQEBAQESAQEBCAsLCCgvgjMFA?= =?us-ascii?q?R4BBYI7AQEBAQIBIx0BGx0BAwwGBQsDCioCAiEBAREBBQEcBhOKFgEDDQigZD+?= =?us-ascii?q?MCoIEBQEcgwkFg2AKGScNVoNPAQEBAQEBBAEBAQEBARoCBhKDFoIChlaCV4Uvg?= =?us-ascii?q?mEBBJ9hPI9EhHSSUowviBcUH4EVNYEsMiEkXhqEbg8QDIIDJDaIUYFTAQEB?= X-IronPort-AV: E=Sophos;i="5.41,353,1498514400"; d="scan'208,217";a="286684869" Received: from mail-lf0-f45.google.com ([209.85.215.45]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/AES128-GCM-SHA256; 10 Aug 2017 15:17:31 +0200 Received: by mail-lf0-f45.google.com with SMTP id o85so3323309lff.3 for ; Thu, 10 Aug 2017 06:17:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=7Z+INLReEjVWFFMKzHHCH22kKAs/djXTHUJ8GWHCrA4=; b=aJ8F28MHMdTXgkahaD6FU/dFkbQEiQOFh1qO2BIcR8tbF8bh8LcWXgzv9DtRaDmYqO AbyH3puUwXG3Mz3bpkNsMkp+NX61qBVqUD5lFDE+VlCYhdyO7dx5T0r1rh4F33gtY7hR 5ZpUWheIsWdiDt387GdEj0XRXTqmNeuhRTUQnb+FQmAU9xTqPSHKZdF+HASTI3zu5t8R xy2ggm8eO2pggkt5hGG+T6xjKMTNXrWoqNlcKQ7unWxxzrmuVFaahhB9RaOeiB4zVyNq 7piwQNswjmlt8X17ERosiahEpSfUdbEZGfNVj0cwa/hysjJnsbSLO7/nyi8cltpEIbke lI4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=7Z+INLReEjVWFFMKzHHCH22kKAs/djXTHUJ8GWHCrA4=; b=D0eh/UW/rXWOE2BwZDuV5ZI+Md/r7ne1hj9X6VPMkNT0uG3Mebap8MKMeD2u2hkKXx mEiOCXUr7SdnQHKCrJNYVj+RthiW86CeewaGY+qsSNJudfqI7w7oAqHpt6Uvmhmnhod8 EVjZWL9hWC/3MnI7wTq2KdpBIFGw4qxGqGM/WpWbL4VnOPYzFf3txm23Fonre2/ooEZ8 1cv/amjZBWaYylqA0hkaGNDWfW79uQ/3JH2NGmnLR0UloJA9e9eXqSbTm1jO3Fk8tdWP 2Vau5Leiyn3/AAVI1F1eeeKJzAyvkuny/QTDppByTr8UkYQgXf3iNLXXc5ZWbIU1YRkK UYUQ== X-Gm-Message-State: AHYfb5h2sW+QNPJ3ozehOrOwTWwXFtmqKCOYm1yLiny3yI071q8/lCFm FRR/lXNJkI/3wmCT7X3HMpMFgRYN6qip X-Received: by 10.25.19.24 with SMTP id j24mr4386703lfi.54.1502371050331; Thu, 10 Aug 2017 06:17:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.46.81.18 with HTTP; Thu, 10 Aug 2017 06:17:29 -0700 (PDT) In-Reply-To: References: From: Kenneth Adam Miller Date: Thu, 10 Aug 2017 09:17:29 -0400 Message-ID: To: Leo White Cc: caml users Content-Type: multipart/alternative; boundary="001a114019b4c6a4f705566604da" Subject: Re: [Caml-list] Overflow with Spacetime --001a114019b4c6a4f705566604da Content-Type: text/plain; charset="UTF-8" Ok, that's understandable. No complaints on my end, spacetime is awesome. On Thu, Aug 10, 2017 at 9:12 AM, Leo White wrote: > > Possibly I ran into a space leak while trying to find a space leak. > > 7GB for a 900MB profile sounds about right unfortunately. The format of > spacetime profiles makes it difficult for the viewer to do much better than > that. I hope to get around to changing the format to fix this, but probably > won't until some time next year. > > I would recommend trying to get a smaller profile. Taking snapshots less > often will help, as will running the program for a shorter time (even with > the same number of snapshots). Failing that try getting access to a machine > with more memory. > > On 9 August 2017 at 08:37, Kenneth Adam Miller < > kennethadammiller@gmail.com> wrote: > >> Hello all, >> >> >> I wrote a particular application, and after profiling I can see that the >> garbage collector is using 95% of the time in the application and that it >> uses up to 6 gigs of memory on a 1 megabyte input file for a run that >> totals out to be about 2 minutes. A 27 kilobyte input file takes seconds to >> run, so I know that there is some explosive leak or otherwise terrible >> issue with the space complexity here. I doubled down, and got compilation >> to run to completion using _tags to make sure that there were symbols and >> profiling turned on (thanks Ivan, Drup). >> >> Additionally, I compiled my application with spacetime and ran it to >> obtain a `spacetime-` binary, which I then attempted to get a report >> on with prof_spacetime. prof_spacetime went up to about 7.3 gigs in memory >> usage before it repeatedly would die with `Killed` or `Stack overflow` as >> the error. I'm trying to figure out what else there is I can do to make >> sure that the process runs to completion. I double checked to make sure >> that I was running the prof_spacetime command built with just a 4.04.0 >> compiler to ensure that it wouldn't profile itself while trying to "Process >> series". My spacetime dump is about 900 Mb, so I understand that it would >> use some memory processing it. Possibly I ran into a space leak while >> trying to find a space leak. >> >> In the meantime, I will try and find some intermediate size input to get >> something I can work on - thought I should report this. >> >> >> > --001a114019b4c6a4f705566604da Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Ok, that's understandable. No complaints on my end, sp= acetime is awesome.

On Thu, Aug 10, 2017 at 9:12 AM, Leo White <lwhite@janestreet.= com> wrote:
>=C2=A0Possibly I ra= n into a space leak while trying to find a space leak.

7GB for a 900MB profile sounds about right unfortunately. The fo= rmat of spacetime profiles makes it difficult for the viewer to do much bet= ter than that. I hope to get around to changing the format to fix this, but= probably won't until some time next year.

I would recommend trying to get a smaller profile. Taking snapshots less = often will help, as will running the program for a shorter time (even with = the same number of snapshots). Failing that try getting access to a machine= with more memory.

On 9 August 201= 7 at 08:37, Kenneth Adam Miller <kennethadammiller@gmail.com= > wrote:
Hello= all,


I wrote a particular application, a= nd after profiling I can see that the garbage collector is using 95% of the= time in the application and that it uses up to 6 gigs of memory on a 1 meg= abyte input file for a run that totals out to be about 2 minutes. A 27 kilo= byte input file takes seconds to run, so I know that there is some explosiv= e leak or otherwise terrible issue with the space complexity here. I double= d down, and got compilation to run to completion using _tags to make sure t= hat there were symbols and profiling turned on (thanks Ivan, Drup).=C2=A0
Additionally, I compiled my application with spacetime and ran it to = obtain a `spacetime-<pid>` binary, which I then attempted to get a re= port on with prof_spacetime. prof_spacetime went up to about 7.3 gigs in me= mory usage before it repeatedly would die with `Killed` or `Stack overflow`= as the error. I'm trying to figure out what else there is I can do to = make sure that the process runs to completion. I double checked to make sur= e that I was running the prof_spacetime command built with just a 4.04.0 co= mpiler to ensure that it wouldn't profile itself while trying to "= Process series". My spacetime dump is about 900 Mb, so I understand th= at it would use some memory processing it. Possibly I ran into a space leak= while trying to find a space leak.=C2=A0

In the meantime, I will tr= y and find some intermediate size input to get something I can work on - th= ought I should report this.




--001a114019b4c6a4f705566604da--