From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail4-relais-sop.national.inria.fr (mail4-relais-sop.national.inria.fr [192.134.164.105]) by walapai.inria.fr (8.13.6/8.13.6) with ESMTP id q34BVJhj032534 for ; Wed, 4 Apr 2012 13:31:19 +0200 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AmQCANcwfE/RVdK2mGdsb2JhbABFuBUIIgEBAQEBCAkNBxQnggkBAQEEEgITGQEbHQEDDAYFCw0uIQEBEQEFARwGExsHh2cLnCwKjBaCcYRhP4h2AQULig2GPASVY4ERiiKDHT2EDA X-IronPort-AV: E=Sophos;i="4.75,367,1330902000"; d="scan'208";a="139035801" Received: from mail-iy0-f182.google.com ([209.85.210.182]) by mail4-smtp-sop.national.inria.fr with ESMTP/TLS/RC4-SHA; 04 Apr 2012 13:31:02 +0200 Received: by iahk25 with SMTP id k25so368939iah.27 for ; Wed, 04 Apr 2012 04:31:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type:content-transfer-encoding; bh=C/0UpyejMaVekrF/n0ODerihDfeTCZnMIj7sz0nYzlU=; b=PW8jZ3bZWX52E752KHMoCcAP97+CYTdFzjI0SbmuK/sJyRh+83IK2/jM90EdCkTKkq ZeqA9VEWJKOZG24wFn1y17Nm82+nV0gSChcbosSpfSOeXUSdR2rMK90iIcGQgIDlCGmI nPIBvYvuHfDa4BW/5DYttdxf9N7Oy1rU9Csr2/PXmfZ+jhG5XH1TaI5wL10+D63ORgQD 9CFr0c0e6R2/N3dxoIzUR5c4ukRmVT/THPPKuWtFML4QpSJfBxwUAFpPs9Oe8rEx9Kqz y7L2H40K+NZRXuwwnO6XCzlUVsywe9kec4eyyV6+GkZk3gVOButVUD8f4ku433v6NABb eMjQ== Received: by 10.50.46.194 with SMTP id x2mr1106635igm.60.1333539061732; Wed, 04 Apr 2012 04:31:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.50.140.100 with HTTP; Wed, 4 Apr 2012 04:30:21 -0700 (PDT) In-Reply-To: References: <20120401195733.GB15870@annexia.org> <20120403121327.GA29159@pps.jussieu.fr> From: Gabriel Scherer Date: Wed, 4 Apr 2012 13:30:21 +0200 Message-ID: To: Hans Ole Rafaelsen Cc: caml-list@inria.fr Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by walapai.inria.fr id q34BVJhj032534 Subject: Re: [Caml-list] Strategies for finding memory leaks May your program leak one of those GTK resources? The effectiveness of your patch seems to indicate that you have a lot of one of these values allocated (and that they were requesting the GC much too frequently). The patch solves the CPU usage induced by additional GC, but does not change the reason why those GC were launched: apparently your code allocates a lot of those resources. If there indeed is a leak in your program, it will use more and more memory even if you fix the CPU-usage effect. An interesting side-effect of your patch is that you could, by selectively disabling some of the change you made (eg. by changing Val_g_boxed but not Val_g_boxed_new), isolate which of those resources were provoking the increased CPU usage, because it was allocated in high number. (Usual candidates that provoke leak are global data structures that store references to your data. A closure will also reference the data corresponding to the variables it captures, so storing closures in such tables can be an indirect cause for "leaks". Do you have global tables of callbacks or values for GTK-land?) On Wed, Apr 4, 2012 at 12:53 PM, Hans Ole Rafaelsen wrote: > Hi, > > Thanks for your suggestions. I tried to patch lablgtk2 with: > > --- src/ml_gdkpixbuf.c.orig     2012-04-03 13:56:29.618264702 +0200 > +++ src/ml_gdkpixbuf.c  2012-04-03 13:56:58.106263510 +0200 > @@ -119,7 +119,7 @@ >    value ret; >    if (pb == NULL) ml_raise_null_pointer(); >    ret = alloc_custom (&ml_custom_GdkPixbuf, sizeof pb, > -                     100, 1000); > +                     0, 1); >    p = Data_custom_val (ret); >    *p = ref ? g_object_ref (pb) : pb; >    return ret; > > --- src/ml_gobject.c.orig       2012-04-03 15:40:11.002004506 +0200 > +++ src/ml_gobject.c    2012-04-03 15:41:04.938002250 +0200 > @@ -219,7 +219,7 @@ >  CAMLprim value ml_g_value_new(void) >  { >      value ret = alloc_custom(&ml_custom_GValue, > sizeof(value)+sizeof(GValue), > -                             20, 1000); > +                             0, 1); >      /* create an MLPointer */ >      Field(ret,1) = (value)2; >      ((GValue*)&Field(ret,2))->g_type = 0; > @@ -272,14 +272,14 @@ >    custom_serialize_default, custom_deserialize_default }; >  CAMLprim value Val_gboxed(GType t, gpointer p) >  { > -    value ret = alloc_custom(&ml_custom_gboxed, 2*sizeof(value), 10, 1000); > +    value ret = alloc_custom(&ml_custom_gboxed, 2*sizeof(value), 0, 1); >      Store_pointer(ret, g_boxed_copy (t,p)); >      Field(ret,2) = (value)t; >      return ret; >  } >  CAMLprim value Val_gboxed_new(GType t, gpointer p) >  { > -    value ret = alloc_custom(&ml_custom_gboxed, 2*sizeof(value), 10, 1000); > +    value ret = alloc_custom(&ml_custom_gboxed, 2*sizeof(value), 0, 1); >      Store_pointer(ret, p); >      Field(ret,2) = (value)t; >      return ret; > > > > At startup is uses > top - 16:40:27 up 1 day,  7:01, 28 users,  load average: 0.47, 0.50, 0.35 > Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie > Cpu(s):  4.8%us,  1.3%sy,  0.0%ni, 93.6%id,  0.2%wa,  0.0%hi,  0.1%si, > 0.0%st > Mem:   4004736k total,  3617960k used,   386776k free,   130704k buffers > Swap:  4070396k total,     9244k used,  4061152k free,  1730344k cached > >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ > COMMAND > 10275 hans      20   0  529m  77m  13m S   14  2.0   0:01.66 > vc_client.nativ > > and 12 hours later > top - 04:40:07 up 1 day, 19:01, 35 users,  load average: 0.00, 0.01, 0.05 > Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie > Cpu(s): 20.2%us,  3.4%sy,  0.0%ni, 76.1%id,  0.1%wa,  0.0%hi,  0.2%si, > 0.0%st > Mem:   4004736k total,  3828308k used,   176428k free,   143928k buffers > Swap:  4070396k total,    10708k used,  4059688k free,  1756524k cached > >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ > COMMAND > 10275 hans      20   0  534m  82m  13m S   17  2.1 110:11.19 > vc_client.nativ > > Without the patch > top - 22:05:38 up 1 day, 12:26, 34 users,  load average: 0.35, 0.16, 0.13 > Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie > Cpu(s):  5.6%us,  1.5%sy,  0.0%ni, 92.6%id,  0.2%wa,  0.0%hi,  0.1%si, > 0.0%st > Mem:   4004736k total,  3868136k used,   136600k free,   140900k buffers > Swap:  4070396k total,     9680k used,  4060716k free,  1837500k cached > >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ > COMMAND > 25111 hans      20   0  453m  76m  13m S   14  2.0   0:13.68 vc_client_old.n > > top - 10:05:19 up 2 days, 26 min, 35 users,  load average: 0.01, 0.04, 0.05 > Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie > Cpu(s): 20.4%us,  3.2%sy,  0.0%ni, 75.8%id,  0.4%wa,  0.0%hi,  0.2%si, > 0.0%st > Mem:   4004736k total,  3830596k used,   174140k free,   261692k buffers > Swap:  4070396k total,    13640k used,  4056756k free,  1640452k cached > >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ > COMMAND > 25111 hans      20   0  453m  76m  13m S   49  2.0 263:05.34 > vc_client_old.n > > So from this it seems that with the patch it still uses more and more CPU, > but at a much lower rate. However, it seems to increase memory usage with > the patch. I also tried to patch the wrappers.h file, but the memory > consumption just exploded. > > So it is working better, but still not good enough. Is there some way to > prevent this kind of behavior? That is, no extra memory usage and no extra > CPU usage. > > I have attached some additional profiling if that would be of any interest. > In short it seems to be that it is the GC that is consuming the CPU. > > Best, > > Hans Ole > > > On Tue, Apr 3, 2012 at 2:13 PM, Jerome Vouillon > wrote: >> >> On Tue, Apr 03, 2012 at 12:42:08PM +0200, Gerd Stolpmann wrote: >> > This reminds me of a problem I had with a specific C binding (for >> > mysql), >> > years ago. That binding allocated custom blocks with badly chosen >> > parameters used/max (see the docs for caml_alloc_custom in >> > http://caml.inria.fr/pub/docs/manual-ocaml/manual032.html#toc144). If >> > the >> > ratio used/max is > 0, these parameters accelerate the GC. If the custom >> > blocks are frequently allocated, this can have a dramatic effect, even >> > for >> > quite small used/max ratios. The solution was to change the code, and to >> > set used=0 and max=1. >> > >> > This type of problem would match your observation that the GC works more >> > and more the longer the program runs, i.e. the more custom blocks have >> > been allocated. >> > >> > The problem basically also exists with bigarrays - with >> > used= and max=256M (hardcoded). >> >> I have also observed this with input-output channels (in_channel and >> out_channel), where used = 1 and max = 1000. A full major GC is >> performed every time a thousand files are opened, which can result on >> a significant overhead when you open lot of files and the heap is >> large. >> >> -- Jerome > >