Re: [Caml-list] Strategies for finding memory leaks

[Hans Ole Rafaelsen <hrafaelsen@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 8912 bytes --]

So just to be clear, it seems like I'm allocating lots of objects of a kind
that I don't free. I have been trying to tracking down this in my ML part
that use the library. You suggest that trying to have a counter in the C
binding part of the library and count each time a object is created (and
maybe each time it is destroyed) might be a better option. I have not
worked much with ML<->C binding code, just want to be sure that this might
be a proper way to do it before I start.

-- 
Hans Ole

On Wed, Apr 4, 2012 at 2:55 PM, Gerd Stolpmann <info@gerd-stolpmann.de>wrote:

> Am Mittwoch, den 04.04.2012, 13:30 +0200 schrieb Gabriel Scherer:
> > May your program leak one of those GTK resources?
> >
> > The effectiveness of your patch seems to indicate that you have a lot
> > of one of these values allocated (and that they were requesting the GC
> > much too frequently). The patch solves the CPU usage induced by
> > additional GC, but does not change the reason why those GC were
> > launched: apparently your code allocates a lot of those resources. If
> > there indeed is a leak in your program, it will use more and more
> > memory even if you fix the CPU-usage effect.
> >
> > An interesting side-effect of your patch is that you could, by
> > selectively disabling some of the change you made (eg. by changing
> > Val_g_boxed but not Val_g_boxed_new), isolate which of those resources
> > were provoking the increased CPU usage, because it was allocated in
> > high number.
>
> Or just increment a counter for each type.
>
> Gerd
>
> > (Usual candidates that provoke leak are global data structures that
> > store references to your data. A closure will also reference the data
> > corresponding to the variables it captures, so storing closures in
> > such tables can be an indirect cause for "leaks". Do you have global
> > tables of callbacks or values for GTK-land?)
> >
> > On Wed, Apr 4, 2012 at 12:53 PM, Hans Ole Rafaelsen
> > <hrafaelsen@gmail.com> wrote:
> > > Hi,
> > >
> > > Thanks for your suggestions. I tried to patch lablgtk2 with:
> > >
> > > --- src/ml_gdkpixbuf.c.orig     2012-04-03 13:56:29.618264702 +0200
> > > +++ src/ml_gdkpixbuf.c  2012-04-03 13:56:58.106263510 +0200
> > > @@ -119,7 +119,7 @@
> > >    value ret;
> > >    if (pb == NULL) ml_raise_null_pointer();
> > >    ret = alloc_custom (&ml_custom_GdkPixbuf, sizeof pb,
> > > -                     100, 1000);
> > > +                     0, 1);
> > >    p = Data_custom_val (ret);
> > >    *p = ref ? g_object_ref (pb) : pb;
> > >    return ret;
> > >
> > > --- src/ml_gobject.c.orig       2012-04-03 15:40:11.002004506 +0200
> > > +++ src/ml_gobject.c    2012-04-03 15:41:04.938002250 +0200
> > > @@ -219,7 +219,7 @@
> > >  CAMLprim value ml_g_value_new(void)
> > >  {
> > >      value ret = alloc_custom(&ml_custom_GValue,
> > > sizeof(value)+sizeof(GValue),
> > > -                             20, 1000);
> > > +                             0, 1);
> > >      /* create an MLPointer */
> > >      Field(ret,1) = (value)2;
> > >      ((GValue*)&Field(ret,2))->g_type = 0;
> > > @@ -272,14 +272,14 @@
> > >    custom_serialize_default, custom_deserialize_default };
> > >  CAMLprim value Val_gboxed(GType t, gpointer p)
> > >  {
> > > -    value ret = alloc_custom(&ml_custom_gboxed, 2*sizeof(value), 10,
> 1000);
> > > +    value ret = alloc_custom(&ml_custom_gboxed, 2*sizeof(value), 0,
> 1);
> > >      Store_pointer(ret, g_boxed_copy (t,p));
> > >      Field(ret,2) = (value)t;
> > >      return ret;
> > >  }
> > >  CAMLprim value Val_gboxed_new(GType t, gpointer p)
> > >  {
> > > -    value ret = alloc_custom(&ml_custom_gboxed, 2*sizeof(value), 10,
> 1000);
> > > +    value ret = alloc_custom(&ml_custom_gboxed, 2*sizeof(value), 0,
> 1);
> > >      Store_pointer(ret, p);
> > >      Field(ret,2) = (value)t;
> > >      return ret;
> > >
> > >
> > >
> > > At startup is uses
> > > top - 16:40:27 up 1 day,  7:01, 28 users,  load average: 0.47, 0.50,
> 0.35
> > > Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
> > > Cpu(s):  4.8%us,  1.3%sy,  0.0%ni, 93.6%id,  0.2%wa,  0.0%hi,  0.1%si,
> > > 0.0%st
> > > Mem:   4004736k total,  3617960k used,   386776k free,   130704k
> buffers
> > > Swap:  4070396k total,     9244k used,  4061152k free,  1730344k cached
> > >
> > >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
> > > COMMAND
> > > 10275 hans      20   0  529m  77m  13m S   14  2.0   0:01.66
> > > vc_client.nativ
> > >
> > > and 12 hours later
> > > top - 04:40:07 up 1 day, 19:01, 35 users,  load average: 0.00, 0.01,
> 0.05
> > > Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
> > > Cpu(s): 20.2%us,  3.4%sy,  0.0%ni, 76.1%id,  0.1%wa,  0.0%hi,  0.2%si,
> > > 0.0%st
> > > Mem:   4004736k total,  3828308k used,   176428k free,   143928k
> buffers
> > > Swap:  4070396k total,    10708k used,  4059688k free,  1756524k cached
> > >
> > >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
> > > COMMAND
> > > 10275 hans      20   0  534m  82m  13m S   17  2.1 110:11.19
> > > vc_client.nativ
> > >
> > > Without the patch
> > > top - 22:05:38 up 1 day, 12:26, 34 users,  load average: 0.35, 0.16,
> 0.13
> > > Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
> > > Cpu(s):  5.6%us,  1.5%sy,  0.0%ni, 92.6%id,  0.2%wa,  0.0%hi,  0.1%si,
> > > 0.0%st
> > > Mem:   4004736k total,  3868136k used,   136600k free,   140900k
> buffers
> > > Swap:  4070396k total,     9680k used,  4060716k free,  1837500k cached
> > >
> > >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
> > > COMMAND
> > > 25111 hans      20   0  453m  76m  13m S   14  2.0   0:13.68
> vc_client_old.n
> > >
> > > top - 10:05:19 up 2 days, 26 min, 35 users,  load average: 0.01, 0.04,
> 0.05
> > > Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
> > > Cpu(s): 20.4%us,  3.2%sy,  0.0%ni, 75.8%id,  0.4%wa,  0.0%hi,  0.2%si,
> > > 0.0%st
> > > Mem:   4004736k total,  3830596k used,   174140k free,   261692k
> buffers
> > > Swap:  4070396k total,    13640k used,  4056756k free,  1640452k cached
> > >
> > >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
> > > COMMAND
> > > 25111 hans      20   0  453m  76m  13m S   49  2.0 263:05.34
> > > vc_client_old.n
> > >
> > > So from this it seems that with the patch it still uses more and more
> CPU,
> > > but at a much lower rate. However, it seems to increase memory usage
> with
> > > the patch. I also tried to patch the wrappers.h file, but the memory
> > > consumption just exploded.
> > >
> > > So it is working better, but still not good enough. Is there some way
> to
> > > prevent this kind of behavior? That is, no extra memory usage and no
> extra
> > > CPU usage.
> > >
> > > I have attached some additional profiling if that would be of any
> interest.
> > > In short it seems to be that it is the GC that is consuming the CPU.
> > >
> > > Best,
> > >
> > > Hans Ole
> > >
> > >
> > > On Tue, Apr 3, 2012 at 2:13 PM, Jerome Vouillon <
> vouillon@pps.jussieu.fr>
> > > wrote:
> > >>
> > >> On Tue, Apr 03, 2012 at 12:42:08PM +0200, Gerd Stolpmann wrote:
> > >> > This reminds me of a problem I had with a specific C binding (for
> > >> > mysql),
> > >> > years ago. That binding allocated custom blocks with badly chosen
> > >> > parameters used/max (see the docs for caml_alloc_custom in
> > >> > http://caml.inria.fr/pub/docs/manual-ocaml/manual032.html#toc144).
> If
> > >> > the
> > >> > ratio used/max is > 0, these parameters accelerate the GC. If the
> custom
> > >> > blocks are frequently allocated, this can have a dramatic effect,
> even
> > >> > for
> > >> > quite small used/max ratios. The solution was to change the code,
> and to
> > >> > set used=0 and max=1.
> > >> >
> > >> > This type of problem would match your observation that the GC works
> more
> > >> > and more the longer the program runs, i.e. the more custom blocks
> have
> > >> > been allocated.
> > >> >
> > >> > The problem basically also exists with bigarrays - with
> > >> > used=<size_of_bigarary> and max=256M (hardcoded).
> > >>
> > >> I have also observed this with input-output channels (in_channel and
> > >> out_channel), where used = 1 and max = 1000. A full major GC is
> > >> performed every time a thousand files are opened, which can result on
> > >> a significant overhead when you open lot of files and the heap is
> > >> large.
> > >>
> > >> -- Jerome
> > >
> > >
> >
> >
>
> --
> ------------------------------------------------------------
> Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
> Creator of GODI and camlcity.org.
> Contact details:        http://www.camlcity.org/contact.html
> Company homepage:       http://www.gerd-stolpmann.de
> *** Searching for new projects! Need consulting for system
> *** programming in Ocaml? Gerd Stolpmann can help you.
> ------------------------------------------------------------
>
>

[-- Attachment #2: Type: text/html, Size: 11385 bytes --]