Garbage collection in OCaml

Garbage collection in OCaml

[David McClain]

Hi,

I have a long running analysis program written in compiled OCaml (ocamlopt).
If I let it run without interference it gradually allocates more and more
memory until the system swap space is exhausted. At that point the program
bombs off with an "out of memory" errror - probably generated by the OCaml
array management routines.

OTOH, I found by tinkering that placing a Gc.compact() in a periodically
performed place, I manage to keep the entire system running within about 70
MBytes. (My machines all have 256 MB RAM or more).

I have found that placing a Gc.full_major() does virtually nothing to
prevent the exhaustion of memory, although it slows it down ever so
slightly.

The program was compiled to run with the default GC settings (whatever those
are). That is to say, I did nothing to configure the GC system at program
startup.

Is this behavior normal? Must I plant strategic Gc.compact() in my code? I
would have thought the GC would be more self monitoring.  Your comments are
most welcome, even if they are critical!

Cheers,

David McClain, Sr. Scientist
Raytheon Missile Systems Co.
Tucson, AZ

Re: Garbage collection in OCaml

[Francois Rouaix]

> OTOH, I found by tinkering that placing a Gc.compact() in a periodically
> performed place, I manage to keep the entire system running within about 70
> MBytes. (My machines all have 256 MB RAM or more).

> I have found that placing a Gc.full_major() does virtually nothing to
> prevent the exhaustion of memory, although it slows it down ever so
> slightly.

That would be typical of an application that is stable in memory
allocation (no leaks), but has a pattern of allocation that causes
fragmentation. I have that a lot in network applications that use
buffers for I/O, and I usually either call Gc.compact once in a while,
or I set max_overhead in Gc.control to something like 50.
For example, one of the servers behind shopping.nbci.com uses anywhere
from 300M to 500M, but I have to leave the compaction to avoid
swapping.

--f

Re: Garbage collection in OCaml

[David McClain]

Yes, Thanks so much for your helpful suggestions! Your Gc settings were what
I was hoping to find from someone with insight to the behavior of the GC. I
am trying them as we speak!

Cheers!

- DM

-----Original Message-----
From: Damien Doligez <Damien.Doligez@inria.fr>
To: caml-list@inria.fr <caml-list@inria.fr>
Date: Monday, October 02, 2000 4:32 AM
Subject: Re: Garbage collection in OCaml


>>From: "David McClain" <dmcclain@azstarnet.com>
>
>>I have a long running analysis program written in compiled OCaml
(ocamlopt).
>>If I let it run without interference it gradually allocates more and more
>>memory until the system swap space is exhausted. At that point the program
>>bombs off with an "out of memory" errror - probably generated by the OCaml
>>array management routines.
>
>Most likely, a fragmentation problem.  If you're using a lot of arrays
>or strings with different sizes, it can happen.
>
>
>>OTOH, I found by tinkering that placing a Gc.compact() in a periodically
>>performed place, I manage to keep the entire system running within about
70
>>MBytes. (My machines all have 256 MB RAM or more).
>
>Thus we know for sure it was a fragmentation problem.
>
>
>>I have found that placing a Gc.full_major() does virtually nothing to
>>prevent the exhaustion of memory, although it slows it down ever so
>>slightly.
>
>Gc.full_major() is not going to help, that's normal.
>
>
>>The program was compiled to run with the default GC settings (whatever
those
>>are). That is to say, I did nothing to configure the GC system at program
>>startup.
>>
>>Is this behavior normal? Must I plant strategic Gc.compact() in my code? I
>>would have thought the GC would be more self monitoring.
>
>Calling Gc.compact() is one solution.  You could also try setting
>max_overhead in the Gc.control record.  For example, if you do:
>
>  Gc.set { (Gc.get ()) with Gc.max_overhead = 200 }
>
>then Gc.compact() will get called automatically when the free memory
>is 200% the size of the live data.  This setting is not activated by
>default because compaction is not incremental, so it introduces a long
>pause in the computation, which would be annoying for an interactive
>program.
>
>-- Damien
>

Re: Garbage collection in OCaml

[Damien Doligez]

>From: "David McClain" <dmcclain@azstarnet.com>

>I have a long running analysis program written in compiled OCaml (ocamlopt).
>If I let it run without interference it gradually allocates more and more
>memory until the system swap space is exhausted. At that point the program
>bombs off with an "out of memory" errror - probably generated by the OCaml
>array management routines.

Most likely, a fragmentation problem.  If you're using a lot of arrays
or strings with different sizes, it can happen.


>OTOH, I found by tinkering that placing a Gc.compact() in a periodically
>performed place, I manage to keep the entire system running within about 70
>MBytes. (My machines all have 256 MB RAM or more).

Thus we know for sure it was a fragmentation problem.


>I have found that placing a Gc.full_major() does virtually nothing to
>prevent the exhaustion of memory, although it slows it down ever so
>slightly.

Gc.full_major() is not going to help, that's normal.


>The program was compiled to run with the default GC settings (whatever those
>are). That is to say, I did nothing to configure the GC system at program
>startup.
>
>Is this behavior normal? Must I plant strategic Gc.compact() in my code? I
>would have thought the GC would be more self monitoring.

Calling Gc.compact() is one solution.  You could also try setting
max_overhead in the Gc.control record.  For example, if you do:

  Gc.set { (Gc.get ()) with Gc.max_overhead = 200 }

then Gc.compact() will get called automatically when the free memory
is 200% the size of the live data.  This setting is not activated by
default because compaction is not incremental, so it introduces a long
pause in the computation, which would be annoying for an interactive
program.

-- Damien