memory profiling

memory profiling

[Christoph Bauer]

Hi,

what is the best option to do memory profiling with ocaml?
Is there a patch of ocaml-memprof for 3.11.0? What about
objsize?

Thanks,

Christoph Bauer

Re: [Caml-list] memory profiling

[dmitry grebeniuk]

2009/5/5 Christoph Bauer <christoph.bauer@lmsintl.com>:

> what is the best option to do memory profiling with ocaml?
> Is there a patch of ocaml-memprof for 3.11.0? What about
> objsize?

  If you want to use objsize with ocaml 3.11, you should get
the new version of objsize -- 0.12:
http://forge.ocamlcore.org/frs/?group_id=3
  OCaml has new heap since 3.11, and old versions won't work.

  objsize has an unresolved make-related problem with building
on msvc/win32 (object/library file extensions, to be specific), so
one should build objsize manually on msvc (not a hard thing.
but I'll fix it in near future).

Re: memory profiling

[Sylvain Le Gall]

Hello,

On 05-05-2009, Christoph Bauer <christoph.bauer@lmsintl.com> wrote:
> Hi,
>
> what is the best option to do memory profiling with ocaml?
> Is there a patch of ocaml-memprof for 3.11.0? What about
> objsize?
>

I use a more simple approach (though I have used objsize to estimate
some datastructure size, but only in the toplevel): GC allocation rate.

You can override a little ocaml-benchmark to measure the allocation rate
of the GC. This gives you a pretty good understanding on the fact you
are allocating too much or not.

Regards,
Sylvain Le Gall

ps: here is a part of my benchmarkExt.ml file


(** Benchmark extension
    @author Sylvain Le Gall
  *)

open Benchmark;;

type t =
    {
      benchmark: Benchmark.t;
      memory_used: float;
    }
;;

let gc_wrap f x =
  (* Extend sample to add GC stat *)
  let add_gc_stat memory_used samples =
    List.map 
      (fun (name, lst) ->
         name,
         List.map 
           (fun bt -> 
              { 
                benchmark = bt; 
                memory_used = memory_used;
              }
           )
           lst
      )
      samples
  in

  (* Call throughput1 and add GC stat *)
  let () = 
    print_string "Cleaning memory before benchmark"; print_newline ();    
    Gc.full_major ()
  in
  let allocated_before = 
    Gc.allocated_bytes ()
  in
  let samples =
    f x
  in
  let () = 
    print_string "Cleaning memory after benchmark"; print_newline ();
    Gc.full_major ()
  in
  let memory_used = 
    ((Gc.allocated_bytes ()) -. allocated_before) 
  in
    add_gc_stat memory_used samples
;;

let throughput1
      ?min_count ?style
      ?fwidth    ?fdigits
      ?repeat    ?name
      seconds 
      f x =
 
  (* Benchmark throughput1 as it should be called *) 
  gc_wrap 
    (throughput1
       ?min_count ?style
       ?fwidth    ?fdigits
       ?repeat    ?name
       seconds f) x
;;

let throughputN 
      ?min_count ?style
      ?fwidth    ?fdigits
      ?repeat    
      seconds name_f_args =
  List.flatten
    (List.map
       (fun (name, f, args) ->
         throughput1 
           ?min_count ?style
           ?fwidth    ?fdigits
           ?repeat    ~name:name
           seconds f args)
       name_f_args)
;;

let latency1 
      ?min_cpu ?style 
      ?fwidth  ?fdigits 
      ?repeat  n 
      ?name    f x =
  gc_wrap 
    (latency1
      ?min_cpu ?style 
      ?fwidth  ?fdigits 
      ?repeat  n 
      ?name    f) x
;;

let latencyN 
      ?min_cpu ?style 
      ?fwidth  ?fdigits 
      ?repeat  
      n name_f_args =
  List.flatten
    (List.map
       (fun (name, f, args) ->
         latency1 
           ?min_cpu   ?style
           ?fwidth    ?fdigits
           ?repeat    ~name:name
           n          f args)
       name_f_args)
;;

Re: [Caml-list] Re: memory profiling

[Jon Harrop]

On Tuesday 05 May 2009 15:39:21 Sylvain Le Gall wrote:
> Hello,
>
> On 05-05-2009, Christoph Bauer <christoph.bauer@lmsintl.com> wrote:
> > Hi,
> >
> > what is the best option to do memory profiling with ocaml?
> > Is there a patch of ocaml-memprof for 3.11.0? What about
> > objsize?
>
> I use a more simple approach (though I have used objsize to estimate
> some datastructure size, but only in the toplevel): GC allocation rate.
>
> You can override a little ocaml-benchmark to measure the allocation rate
> of the GC. This gives you a pretty good understanding on the fact you
> are allocating too much or not.

FWIW, I automated the same approach using camlp4 and intend to write it up in 
the OCaml Journal. The results were quite good although run-time performance 
was seriously degraded (~50x slower) and it would have been nice to visualize 
the results in an IDE.

-- 
Dr Jon Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/?e

Re: [Caml-list] memory profiling

[Richard Jones]

On Sat, Jan 09, 2010 at 01:25:52PM +0000, Jon Harrop wrote:
> On Tuesday 05 May 2009 13:45:18 dmitry grebeniuk wrote:
> > 2009/5/5 Christoph Bauer <christoph.bauer@lmsintl.com>:
> > > what is the best option to do memory profiling with ocaml?
> > > Is there a patch of ocaml-memprof for 3.11.0? What about
> > > objsize?
> >
> >   If you want to use objsize with ocaml 3.11, you should get
> > the new version of objsize -- 0.12:
> > http://forge.ocamlcore.org/frs/?group_id=3
> >   OCaml has new heap since 3.11...
> 
> Can anyone elaborate on this?

Not sure about "new heap", but the way that heap pages are tracked
changed from 3.10 -> 3.11.  In 3.10 a flat bitmap was used.  This was
unsuitable for 64 bit address spaces[1] and in 3.11 a sparse structure
is used (a hash table).

Rich.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9

-- 
Richard Jones
Red Hat

Re: [Caml-list] memory profiling

[Jon Harrop]

On Tuesday 05 May 2009 13:45:18 dmitry grebeniuk wrote:
> 2009/5/5 Christoph Bauer <christoph.bauer@lmsintl.com>:
> > what is the best option to do memory profiling with ocaml?
> > Is there a patch of ocaml-memprof for 3.11.0? What about
> > objsize?
>
>   If you want to use objsize with ocaml 3.11, you should get
> the new version of objsize -- 0.12:
> http://forge.ocamlcore.org/frs/?group_id=3
>   OCaml has new heap since 3.11...

Can anyone elaborate on this?

-- 
Dr Jon Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/?e