[Caml-list] Netmcore problems.

[Caml-list] Netmcore problems.

[Anders Fugmann]

Hi,

I'm having a problem using netmcore from ocamlnet 3.4 library.

When I try to place a large data structure into Netmcore_array, the 
controlling process dies with the following error:

[Fri Jan 13 16:21:38 2012] [netplex.controller] [alert] Process 28067 
for service netmcore_0 terminated with signal 7

The error occurs while trying to initialise a Netmcore_array from an 
array of 4*10^6 strings all of length 150. This is done in the context 
of the "first" process.

The controlling process initializes a memory pool by using

   let pool_id = Netmcore_mempool.create_mempool (1024 * 1024 * 1024 * 
10) in

Running the code on a smaller dataset seems to work.

I have increased kernel variables shmall and shmmax to very high numbers 
(100Gb), but it does not solve the problem.

All ideas are welcome.

On other question while I'm at it; am I allowed to create multiple 
shared arrays from the same memory pool, or do I need to create one pool 
for each shared array?

Regards
Anders Fugmann

Re: [Caml-list] Netmcore problems.

[Gerd Stolpmann]

Am Freitag, den 13.01.2012, 17:30 +0100 schrieb Anders Fugmann:
> Hi,
> 
> I'm having a problem using netmcore from ocamlnet 3.4 library.
> 
> When I try to place a large data structure into Netmcore_array, the 
> controlling process dies with the following error:
> 
> [Fri Jan 13 16:21:38 2012] [netplex.controller] [alert] Process 28067 
> for service netmcore_0 terminated with signal 7

That's sigbus. You get it when the OS does not have memory anymore - but
not when the memory is allocated, but first when it is filled with data.
You can change this by disallowing that memory is overcommitted (set the
kernel param vm.overcommit_memory to 1 or even 2).

> The error occurs while trying to initialise a Netmcore_array from an 
> array of 4*10^6 strings all of length 150. This is done in the context 
> of the "first" process.
> 
> The controlling process initializes a memory pool by using
> 
>    let pool_id = Netmcore_mempool.create_mempool (1024 * 1024 * 1024 * 
> 10) in
> 
> Running the code on a smaller dataset seems to work.
> 
> I have increased kernel variables shmall and shmmax to very high numbers 
> (100Gb), but it does not solve the problem.

No, this cannot solve it. These variables only control System V shared
memory, but Netmcore uses POSIX shared memory. On Linux, you can change
the max of this memory by re-mounting /dev/shm, e.g.

mount -o remount,size=80% /dev/shm

The default is 50% of available RAM.

> All ideas are welcome.
> 
> On other question while I'm at it; am I allowed to create multiple 
> shared arrays from the same memory pool, or do I need to create one pool 
> for each shared array?

The pools can be shared.

Gerd

> 
> Regards
> Anders Fugmann
> 
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
*** Searching for new projects! Need consulting for system
*** programming in Ocaml? Gerd Stolpmann can help you.
------------------------------------------------------------

Re: [Caml-list] Netmcore problems.

[Anders Fugmann]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

Thank you very much. Remounting with size=80% solved the problem.

It would seem that on debian, /run/shm (which is symlinked from
/dev/shm) is mounted with size=20%.. Quite conservative it would seem.

Again, thanks.
Anders

- --
Anders Fugmann

On 2012-01-13 18:06, Gerd Stolpmann wrote:
> Am Freitag, den 13.01.2012, 17:30 +0100 schrieb Anders Fugmann:
>> Hi,
>>
>> I'm having a problem using netmcore from ocamlnet 3.4 library.
>>
>> When I try to place a large data structure into Netmcore_array, the 
>> controlling process dies with the following error:
>>
>> [Fri Jan 13 16:21:38 2012] [netplex.controller] [alert] Process 28067 
>> for service netmcore_0 terminated with signal 7
> 
> That's sigbus. You get it when the OS does not have memory anymore - but
> not when the memory is allocated, but first when it is filled with data.
> You can change this by disallowing that memory is overcommitted (set the
> kernel param vm.overcommit_memory to 1 or even 2).
> 
>> The error occurs while trying to initialise a Netmcore_array from an 
>> array of 4*10^6 strings all of length 150. This is done in the context 
>> of the "first" process.
>>
>> The controlling process initializes a memory pool by using
>>
>>    let pool_id = Netmcore_mempool.create_mempool (1024 * 1024 * 1024 * 
>> 10) in
>>
>> Running the code on a smaller dataset seems to work.
>>
>> I have increased kernel variables shmall and shmmax to very high numbers 
>> (100Gb), but it does not solve the problem.
> 
> No, this cannot solve it. These variables only control System V shared
> memory, but Netmcore uses POSIX shared memory. On Linux, you can change
> the max of this memory by re-mounting /dev/shm, e.g.
> 
> mount -o remount,size=80% /dev/shm
> 
> The default is 50% of available RAM.
> 
>> All ideas are welcome.
>>
>> On other question while I'm at it; am I allowed to create multiple 
>> shared arrays from the same memory pool, or do I need to create one pool 
>> for each shared array?
> 
> The pools can be shared.
> 
> Gerd
> 
>>
>> Regards
>> Anders Fugmann
>>
>>
>>
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk8UYo0ACgkQ9GexfkaZ7hSfLwCgiu+grFYsZXyQlvvnUBZgna+4
twkAoJKyNJ0hXlI0u9IqmuapqQ48YGDz
=RCbL
-----END PGP SIGNATURE-----