memory management, allocation and collection in kernel mode.

memory management, allocation and collection in kernel mode.

[Sussillo, David]

Hello,


Est ce qu'il ya quelqun qui sait ce que j'ai besoin de changer dans le "   
 OCaml memory management system " pour que le "program" funcion bien dans   
le "kernel" mode?

 ------------------------------

Does anybody have an idea what would need to change in the OCaml memory   
management system in order for an OCaml program to function properly in   
kernel mode?  (Linux on a PII).  Linux has a physically mapped kernel   
space because it doesn' make much sense for a kernel to manage it's own   
virtual memory, so it just ignores the whole problem by ignoring virtual   
addressing.  What that means is one has to deal with a physical   
addressing scheme if one wants to run in kernel/system mode.

I've not looke at the OCaml sources at all, but I would assume that the   
memory management routines are assuming that whatever process they are   
running in is running in user mode.  At this point I don't understand the   
ramifications of this fact.

As far as I can tell, there are a couple of alternatives:
   a .   The allocation and GC routines might not even think about the   
difference between user mode and system mode at all since memory is   
segmented and each process is running in it's own address space. However,   
if one was to run these routines in the kernel, then it would be   
important to 'gel' with the way memory is allocated in a kernel, at the   
very least, an offset of something like 0xffff0000.
   b.    The allocation and GC rountines take full advantage of running   
in their own address space (and segment) and put memory all over the   
place.  In which case, one would be hard pressed to figure out how to   
make it work the memory management machinery work in kernel mode.

So then there are a few questions:
1.  Will someone point out to me where I might look in the compiler   
source for all of the memory management routines, allocataion and GC? I   
guess it's important to see how the compiler reasons about the memory   
requirements of OCaml structures, as well as the emitted code (or   
libraries?) that it using during runtime.
   

2. If anyone has any insight into the memory management ramifications of   
running an OCaml program in kernel mode, might they elucidate? As I'm   
just starting out, I'm interested in both general and specific remarks.

Thanks,
 -dave

Re: memory management, allocation and collection in kernel mode.

[Stefan Monnier]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1255 bytes --]

>>>>> "Sussillo," == Sussillo, David <sussillo@ftintl.com> writes:
> Est ce qu'il ya quelqun qui sait ce que j'ai besoin de changer dans le "   
>  OCaml memory management system " pour que le "program" funcion bien dans   
> le "kernel" mode?

Problèmes:

1 - l'allocation de mémoire dans le kernel se fait généralement par page
    (c'est difficile est surtout coûteux d'allouer plusieurs pages
    consécutives).  Donc le GC doit garder trace de tout plein de pages
    un peu partout.

2 - l'adressage physique signifie qu'il n'y a pas de swapping.
    Donc si tu alloues 32MB et que tu ne les utilises pas, c'est 32MB
    temporairement inutilisables (plutôt que 32MB swappés sur disque
    où ils n'ennuient personne).  Il est donc désirable d'utiliser très peu de
    mémoire.  Un GC a tendance à gaspiller la mémoire un peu pour diminuer la
    fréquence des collections.

3 - le kernel n'est souvent pas préemptible (?), donc il faudra faire attention
    à appeler explicitement le yield() de temps à autre pour ne pas perdre trop
    d'interruptions et ne pas rendre la machine unresponsive.  O'Caml utilise
    déjà un GC incrémental si je ne me trompe, donc c'est peut-être pas trop
    difficile (famous last words).

4 - ...


-- Stefan

Re: memory management, allocation and collection in kernel mode.

[Mark Hayden]

It is not clear to me exactly what you are asking about,
however, I can tell that it is possible to run Ocaml
code in the Linux kernel.  I got an Ocaml program to run
as a dynamically loadable kernel module without
too many problems.  As far as memory management is concerned,
the approach I took was to intercept the calls to malloc()
that Ocaml made to add new heap segments.

--Mark

"Sussillo, David" wrote:
> 
> Hello,
> 
> Est ce qu'il ya quelqun qui sait ce que j'ai besoin de changer dans le "
>  OCaml memory management system " pour que le "program" funcion bien dans
> le "kernel" mode?
> 
>  ------------------------------
> 
> Does anybody have an idea what would need to change in the OCaml memory
> management system in order for an OCaml program to function properly in
> kernel mode?  (Linux on a PII).  Linux has a physically mapped kernel
> space because it doesn' make much sense for a kernel to manage it's own
> virtual memory, so it just ignores the whole problem by ignoring virtual
> addressing.  What that means is one has to deal with a physical
> addressing scheme if one wants to run in kernel/system mode.
> 
> I've not looke at the OCaml sources at all, but I would assume that the
> memory management routines are assuming that whatever process they are
> running in is running in user mode.  At this point I don't understand the
> ramifications of this fact.
> 
> As far as I can tell, there are a couple of alternatives:
>    a .   The allocation and GC routines might not even think about the
> difference between user mode and system mode at all since memory is
> segmented and each process is running in it's own address space. However,
> if one was to run these routines in the kernel, then it would be
> important to 'gel' with the way memory is allocated in a kernel, at the
> very least, an offset of something like 0xffff0000.
>    b.    The allocation and GC rountines take full advantage of running
> in their own address space (and segment) and put memory all over the
> place.  In which case, one would be hard pressed to figure out how to
> make it work the memory management machinery work in kernel mode.
> 
> So then there are a few questions:
> 1.  Will someone point out to me where I might look in the compiler
> source for all of the memory management routines, allocataion and GC? I
> guess it's important to see how the compiler reasons about the memory
> requirements of OCaml structures, as well as the emitted code (or
> libraries?) that it using during runtime.
> 
> 
> 2. If anyone has any insight into the memory management ramifications of
> running an OCaml program in kernel mode, might they elucidate? As I'm
> just starting out, I'm interested in both general and specific remarks.
> 
> Thanks,
>  -dave

Re: memory management, allocation and collection in kernel mode.

[doligez]

>From: "Sussillo, David" <sussillo@ftintl.com>

>Does anybody have an idea what would need to change in the OCaml memory   
>management system in order for an OCaml program to function properly in   
>kernel mode?

My first answer would be nothing at all if the kernel has a malloc
function.


>1.  Will someone point out to me where I might look in the compiler   
>source for all of the memory management routines, allocataion and GC? I   
>guess it's important to see how the compiler reasons about the memory   
>requirements of OCaml structures, as well as the emitted code (or   
>libraries?) that it using during runtime.

I guess you should start with the byte-code interpreter, and when
that's working, you can tackle the native code stuff.  The files to
look for are all in byterun:

alloc.[ch]
compact.[ch]
freelist.[ch]
gc.h
gc_ctrl.[ch]
major_gc.[ch]
minor_gc.[ch]
mlvalues.h
roots.[ch]
stacks.[ch]
weak.[ch]


>2. If anyone has any insight into the memory management ramifications of   
>running an OCaml program in kernel mode, might they elucidate? As I'm   
>just starting out, I'm interested in both general and specific remarks.

The memory management should be easy if you have malloc.  For the
native code system, you might have to do something for the stack.

Stefan Monnier made a very good remark:
>3 - le kernel n'est souvent pas priemptible (?), donc il faudra faire attention
>    ` appeler explicitement le yield() de temps ` autre pour ne pas perdre trop
>    d'interruptions et ne pas rendre la machine unresponsive.  O'Caml utilise
>    dij` un GC incrimental si je ne me trompe, donc c'est peut-jtre pas trop
>    difficile (famous last words).

Translation: since kernel code is never preempted, you'll have to call
yield() from time to time.

Fortunately, that is exactly what the Macintosh version does for
cooperative multi-tasking.  Look for "#if macintosh" in interp.c and
replace the call to rotatecursor_action() with yield().  That should
be enough.  For native code, I don't have an easy answer.


-- Damien