PIC

PIC

[skaller]

I notice Ocaml 3.09 has -fPIC option. Thanks! This is a step
towards dynamic loading in native code. 

I wonder, is it actually supported -- or even possible 
to load native code (on suitable platforms) at startup
from C? From an Ocaml mainline? 

What's left to be able to 'dlopen()' a function library?
>From C? From Ocaml? Will that ever be possible?

[Another question, whether one can wrap native code
so it can be called from bytecode.. this would allow
bytecode programs to generate and compile bytecode
extensions .. without the bulk of the code running
slowly]

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] PIC

[Xavier Leroy]

> I notice Ocaml 3.09 has -fPIC option. Thanks! This is a step
> towards dynamic loading in native code.
>
> I wonder, is it actually supported -- or even possible
> to load native code (on suitable platforms) at startup
> from C? From an Ocaml mainline?
>
> What's left to be able to 'dlopen()' a function library?
> From C? From Ocaml? Will that ever be possible?

Encapsulation of compiled OCaml code as a shared library that can be
called from C is possible, on x86 at least.

The procedure is similar to that described in section 18.8 of the
reference manual, namely, use "ocamlc -output-obj" or
"ocamlopt -output-obj" to package the Caml code as a .o file, then use
"gcc -shared" (or equivalent commands to build shared objects) to group
together this .o file, the C stub code that calls back into OCaml, and
the OCaml run-time system.

This is a very good approach to use OCaml code from other languages
(e.g. Java) or as plug-ins (e.g. in Apache).

The reason it works particularly well for x86 is that dynamic loaders
for x86 (both under Unix/Linux and Windows) can relocate arbitrary
code, not necessarily PIC code.  This is unfortunately not the case
for all target architectures.  In particular, dynamic loaders for
x86_64 (AMD64) are much less forgiving.  The -fPIC option to ocamlopt
you mention was added to the AMD64 code generator in an attempt to
generate "more position-independent" code.  However, it does not quite
work yet.  A difficulty is the paucity of documentation on what,
exactly, the Linux AMD64 dynamic loader can relocate.

Dynamic loading of OCaml code from OCaml code raises many other
issues.  It is currently supported for bytecode only, and will not be
available for native code in the forseeable future.  I have already
discussed this on this list earlier.

> [Another question, whether one can wrap native code
> so it can be called from bytecode.. this would allow
> bytecode programs to generate and compile bytecode
> extensions .. without the bulk of the code running
> slowly]

Again, I believe this has been discussed at length before.  The short
answer is: no.  A longer answer is the asmdynlink library by Fabrice
Le Fessant, which (used to) enable this kind of things with some speed
penalty on the interpretation of the bytecode.

- Xavier Leroy

Re: [Caml-list] PIC

[skaller]

On Tue, 2005-12-27 at 11:03 +0100, Xavier Leroy wrote:
> > I notice Ocaml 3.09 has -fPIC option. Thanks! This is a step
> > towards dynamic loading in native code.

Thanks for your summary!

> Encapsulation of compiled OCaml code as a shared library that can be
> called from C is possible, on x86 at least.

[]

> This is a very good approach to use OCaml code from other languages
> (e.g. Java) or as plug-ins (e.g. in Apache).

Indeed! If I understand correctly, this requires redistribution
of "LGPL with Linking Exception" parts of Ocaml only?

> The reason it works particularly well for x86 is that dynamic loaders
> for x86 (both under Unix/Linux and Windows) can relocate arbitrary
> code, not necessarily PIC code.  This is unfortunately not the case
> for all target architectures.  

> In particular, dynamic loaders for
> x86_64 (AMD64) are much less forgiving.  

I suspect you're being polite .. 'broken' seems to the
correct term ;( The same techniques should work for the
x86_64 as the x86 .. just making sure 64 bit values are
supported (by the loader and ELF formats) should be enough.

> The -fPIC option to ocamlopt
> you mention was added to the AMD64 code generator in an attempt to
> generate "more position-independent" code.  However, it does not quite
> work yet.  A difficulty is the paucity of documentation on what,
> exactly, the Linux AMD64 dynamic loader can relocate.

I see. Ok, thanks for that info. Having looked into this
myself .. I'm not surprised ..  Sigh. Is the ia64 any better?

> Dynamic loading of OCaml code from OCaml code raises many other
> issues. 

Type safety, version compatibility, etc being a few.
But one can work around this problem, perhaps messily and
riskily, if the codes can be loaded and connected together
with C glue.

>  It is currently supported for bytecode only, and will not be
> available for native code in the forseeable future.  I have already
> discussed this on this list earlier.

Yes, thanks. I hope I covered the combinations reasonably
well, so your responses are fairly clear summary of the
situation, which is what I was looking for: thanks!

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] PIC

[Sven Luther]

On Tue, Dec 27, 2005 at 11:03:24AM +0100, Xavier Leroy wrote:
> The reason it works particularly well for x86 is that dynamic loaders
> for x86 (both under Unix/Linux and Windows) can relocate arbitrary
> code, not necessarily PIC code.  This is unfortunately not the case
> for all target architectures.  In particular, dynamic loaders for
> x86_64 (AMD64) are much less forgiving.  The -fPIC option to ocamlopt
> you mention was added to the AMD64 code generator in an attempt to
> generate "more position-independent" code.  However, it does not quite
> work yet.  A difficulty is the paucity of documentation on what,
> exactly, the Linux AMD64 dynamic loader can relocate.

What is the situation for other architectures, in particular, i would be
interested in the powerpc case.

Friendly,

Sven Luther

Re: [Caml-list] PIC

[David Fox]

Xavier Leroy wrote:

>Dynamic loading of OCaml code from OCaml code raises many other
>issues.  It is currently supported for bytecode only, and will not be
>available for native code in the forseeable future.  I have already
>discussed this on this list earlier.
>  
>
I can seem to locate this discussion - do you remember any keywords I 
might search for?

Re: [Caml-list] PIC

[Ker Lutyn]

<http://groups.google.com/groups?q=group:fa.caml+author:xavier+loading+native>

--- David Fox <david.fox@linspire.com> wrote:

> Xavier Leroy wrote:
> 
> >Dynamic loading of OCaml code from OCaml code raises many other
> >issues.  It is currently supported for bytecode only, and will not be
> >available for native code in the forseeable future.  I have already
> >discussed this on this list earlier.
> >  
> >
> I can seem to locate this discussion - do you remember any keywords I 
> might search for?
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 



		
__________________________________________ 
Yahoo! DSL – Something to write home about. 
Just $16.99/mo. or less. 
dsl.yahoo.com 

Re: [Caml-list] PIC

[Nathaniel Gray]

Maybe these links will be more useful:
http://groups.google.com/group/fa.caml/browse_thread/thread/2827ec9d553e7760/183ac101fef47132?lnk=st&q=group%3Afa.caml+author%3Axavier+loading+native&rnum=10#183ac101fef47132
http://groups.google.com/group/fa.caml/browse_thread/thread/efc434c41cc88ded/663f229e008648e0?lnk=st&q=group%3Afa.caml+author%3Axavier+binary+compatibility&rnum=4#663f229e008648e0

In the first (newer) post linked above Xavier says:

"""
Feature 3- (dynamic code loading) is already available in bytecode
through the Dynlink API.  I understand there's a demand for having it
in native-code as well, and that might be possible without too much fuss,
at least on selected operating systems.
"""

This seems to conflict with Xavier's statements earlier in this thread.  I
hope dynamic loading is still being considered, as I can't see using
OCaml for medium-to-large scale applications programming without it.
Who wants to write an app without plugins these days?

Cheers,
-n8

On 1/1/06, Ker Lutyn <ker527mail@yahoo.com> wrote:
> <http://groups.google.com/groups?q=group:fa.caml+author:xavier+loading+native>
>
> --- David Fox <david.fox@linspire.com> wrote:
>
> > Xavier Leroy wrote:
> >
> > >Dynamic loading of OCaml code from OCaml code raises many other
> > >issues.  It is currently supported for bytecode only, and will not be
> > >available for native code in the forseeable future.  I have already
> > >discussed this on this list earlier.
> > >
> > >
> > I can seem to locate this discussion - do you remember any keywords I
> > might search for?
> >

--
>>>-- Nathaniel Gray -- Caltech Computer Science ------>
>>>-- Mojave Project -- http://mojave.cs.caltech.edu -->

Re: [Caml-list] PIC

[skaller]

On Tue, 2006-01-03 at 15:27 -0800, Nathaniel Gray wrote:

> Who wants to write an app without plugins these days?

Who is willing to write applications that are unsound?

Once Ocaml generates -fPIC it can be linked with a C wrapper
to make a shared library, and your mainline (or another
wrapped library) can embed a function that loads such 
libraries with dlopen() and retrieves a record of functions, 
then casts it to the expected type .. 
voila, you have native code plugins.

Note even bytecode cannot *unload* plugins .. which is
needed to execute user scripts in web pages with 
long running server side scripting support. 

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Re: [Caml-list] PIC

[Nathaniel Gray]

On 1/3/06, skaller <skaller@users.sourceforge.net> wrote:
> On Tue, 2006-01-03 at 15:27 -0800, Nathaniel Gray wrote:
>
> > Who wants to write an app without plugins these days?
>
> Who is willing to write applications that are unsound?
>
> Once Ocaml generates -fPIC it can be linked with a C wrapper
> to make a shared library, and your mainline (or another
> wrapped library) can embed a function that loads such
> libraries with dlopen() and retrieves a record of functions,
> then casts it to the expected type ..
> voila, you have native code plugins.

This could solve part of the problem, namely alleviating the need for
recompiling the app for each plugin the user adds.  However, given the
OCaml team's indifference towards binary compatibility, trying to keep
everybody on the same version of the compiler would be too restrictive
to be practical.

-n8

--
>>>-- Nathaniel Gray -- Caltech Computer Science ------>
>>>-- Mojave Project -- http://mojave.cs.caltech.edu -->