Dynlinking plugins defining the same unit name but with different implementations.

Dynlinking plugins defining the same unit name but with different implementations.

[Daniel Bünzli]

Hello,

On osx 10.6.4 with ocaml 3.12.0+beta1

Suppose I have two following two plugins

a.cmxs, containing a/a.cmx using M.test and a/m.cmx defining M.test
b.cmxs, containing b/b.cmx using M.test and M.test2 and b/m.cmx
defining M.test and M.test2

of course a/m.cmx and b/m.cmx are different implementations except
they define the same interface for M.test

If I Dynlink.loadfile or Dynlink.loadfile_private first a.cmxs and
then b.cmxs there are no errors reported and b uses M.test of a/m.cmx
but M.test2 of b/m.cmx (since the m.cmx in a.cmxs doesn't define it)

Is that expected behaviour ? That looks like dll hell.

Best,

Daniel

P.S. Here's how the plugins are constructed and an invocation of a
test program to give an idea :

> > ocamlbuild a.cmxs b.cmxs test.native
> /usr/local/bin/ocamlopt.opt -shared a/m.cmx a/a.cmx -o a/a.cmxs
> /usr/local/bin/ocamlopt.opt -shared b/m.cmx b/b.cmx -o b/b.cmxs

> > ./test.native
> A using M.test in a/m.ml
> B using M.test in a/m.ml and M.test2 b/m.ml

The last line should be

> B using M.test in b/ml.ml and M.test2 b/m.ml

or some kind of error should be reported.

Re: Dynlinking plugins defining the same unit name but with different implementations.

[Daniel Bünzli]

> Is that expected behaviour ? That looks like dll hell.

Well in fact this looks like dll bug since if the interface of M.test
doesn't match in a/m.cmx and b/m.cmx then a segfault occurs. Something
similar is reported here :

http://caml.inria.fr/mantis/view.php?id=4839
http://caml.inria.fr/mantis/view.php?id=4229

Is there any fix expected for the final 3.12 ? As this makes plugins
quite an unreliable feature (if you expect your plugins to come from
different sources).

Best,

Daniel

Re: [Caml-list] Re: Dynlinking plugins defining the same unit name but with different implementations.

[Alain Frisch]

On 07/01/2010 09:16 PM, Daniel Bünzli wrote:
> Well in fact this looks like dll bug since if the interface of M.test
> doesn't match in a/m.cmx and b/m.cmx then a segfault occurs. Something
> similar is reported here :
>
> http://caml.inria.fr/mantis/view.php?id=4839
> http://caml.inria.fr/mantis/view.php?id=4229
>
> Is there any fix expected for the final 3.12 ? As this makes plugins
> quite an unreliable feature (if you expect your plugins to come from
> different sources).

Another related bug:

http://caml.inria.fr/mantis/view.php?id=4231

This one shows that the same problem (linking modules with the same name 
several times, but different implementations, thus breaking the type 
system) can appear in the following situations:

- Static linking with ocamlc (no check at all that two modules with the 
same name are not linked together).

- Static linking with ocamlopt (there is a check that the global symbols 
corresponding to module themselves are unique; but it can be broken with 
-for-pack and -pack).

So the problem is more general than dynlink.


(Note: for dynlink, I believe that loading modules in private mode 
should be safe.)


Alain

Re: [Caml-list] Re: Dynlinking plugins defining the same unit name but with different implementations.

[Daniel Bünzli]

> (Note: for dynlink, I believe that loading modules in private mode should be safe.)

No, at least not in 3.12.0+beta1.

Daniel

Re: [Caml-list] Re: Dynlinking plugins defining the same unit name but with different implementations.

[Alain Frisch]

On 07/02/2010 10:27 AM, Daniel Bünzli wrote:
>> (Note: for dynlink, I believe that loading modules in private mode should be safe.)
>
> No, at least not in 3.12.0+beta1.

Ah yes, sorry, I did not ready your original post carefully enough.

I think the problem is that dlopen is called (in loadfile_private mode), 
without RTLD_GLOBAL (but also without RTLD_LOCAL, which is never used).

According to:

http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man3/dlopen.3.html

the default is RTLD_GLOBAL under Mac OS, contrary to Linux (default is 
RTLD_LOCAL).


I'll try to apply a fix.


Alain

Re: [Caml-list] Re: Dynlinking plugins defining the same unit name but with different implementations.

[Alain Frisch]

On 07/02/2010 10:40 AM, Alain Frisch wrote:
> I'll try to apply a fix.

I've committed a tentative fix (version/3.12 on the SVN repository). Can 
you try it and see if it solves your problem under Mac OS ?


Alain

Re: [Caml-list] Re: Dynlinking plugins defining the same unit name but with different implementations.

[Daniel Bünzli]

> Can you try it and see if it solves your problem under Mac OS ?

It does. Thanks for the fix. Btw. dynlink with first class modules is cool.

Best,

Daniel

Re: [Caml-list] Re: Dynlinking plugins defining the same unit name but with different implementations.

[Daniel Bünzli]

Now on Linux 2.6.32-23-generic #37-Ubuntu SMP Fri Jun 11 07:54:58 UTC
2010 i686 GNU/Linux

with exactly the same code that know works on osx and ocaml 3.12 from
svn. I get the following problem when running my test program.

error loading shared library:
/home/dbuenzli/sync/dynlinkt/_build/a/a.cmxs: undefined symbol:
camlPervasives__$5e_1112
error loading shared library:
/home/dbuenzli/sync/dynlinkt/_build/b/b.cmxs: undefined symbol:
camlPervasives__$5e_1112

I tried to add /usr/local/lib/ocaml to LD_LIBRARY_PATH but without
success. Any hint ? Is that a bug or am I missing something ?

Thanks for your help,

Daniel

P.S. Here's how these things are compiled by ocamlbuild

ocamlbuild -classic-display a.cmxs b.cmxs test.native
/usr/local/bin/ocamldep.opt -modules a/a.ml > a/a.ml.depends
/usr/local/bin/ocamldep.opt -modules a/m.ml > a/m.ml.depends
/usr/local/bin/ocamldep.opt -modules plugin.ml > plugin.ml.depends
/usr/local/bin/ocamlc.opt -c -I a -I c -I b -o a/m.cmo a/m.ml
/usr/local/bin/ocamlc.opt -c -I c -I a -I b -o plugin.cmo plugin.ml
/usr/local/bin/ocamlc.opt -c -I a -I c -I b -o a/a.cmo a/a.ml
/usr/local/bin/ocamlopt.opt -c -I a -I c -I b -o a/m.cmx a/m.ml
/usr/local/bin/ocamlopt.opt -c -I c -I a -I b -o plugin.cmx plugin.ml
/usr/local/bin/ocamlopt.opt -c -I a -I c -I b -o a/a.cmx a/a.ml
/usr/local/bin/ocamlopt.opt -shared a/m.cmx a/a.cmx -o a/a.cmxs
/usr/local/bin/ocamldep.opt -modules b/b.ml > b/b.ml.depends
/usr/local/bin/ocamldep.opt -modules b/m.ml > b/m.ml.depends
/usr/local/bin/ocamlc.opt -c -I b -I c -I a -o b/m.cmo b/m.ml
/usr/local/bin/ocamlc.opt -c -I b -I c -I a -o b/b.cmo b/b.ml
/usr/local/bin/ocamlopt.opt -c -I b -I c -I a -o b/m.cmx b/m.ml
/usr/local/bin/ocamlopt.opt -c -I b -I c -I a -o b/b.cmx b/b.ml
/usr/local/bin/ocamlopt.opt -shared b/m.cmx b/b.cmx -o b/b.cmxs
/usr/local/bin/ocamldep.opt -modules test.ml > test.ml.depends
/usr/local/bin/ocamlc.opt -c -I c -I a -I b -o test.cmo test.ml
/usr/local/bin/ocamlopt.opt -c -I c -I a -I b -o test.cmx test.ml
/usr/local/bin/ocamlopt.opt dynlink.cmxa plugin.cmx test.cmx -o test.native

Re: [Caml-list] Re: Dynlinking plugins defining the same unit name but with different implementations.

[Alain Frisch]

On 07/02/2010 07:48 PM, Daniel Bünzli wrote:
> Now on Linux 2.6.32-23-generic #37-Ubuntu SMP Fri Jun 11 07:54:58 UTC
> 2010 i686 GNU/Linux
>
> with exactly the same code that know works on osx and ocaml 3.12 from
> svn. I get the following problem when running my test program.
>
> error loading shared library:
> /home/dbuenzli/sync/dynlinkt/_build/a/a.cmxs: undefined symbol:
> camlPervasives__$5e_1112
> error loading shared library:
> /home/dbuenzli/sync/dynlinkt/_build/b/b.cmxs: undefined symbol:
> camlPervasives__$5e_1112
>
> I tried to add /usr/local/lib/ocaml to LD_LIBRARY_PATH but without
> success. Any hint ? Is that a bug or am I missing something ?

Could you double-check that you do not mix files compiled with 
different versions?  Then, please send a reproduction case (with source 
code and build instructions) to the Caml bug tracker.

Thanks!

Alain

Re: [Caml-list] Re: Dynlinking plugins defining the same unit name but with different implementations.

[Daniel Bünzli]

> Could you double-check that you do not mix files compiled with different
> versions?

Yes, that's impossible it was on a clean system. Here's another error
I get in another example :

error loading shared library: ...: undefined symbol: caml_young_ptr

Is the whole runtime system missing to them ? Note that these errors
happen with both loadfile and loadfile_private

> Then, please send a reproduction case (with source code and build
> instructions) to the Caml bug tracker.

Will do this evening.

Best,

Daniel

Re: [Caml-list] Re: Dynlinking plugins defining the same unit name but with different implementations.

[Alain Frisch]

On 07/02/2010 07:48 PM, Daniel Bünzli wrote:
> I tried to add /usr/local/lib/ocaml to LD_LIBRARY_PATH but without
> success. Any hint ? Is that a bug or am I missing something ?

As a side node , you should also always link the main program with 
-linkall (to make sure all the modules that plugins might want to use 
are indeed present in the main program).


Alain