Re: [Caml-list] Accelerating compilation

[Gerd Stolpmann <info@gerd-stolpmann.de>]

[-- Attachment #1: Type: text/plain, Size: 4249 bytes --]

Good point. However, I think without RTLD_NOW there is not enough
protection against errors. The better solution would be if ocamlc.opt
spawned a child process that is a real byterun executable, and that does
the loading check for it. For simplicity, this executable should be the
just created executable, and an environment variable requests to do the
load check only (no real execution beyond that). NB. The "ldd" utility
on Linux uses exactly the same trick.

Gerd

Am Dienstag, den 10.09.2013, 02:01 +0000 schrieb oleg@okmij.org:
> > Or are there compelling arguments in favor of using the bytecode version 
> > of ocamlc/ocamlopt?
> 
> I have come across one case of the meaningful difference between
> ocamlc and ocamlc.opt: a particular project can only be linked on
> MacOS using ocamlc; ocamlc.opt fails to link. Admittedly, the real
> problem is in what I think a hard-to-justify design decision in
> OCaml linker.
> 
> The problem occurs when linking byte-compiled executables that use the
> delimcc library. I will only talk about byte-compiled version of
> delimcc in the message (there are no problems for the native version).
> 
> The following message
>         https://sympa.inria.fr/sympa/arc/caml-list/2013-04/msg00072.html
> gives more detail.
> 
> Similar to the nums library, delimcc includes some C code, which is
> arranged in a shared library libdelimcc.so. That shared library refers
> to a particular symbol caml_realloc_stack, which is provided by the
> byte-code run-time system (that is, ocamlrun). When we invoke the
> byte-code executable that uses delimcc, ocamlrun loads libdelimcc.so,
> the library looks for the symbol caml_realloc_stack. Since ocamlrun is
> the running executable and its provides the symbol, the resolution
> succeeds and everyone is happy.
> 
> The problem occurs when linking the executable that uses delimcc. When
> OCaml linker processes delimcc.cma, it notices that a shared library
> is required. The OCaml linker then loads the library *and forces the
> resolution of all undefined references there*. Therefore, the symbol
> caml_realloc_stack required by libdelimcc.so has to be found. When the
> linking is done by ocamlc, then the running executable is ocamlrun,
> which provides the symbol. However, if the linking is done by
> ocamlc.opt, it has a different run-time that does not provide the
> symbol. Linking fails. Therefore, although ocamlc.opt can be used to
> compile projects with delimcc, the final linking step must be done
> with ocamlc rather than ocamlc.opt
> 
> I managed to get around the problem using a weak reference. Alas, the
> subterfuge does not seem to work on Mac OS.
> 
> The real problem in my view is the strange decision to load shared
> libraries at link time and force the resolution of their undefined
> references. This decision certainly makes linking slower, without
> providing much benefit, it seems. After all, if the resolution
> succeeded at link time, it may still fail at run time since the
> linked executable can be run in a different location or even a
> different computer.
> 
> So, the real problem to me is ocamlc using RTLD_NOW flag when loading
> shared library. Removing the flag would make linking faster, and less
> painful.
> 
> A few technical details: dlopen with the problematic flag occurs in
> the function caml_dlopen in the file byterun/unix.c
> 
> void * caml_dlopen(char * libname, int for_execution, int global)
> {
>   return dlopen(libname, RTLD_NOW | (global ? RTLD_GLOBAL : RTLD_LOCAL) | RTLD_NODELETE);
>   /* Could use RTLD_LAZY if for_execution == 0, but needs testing */
> }
> 
> That function caml_dlopen is used within caml_dynlink_open_lib,
> which, under alias dll_open, is called in Dll.open_dll. The latter
> function is invoked in Bytelink.link_bytecode.
> 
> 
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]