Re: [Caml-list] Accelerating compilation

[oleg@okmij.org]

> Or are there compelling arguments in favor of using the bytecode version 
> of ocamlc/ocamlopt?

I have come across one case of the meaningful difference between
ocamlc and ocamlc.opt: a particular project can only be linked on
MacOS using ocamlc; ocamlc.opt fails to link. Admittedly, the real
problem is in what I think a hard-to-justify design decision in
OCaml linker.

The problem occurs when linking byte-compiled executables that use the
delimcc library. I will only talk about byte-compiled version of
delimcc in the message (there are no problems for the native version).

The following message
        https://sympa.inria.fr/sympa/arc/caml-list/2013-04/msg00072.html
gives more detail.

Similar to the nums library, delimcc includes some C code, which is
arranged in a shared library libdelimcc.so. That shared library refers
to a particular symbol caml_realloc_stack, which is provided by the
byte-code run-time system (that is, ocamlrun). When we invoke the
byte-code executable that uses delimcc, ocamlrun loads libdelimcc.so,
the library looks for the symbol caml_realloc_stack. Since ocamlrun is
the running executable and its provides the symbol, the resolution
succeeds and everyone is happy.

The problem occurs when linking the executable that uses delimcc. When
OCaml linker processes delimcc.cma, it notices that a shared library
is required. The OCaml linker then loads the library *and forces the
resolution of all undefined references there*. Therefore, the symbol
caml_realloc_stack required by libdelimcc.so has to be found. When the
linking is done by ocamlc, then the running executable is ocamlrun,
which provides the symbol. However, if the linking is done by
ocamlc.opt, it has a different run-time that does not provide the
symbol. Linking fails. Therefore, although ocamlc.opt can be used to
compile projects with delimcc, the final linking step must be done
with ocamlc rather than ocamlc.opt

I managed to get around the problem using a weak reference. Alas, the
subterfuge does not seem to work on Mac OS.

The real problem in my view is the strange decision to load shared
libraries at link time and force the resolution of their undefined
references. This decision certainly makes linking slower, without
providing much benefit, it seems. After all, if the resolution
succeeded at link time, it may still fail at run time since the
linked executable can be run in a different location or even a
different computer.

So, the real problem to me is ocamlc using RTLD_NOW flag when loading
shared library. Removing the flag would make linking faster, and less
painful.

A few technical details: dlopen with the problematic flag occurs in
the function caml_dlopen in the file byterun/unix.c

void * caml_dlopen(char * libname, int for_execution, int global)
{
  return dlopen(libname, RTLD_NOW | (global ? RTLD_GLOBAL : RTLD_LOCAL) | RTLD_NODELETE);
  /* Could use RTLD_LAZY if for_execution == 0, but needs testing */
}

That function caml_dlopen is used within caml_dynlink_open_lib,
which, under alias dll_open, is called in Dll.open_dll. The latter
function is invoked in Bytelink.link_bytecode.