[Caml-list] Separate compilation

[Caml-list] Separate compilation

[Oleg]

In designing separate compilation, OCaml has decided to restrict the
correspondence between separately compiled modules and their
interfaces to one-to-one. The restriction is indeed limiting: making
the linking with alternative/improved implementations, and the
implementation extension/evolution ungainly (in fact, seemingly
impossible).

To be concrete: suppose we have an interface A.mli and two implementations 
of it: A1.ml and A2.ml. We also have the user code B.ml:
        open A
        ... using the operations of A
We want to compile B.ml and link that B.cmo with either
implementation. As posed, the problem seem unsolvable: we need either
modify the source B.ml to refer to a particular implementation, or turn
B.ml into a functor (which could be inconvenient, and also suboptimal:
calls to A operations become indirect). 

For the next problem, assume the interface U.mli and its
implementation U.ml. We want to extend them -- say, add a new
operation to the interface and the implementation -- without modifying
U.mli/U.ml and without cut-and-paste. That is, we want merely `include'
U and add the new operation -- making a `diff' so to speak. If U.mli
abstracts away the implementation details, and if they are needed to
implement the new operation, the problem seem unsolvable.

It turns out there are work-arounds for both problems, which solve
them, albeit not elegantly. The full explanation is a bit too long for
this message; please see
        https://okmij.org/ftp/ML/module-extensibility.html

The work-arounds are simple. Should OCaml developers be so inclined, they
may be incorporated in OCaml.

Re: [Caml-list] Separate compilation

[Edwin Török]

On Sat, 2025-04-05 at 13:26 +0900, Oleg wrote:
> 
> In designing separate compilation, OCaml has decided to restrict the
> correspondence between separately compiled modules and their
> interfaces to one-to-one. The restriction is indeed limiting: making
> the linking with alternative/improved implementations, and the
> implementation extension/evolution ungainly (in fact, seemingly
> impossible).
> 
> To be concrete: suppose we have an interface A.mli and two
> implementations 
> of it: A1.ml and A2.ml. We also have the user code B.ml:
>         open A
>         ... using the operations of A
> We want to compile B.ml and link that B.cmo with either
> implementation. As posed, the problem seem unsolvable: we need either
> modify the source B.ml to refer to a particular implementation, or
> turn
> B.ml into a functor (which could be inconvenient, and also
> suboptimal:
> calls to A operations become indirect). 

Thanks for exploring solutions to this issue.

FWIW this is what Mirage does, but the functors spread across the
codebase and can become quite big, e.g. a tutorial HTTP application has
a functor that needs 5 modules:
https://github.com/mirage/mirage-skeleton/blob/main/applications/http/unikernel.ml#L32

This is the code to compose it all (IIUC it runs at build time):
https://github.com/mirage/mirage-skeleton/blob/main/applications/http/config.ml


An alternative is Dune with virtual libraries:
https://dune.readthedocs.io/en/stable/virtual-libraries.html
This doesn't require functors, but it prevents cross-module inlining
from working at all
(indeed the actual implementation may only be known at the time the
final application is linked). 
However the module chosen *is* still known at application build time,
so it is a bit unfortunate to have to chose between functors (where
perhaps Flambda2 can help remove the overhead), or the loss of inlining
(not sure if Flambda2 can help with this one due -opaque).

Nowadays even GCC does LTO by default at application link time on some
systems (which in certain cases also increases OCaml link times
considerably as the entire OCaml C runtime is processed by LTO).
We already have separate development and release build profiles (in
Dune) that make different tradeoffs between compilation speed and
application runtime speed.
So doing more work at link time (even recompiling modules) may not
necessarily be a bad idea (as long as that work is parallelizable, with
itself or the C LTO).

> 
> For the next problem, assume the interface U.mli and its
> implementation U.ml. We want to extend them -- say, add a new
> operation to the interface and the implementation -- without
> modifying
> U.mli/U.ml and without cut-and-paste. That is, we want merely
> `include'
> U and add the new operation -- making a `diff' so to speak. If U.mli
> abstracts away the implementation details, and if they are needed to
> implement the new operation, the problem seem unsolvable.
> 
> It turns out there are work-arounds for both problems, which solve
> them, albeit not elegantly. The full explanation is a bit too long
> for
> this message; please see
>         https://okmij.org/ftp/ML/module-extensibility.html

Would be interesting to also explore how this affects 'ocamlopt', where
the various choices also result in working inlining or not (and
potentially runtime performance impact).
The use of -opaque might be needed in some cases to avoid tying users
of a module to a particular implementation too early (through cross-
module inlining, which then effectively makes it incompatible with
other implementations, should it be switched out later).

*If* we want both separate compilation, and good runtime performance
(cross-module inlining) I think we'd have to give up part of separate
compilation, and move code generation to link time, at least for
functions that (transitively) cross such a multi-implementation
interface boundary.
(we can't really recompile all users for each implementation as that
might lead to a combinatorial explosion)



For some situations there is another alternative: build contexts.


If you want to consistently choose implementation X for the interface
A, then all libraries/applications built in that build context could be
configured to do the necessary symlinking/renaming to treat X as the
implementation of A with cross-module inlining intact.
(This is effectively what Mirage has to do for cross-compilation
anyway, so then why does it still need functors...)
Configurations that don't want to use a complicated build setup (or for
quick edit-compile cycles) could fall back to using -opaque and
defering the choice to link time, and losing inlining.

This only works if X itself doesn't depend on more such multi-
implementation interfaces (or if it does the overall number of
configurations is small, i.e. not the full cartesian product). 
At the time of building/installing library Y you could then precompile
it for all the N build contexts (perhaps dependency caching/hashing
should be able to tell you where you can reuse build artifacts from
other contexts).

I think the use of build contexts would actually be compatible with
your proposal. All that would change is build order and whether -opaque
is used or not:
* without build contexts you'd build the opaque interface first, then
its users, then its implementation, then link final application
* with build contexts you'd build the interface (without -opaque!), one
of its implementations, then all users of that implementation, then
link the final application (and changing the implementation would
require recompiling all its users transitively, but that'd be an
acceptable situation for better performance)

> 
> The work-arounds are simple. Should OCaml developers be so inclined,
> they
> may be incorporated in OCaml.
> 

I'd be interested to hear the opinion of Mirage unikernel users, as
they're most likely affected by the current situation, and benefit from
this improvement (I'm not a heavy Mirage unikernel user myself -- yet).

Best regards,
--Edwin

[Caml-list] separate compilation

[Walter Cazzola]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1142 bytes --]

Dear Experts,
I'm a sort of newbe so be patient with if I ask something too silly
(I've googled around but I didn't find anything really usefull).

At the moment I'm investigating on how the OCaML module system and the
separate compilation works.

Nothing original I took the priority queue example from the user manual
and split it into three files (attached): one with the structure (A.ml)
one with the signature (A.mli) and one with a silly main (main.ml).

Following the instructions in the manual I have tried to compile each
file without linking but the only that compiles with success is the file
with the signature since it doesn't refer to any other file.

When I compile the structure or the main I get this error:
   >ocamlc -c A.ml
   File "A.ml", line 17, characters 39-52:
   Error: Unbound module type CharPQueueAbs

I have tried to put #use or #load in the A.ml and main.ml file but the
only result is to get a syntax error (e.g., with #use "A.mli" at the
beggining of A.ml)

   >ocamlc -c A.ml
   File "A.ml", line 1, characters 0-1:
   Error: Syntax error

What am I doing wrong? Any help is appreciate, thanks.

Walter

-- 

[-- Attachment #2: Type: TEXT/PLAIN, Size: 271 bytes --]

module type CharPQueueAbs =
   sig
     type priority = int         (* still concrete *)
     type char_queue            (* now abstract *)
     val empty : char_queue
     val insert : char_queue -> int -> char -> char_queue
     exception QueueIsEmpty
   end;;

[-- Attachment #3: Type: TEXT/PLAIN, Size: 572 bytes --]

#use "A.mli"

module PrioQueue =
   struct
     type priority = int
     type char_queue = Empty | Node of priority * char * char_queue * char_queue
     exception QueueIsEmpty
     
     let empty = Empty
     let rec insert queue prio elt =
       match queue with
         Empty -> Node(prio, elt, Empty, Empty)
       | Node(p, e, left, right) ->
           if prio <= p
           then Node(prio, elt, insert right p e, left)
           else Node(p, e, insert right prio elt, left)
   end;;

module AbstractPrioQueue = (PrioQueue: CharPQueueAbs);;

[-- Attachment #4: Type: TEXT/PLAIN, Size: 59 bytes --]

open AbstractPrioQueue;;

let x = insert empty 1 'a' ;;

Re: [Caml-list] separate compilation

[Philippe Wang]

#stuff are generally for interactive top-level use.
#use and #load definitely are. They can't be compiled.

First, compile interface files:
ocamlc -c a.mli
then compile your ml :
ocamlc -c a.ml
because your ml uses (at least it seems to) information contained in
your mli file.

If your files depend on other files that are in other directories, you
may need to use -I, like in
ocamlc -c -I lib/stuff plop.ml
if you have for instance a dependance on a module contained in lib/stuff.
(Notice that modules on which you depend have to be compiled)

And to use mli information, it has to be compiled (into a cmi file).

Cheers,

-- 
Philippe Wang
   mail@philippewang.info



On Thu, Sep 8, 2011 at 4:26 PM, Walter Cazzola <cazzola@dico.unimi.it> wrote:
> Dear Experts,
> I'm a sort of newbe so be patient with if I ask something too silly
> (I've googled around but I didn't find anything really usefull).
>
> At the moment I'm investigating on how the OCaML module system and the
> separate compilation works.
>
> Nothing original I took the priority queue example from the user manual
> and split it into three files (attached): one with the structure (A.ml)
> one with the signature (A.mli) and one with a silly main (main.ml).
>
> Following the instructions in the manual I have tried to compile each
> file without linking but the only that compiles with success is the file
> with the signature since it doesn't refer to any other file.
>
> When I compile the structure or the main I get this error:
>  >ocamlc -c A.ml
>  File "A.ml", line 17, characters 39-52:
>  Error: Unbound module type CharPQueueAbs
>
> I have tried to put #use or #load in the A.ml and main.ml file but the
> only result is to get a syntax error (e.g., with #use "A.mli" at the
> beggining of A.ml)
>
>  >ocamlc -c A.ml
>  File "A.ml", line 1, characters 0-1:
>  Error: Syntax error
>
> What am I doing wrong? Any help is appreciate, thanks.
>
> Walter
>
> --
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

Re: [Caml-list] separate compilation

[Walter Cazzola]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1356 bytes --]

On Thu, 8 Sep 2011, Philippe Wang wrote:

> #stuff are generally for interactive top-level use.
> #use and #load definitely are. They can't be compiled.

thanks I missed this particular.

> First, compile interface files:
> ocamlc -c a.mli
> then compile your ml :
> ocamlc -c a.ml
> because your ml uses (at least it seems to) information contained in
> your mli file.

this is exactly how I've tried to compile it, in my current dir I've:

   >ll
   total 16
   -rw-r--r--. 1 cazzola collab 382 Sep  8 15:51 A.cmi
   -rw-r--r--. 1 cazzola collab 553 Sep  8 16:25 A.ml
   -rw-r--r--. 1 cazzola collab 263 Sep  8 15:42 A.mli
   -rw-r--r--. 1 cazzola collab  56 Sep  8 15:46 main.ml

so the .cmi for the signature is there but when I do:

   >ocamlc -c A.ml
   File "A.ml", line 17, characters 39-52:
   Error: Unbound module type CharPQueueAbs

I don't know what to do, I've also tried to change the file name to the
structure/signature name but nothing changed.

Any other suggestion?

Walter

-- 
Walter Cazzola, PhD - Associate Professor, DICo, University of Milano
E-mail: cazzola@dico.unimi.it  Ph.: +39 02 503 16300  Fax: +39 02 503 16253
· · · ---------------------------- · · · ---------------------------- · · ·
                ... recursive: adjective, see recursive ...
· · · ---------------------------- · · · ---------------------------- · · ·

Re: [Caml-list] separate compilation

[Esther Baruk]

[-- Attachment #1: Type: text/plain, Size: 1850 bytes --]

You must also put the signature of the module type CharPQueueAbs in the
implementation (A.ml).
Note that the directive #use or #load are toplevel directives. They cannot
be used at compilation.



Esther Baruk


On Thu, Sep 8, 2011 at 4:26 PM, Walter Cazzola <cazzola@dico.unimi.it>wrote:

> Dear Experts,
> I'm a sort of newbe so be patient with if I ask something too silly
> (I've googled around but I didn't find anything really usefull).
>
> At the moment I'm investigating on how the OCaML module system and the
> separate compilation works.
>
> Nothing original I took the priority queue example from the user manual
> and split it into three files (attached): one with the structure (A.ml)
> one with the signature (A.mli) and one with a silly main (main.ml).
>
> Following the instructions in the manual I have tried to compile each
> file without linking but the only that compiles with success is the file
> with the signature since it doesn't refer to any other file.
>
> When I compile the structure or the main I get this error:
>  >ocamlc -c A.ml
>  File "A.ml", line 17, characters 39-52:
>  Error: Unbound module type CharPQueueAbs
>
> I have tried to put #use or #load in the A.ml and main.ml file but the
> only result is to get a syntax error (e.g., with #use "A.mli" at the
> beggining of A.ml)
>
>  >ocamlc -c A.ml
>  File "A.ml", line 1, characters 0-1:
>  Error: Syntax error
>
> What am I doing wrong? Any help is appreciate, thanks.
>
> Walter
>
> --
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/**wws/info/caml-list<https://sympa-roc.inria.fr/wws/info/caml-list>
> Beginner's list: http://groups.yahoo.com/group/**ocaml_beginners<http://groups.yahoo.com/group/ocaml_beginners>
> Bug reports: http://caml.inria.fr/bin/caml-**bugs<http://caml.inria.fr/bin/caml-bugs>
>
>

[-- Attachment #2: Type: text/html, Size: 2574 bytes --]

Re: [Caml-list] separate compilation

[Walter Cazzola]

On Thu, 8 Sep 2011, Esther Baruk wrote:

> You must also put the signature of the module type CharPQueueAbs in the
> implementation (A.ml).

this means that can't I separate signature from the implementation? That is
do I have to keep both struct and sig in the same file? or do you mean
something different.

Thanks for the hints

Walter
-- 

Re: [Caml-list] separate compilation

[AUGER Cedric]

[-- Attachment #1: Type: text/plain, Size: 2119 bytes --]

Le Thu, 8 Sep 2011 16:42:39 +0200 (CEST),
Walter Cazzola <cazzola@dico.unimi.it> a écrit :

> On Thu, 8 Sep 2011, Esther Baruk wrote:
> 
> > You must also put the signature of the module type CharPQueueAbs in
> > the implementation (A.ml).
> 
> this means that can't I separate signature from the implementation?
> That is do I have to keep both struct and sig in the same file? or do
> you mean something different.
> 
> Thanks for the hints
> 
> Walter

You seem to have misunderstood the system of modules, I send you your
files with variants which are compilable.
Never forgot that any object declared in an interface must be
implemented (it wasn't the case in your files, since CharPQueueAbst
was declared in the mli file, but not implemented in your ml file;
note also the difference between "implemented" and "instanciated").
========================================================
Implementation                 Interface
========================================================
module type X =                module type X =
 sig                            sig
  type t                         type t
  val habitant : t               val habitant : t
 end                            end

module Habited =               module Habited : X
 struct
  type t = bool
  let habitant = true
  let some_other = false
 end

module Habited2 =              module Habited2 :
 struct                         sig
  type t = bool                  type t
  let habitant = true            val habitant : t
  let some_other = false         val some_other : t
 end                            end

module Habited3 =              module Habited3 :
 struct                         sig
  type t = bool                  type t = bool
  let habitant = true            val habitant : bool
  let some_other = false         val some_other : t
 end                            end
========================================================
You can generate an interface with the "-i" option.
Note that interface of 'X' is itself;
and the same module may have many possible interfaces

[-- Attachment #2: archive.tgz --]
[-- Type: application/x-compressed-tar, Size: 1281 bytes --]

Re: [Caml-list] separate compilation

[Walter Cazzola]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2738 bytes --]

Hi,
thanks Cedric i got the point, I can separate interface from the
implementation but:
  - if I use the module keyword I have to qualify the use of the module
    through the file name and the interface file should have a base name
    different from the implementation file
  - otherwise the file itself can be considered a module and I can remove
    module, struct and sig keywords from the code and have the same name
    for the interface and implementation file.

Thanks a lot from the guide all these details are missing or very well
hidden.

Walter

On Thu, 8 Sep 2011, AUGER Cedric wrote:

> Le Thu, 8 Sep 2011 16:42:39 +0200 (CEST),
> Walter Cazzola <cazzola@dico.unimi.it> a écrit :
>
>> On Thu, 8 Sep 2011, Esther Baruk wrote:
>>
>>> You must also put the signature of the module type CharPQueueAbs in
>>> the implementation (A.ml).
>>
>> this means that can't I separate signature from the implementation?
>> That is do I have to keep both struct and sig in the same file? or do
>> you mean something different.
>>
>> Thanks for the hints
>>
>> Walter
>
> You seem to have misunderstood the system of modules, I send you your
> files with variants which are compilable.
> Never forgot that any object declared in an interface must be
> implemented (it wasn't the case in your files, since CharPQueueAbst
> was declared in the mli file, but not implemented in your ml file;
> note also the difference between "implemented" and "instanciated").
> ========================================================
> Implementation                 Interface
> ========================================================
> module type X =                module type X =
> sig                            sig
>  type t                         type t
>  val habitant : t               val habitant : t
> end                            end
>
> module Habited =               module Habited : X
> struct
>  type t = bool
>  let habitant = true
>  let some_other = false
> end
>
> module Habited2 =              module Habited2 :
> struct                         sig
>  type t = bool                  type t
>  let habitant = true            val habitant : t
>  let some_other = false         val some_other : t
> end                            end
>
> module Habited3 =              module Habited3 :
> struct                         sig
>  type t = bool                  type t = bool
>  let habitant = true            val habitant : bool
>  let some_other = false         val some_other : t
> end                            end
> ========================================================
> You can generate an interface with the "-i" option.
> Note that interface of 'X' is itself;
> and the same module may have many possible interfaces
>

-- 

RE: [Caml-list] separate compilation

[David Allsopp]

Walter Cazzola wrote:
> Hi,
> thanks Cedric i got the point, I can separate interface from the
> implementation but:

The main point of interfaces is to constrain the inferred interface of a module (e.g. hide functions or types which shouldn't be exposed to another module or constraint polymorphic variables to a specific type). It's not like C - you're not declaring a header file which you then usually include in the implementation itself (and, yes, it does unfortunately result in some code duplication - at least in terms of type definitions)

>   - if I use the module keyword I have to qualify the use of the module
>     through the file name and the interface file should have a base name
>     different from the implementation file

Yes, because the module is a sub-module of the module defined by the file.

>   - otherwise the file itself can be considered a module and I can remove
>     module, struct and sig keywords from the code and have the same name
>     for the interface and implementation file.

It's not that the file "can be considered a module" - it entirely is a module with the case sensitive (except for the first character) filename being the module name (so module Foo should be kept in foo.ml or Foo.ml)

> Thanks a lot from the guide all these details are missing or very well
> hidden.

Chapter 2 of the manual "The module system", section 5 "Modules and separate compilation" - http://caml.inria.fr/pub/docs/manual-ocaml/manual004.html#toc17


David

<snipped>