[Caml-list] Dependencies between plugins

[Caml-list] Dependencies between plugins

[François Bobot]

Hi everyone,

	I'm wondering how people are handling dependencies between libraries dynamically linked.

Even if many libraries compile a cmxs and have in their META file `archive(plugin,native)=...`. I 
know only one tool that uses this information: the ocsigen server. The code to gather all the cmxs 
or cma is not hard to write, thanks to findlib:

https://github.com/ocsigen/ocsigenserver/blob/master/src/baselib/ocsigen_loader.ml#L165

Does someone wrote a library that does that?

Gerd, do you think that something that does that could be added to ocamlfind? One tricky thing is to 
know the library statically linked (ie. `Ocsigen_config.builtin_packages`), perhaps ocamlfind can 
during linking add this information.

Regards,

-- 
François

Re: [Caml-list] Dependencies between plugins

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 1501 bytes --]

Maxence Guesdon's static blog/website generator Stog (
https://zoggy.github.io/stog/index.html ) supports dynamic plugins, and its
loading code is available at


https://github.com/zoggy/stog/blob/e83c363c83465a7bfd1595816b3d9bc8331af560/stog_dyn.ml#L119-L146

It also uses ocamlfind, but through the command-line rather than the
Findlib library.

On Tue, Mar 3, 2015 at 2:15 PM, François Bobot <francois.bobot@cea.fr>
wrote:

> Hi everyone,
>
>         I'm wondering how people are handling dependencies between
> libraries dynamically linked.
>
> Even if many libraries compile a cmxs and have in their META file
> `archive(plugin,native)=...`. I know only one tool that uses this
> information: the ocsigen server. The code to gather all the cmxs or cma is
> not hard to write, thanks to findlib:
>
> https://github.com/ocsigen/ocsigenserver/blob/master/src/
> baselib/ocsigen_loader.ml#L165
>
> Does someone wrote a library that does that?
>
> Gerd, do you think that something that does that could be added to
> ocamlfind? One tricky thing is to know the library statically linked (ie.
> `Ocsigen_config.builtin_packages`), perhaps ocamlfind can during linking
> add this information.
>
> Regards,
>
> --
> François
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 2476 bytes --]

Re: [Caml-list] Dependencies between plugins

[François Bobot]

On 03/03/2015 14:40, Gabriel Scherer wrote:
> Maxence Guesdon's static blog/website generator Stog ( https://zoggy.github.io/stog/index.html )
> supports dynamic plugins, and its loading code is available at
>
> https://github.com/zoggy/stog/blob/e83c363c83465a7bfd1595816b3d9bc8331af560/stog_dyn.ml#L119-L146
>
> It also uses ocamlfind, but through the command-line rather than the Findlib library.

Thank you for the pointer. His code is even able to create cmxs from cmxa if needed!

-- 
François

Re: [Caml-list] Dependencies between plugins

[Maxence Guesdon]

On Tue, 03 Mar 2015 15:23:34 +0100
François Bobot <francois.bobot@cea.fr> wrote:

> On 03/03/2015 14:40, Gabriel Scherer wrote:
> > Maxence Guesdon's static blog/website generator Stog ( https://zoggy.github.io/stog/index.html )
> > supports dynamic plugins, and its loading code is available at
> >
> > https://github.com/zoggy/stog/blob/e83c363c83465a7bfd1595816b3d9bc8331af560/stog_dyn.ml#L119-L146
> >
> > It also uses ocamlfind, but through the command-line rather than the Findlib library.
> 
> Thank you for the pointer. His code is even able to create cmxs from cmxa if needed!

Warning, as you see according to the name: it's a hack :)

This discussion is an opportunity to call every library developer
to compile and install .cmxs files, *please*.

-- 
Maxence

Re: [Caml-list] Dependencies between plugins

[Ivan Gotovchits]

[-- Attachment #1: Type: text/plain, Size: 1019 bytes --]

But beware that Stog code is distributed under the terms of GPL v3.

You may be also interested in JaneStreets ocaml-plugin library.

On Tue, Mar 3, 2015 at 9:23 AM, François Bobot <francois.bobot@cea.fr>
wrote:

> On 03/03/2015 14:40, Gabriel Scherer wrote:
>
>> Maxence Guesdon's static blog/website generator Stog (
>> https://zoggy.github.io/stog/index.html )
>> supports dynamic plugins, and its loading code is available at
>>
>> https://github.com/zoggy/stog/blob/e83c363c83465a7bfd1595816b3d9b
>> c8331af560/stog_dyn.ml#L119-L146
>>
>> It also uses ocamlfind, but through the command-line rather than the
>> Findlib library.
>>
>
> Thank you for the pointer. His code is even able to create cmxs from cmxa
> if needed!
>
>
> --
> François
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 2012 bytes --]

Re: [Caml-list] Dependencies between plugins

[Sebastien Mondet]

[-- Attachment #1: Type: text/plain, Size: 1421 bytes --]

We have also some code doing that there:
https://github.com/hammerlab/ketrew/blob/master/src/lib/pure/ketrew_plugin.ml#L52
One of the tricks used is that, at configure time, the list of findlib
packages already linked is given to the library itself to avoid double
loading.



On Tue, Mar 3, 2015 at 9:32 AM, Ivan Gotovchits <ivg@ieee.org> wrote:

> But beware that Stog code is distributed under the terms of GPL v3.
>
> You may be also interested in JaneStreets ocaml-plugin library.
>
> On Tue, Mar 3, 2015 at 9:23 AM, François Bobot <francois.bobot@cea.fr>
> wrote:
>
>> On 03/03/2015 14:40, Gabriel Scherer wrote:
>>
>>> Maxence Guesdon's static blog/website generator Stog (
>>> https://zoggy.github.io/stog/index.html )
>>> supports dynamic plugins, and its loading code is available at
>>>
>>> https://github.com/zoggy/stog/blob/e83c363c83465a7bfd1595816b3d9b
>>> c8331af560/stog_dyn.ml#L119-L146
>>>
>>> It also uses ocamlfind, but through the command-line rather than the
>>> Findlib library.
>>>
>>
>> Thank you for the pointer. His code is even able to create cmxs from cmxa
>> if needed!
>>
>>
>> --
>> François
>>
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>
>
>

[-- Attachment #2: Type: text/html, Size: 2788 bytes --]

Re: [Caml-list] Dependencies between plugins

[François Bobot]

On 03/03/2015 15:42, Sebastien Mondet wrote:
>
> We have also some code doing that there:
> https://github.com/hammerlab/ketrew/blob/master/src/lib/pure/ketrew_plugin.ml#L52
> One of the tricks used is that, at configure time, the list of findlib packages already linked is
> given to the library itself to avoid double loading.

Thanks for the link! I like the use of findlib for computing the set of package already linked!

```
let ketrew_deep_ancestors () =
   Lazy.force (lazy (
    Findlib.package_deep_ancestors ["native"] Ketrew_metadata.findlib_packages
   ))
```

PS: I think that `fun () -> Lazy.force (lazy e)` is equivalent to `fun () -> e`.
You surely wanted `let l = lazy e in fun () -> Lazy.force l`, no?

-- 
François

Re: [Caml-list] Dependencies between plugins

[Sebastien Mondet]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

On Tue, Mar 3, 2015 at 10:02 AM, François Bobot <francois.bobot@cea.fr>
wrote:

> On 03/03/2015 15:42, Sebastien Mondet wrote:
>
>>
>> We have also some code doing that there:
>> https://github.com/hammerlab/ketrew/blob/master/src/lib/
>> pure/ketrew_plugin.ml#L52
>> One of the tricks used is that, at configure time, the list of findlib
>> packages already linked is
>> given to the library itself to avoid double loading.
>>
>
> Thanks for the link! I like the use of findlib for computing the set of
> package already linked!
>
> ```
> let ketrew_deep_ancestors () =
>   Lazy.force (lazy (
>    Findlib.package_deep_ancestors ["native"] Ketrew_metadata.findlib_
> packages
>   ))
> ```
>
> PS: I think that `fun () -> Lazy.force (lazy e)` is equivalent to `fun ()
> -> e`.
> You surely wanted `let l = lazy e in fun () -> Lazy.force l`, no?
>
>
ah yes, that used to be a try-to-work-around "unbound type variables" that
I didn't clean-up




> --
> François
>

[-- Attachment #2: Type: text/html, Size: 1856 bytes --]

Re: [Caml-list] Dependencies between plugins

[François Bobot]

On 03/03/2015 15:32, Ivan Gotovchits wrote:
> You may be also interested in JaneStreets ocaml-plugin library.

It is an interesting library for adding scripting capacity to a tool (ie: compile .ml and load them) 
but I think it doesn't use ocamlfind/findlib/META and doesn't load dynlink recursively. Do you agree?


Thanks,

-- 
François

Re: [Caml-list] Dependencies between plugins

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 1401 bytes --]

Am Dienstag, den 03.03.2015, 14:15 +0100 schrieb François Bobot:
> Hi everyone,
> 
> 	I'm wondering how people are handling dependencies between libraries dynamically linked.
> 
> Even if many libraries compile a cmxs and have in their META file `archive(plugin,native)=...`. I 
> know only one tool that uses this information: the ocsigen server. The code to gather all the cmxs 
> or cma is not hard to write, thanks to findlib:
> 
> https://github.com/ocsigen/ocsigenserver/blob/master/src/baselib/ocsigen_loader.ml#L165
> 
> Does someone wrote a library that does that?
> 
> Gerd, do you think that something that does that could be added to ocamlfind? One tricky thing is to 
> know the library statically linked (ie. `Ocsigen_config.builtin_packages`), perhaps ocamlfind can 
> during linking add this information.

I think so. For toploops, there is already code that tracks libraries
already linked into the executable (i.e. if you ocamlmktop your
toploop).

Gerd


> 
> Regards,
> 
> -- 
> François
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] Dependencies between plugins

[François Bobot]

[-- Attachment #1: Type: text/plain, Size: 3269 bytes --]

On 03/03/2015 15:55, Gerd Stolpmann wrote:
> Am Dienstag, den 03.03.2015, 14:15 +0100 schrieb François Bobot:
>> Gerd, do you think that something that does that could be added to ocamlfind? One tricky thing is to
>> know the library statically linked (ie. `Ocsigen_config.builtin_packages`), perhaps ocamlfind can
>> during linking add this information.
>
> I think so. For toploops, there is already code that tracks libraries
> already linked into the executable (i.e. if you ocamlmktop your
> toploop).

All was already in place indeed! It was easy to add. I kept caml-list in CC for discussing the big 
picture in order to get comments from people. Is mail still your preferred way of receiving patch?

I kept it simple, no hack (no automatic: cmxa -> cmxs) because I prefer problems in library META to 
be found than to be paper over.

The first patch adds:
- A library `findlib.dynlink` that depends on `findlib` and `dynlink`
- During linking (using `ocamlfind ocaml*`) if `-package findlib.dynlink` and `-linkpkg` are used 
then a module `Findlib_initl...` is linked after all packages and it stores the names of packages 
linked in `Findlib.linked_packages`.
- In the main program `Fl_dynlink.load_packages ["yojson"]` can be used to dynlink packages

The second patch forbids to run `Fl_dynlink.load_packages` during the initialization of packages 
(statically or dynamically linked), because `Findlib_initl...` is not yet run and because if you 
want to load a package that depend on a statically linked package not yet initialized, there is no 
sane semantic.

Problems:
- The package is named `findlib.dynlink`, the archive `findlib_dynlink.cm*` and the module 
`Fl_dynlink` ...
- If you don't use `-linkall` static packages could only be partially linked, and you can't link the 
remaining part later. So perhaps `-linkall` must be automatically added if `findlib.dynlink` is used.
- If you define `archive(native)` and not `archive(native,plugin)` the error is not nice (in 
Dynlink.Error). Perhaps I should add a `Package_not_dynamically_loadable of string` error, that 
should catch the loading of something else than `*.cmxs`.
- Often you link your binary with your own library without using `-package` (the library is not yet 
installed), and plugins for your tools depend on your library. Currently you should do before any 
`Fl_dynlink.load_packages`: `Findlib.linked_packages := "mylib"::Findlib.linked_packages`. For 
simplicity, I don't know if I should add a function `Fl_dynlink.add_linked_packages`, or an option 
to ocamlfind `-additional-package-statically-linked `.
- During the initialization of your own library (linked without -package) you should not use 
`Fl_dynlink.load_packages`, but the library doesn't protect you against this error.

Choices:
- If you don't link with `findlib.dynlink` or use `create_toploop`, the variables 
`Findlib.linked_packages` and `Findlib.linked_predicates` are empty because I don't wanted to add 
backward change by adding `Findlib_initl...` when `findlib` is linked.


Remains to do:
- Fix problems
- Documentations (add `plugin` in the list of standard predicates, ...)


Gerg, what do you think of this first version of the patch? Of the way to fix the problems?

Thanks,

Regards,

-- 
François

[-- Attachment #2: 0001-Dynlink-add-utilities-for-dynamic-linking-packages.patch --]
[-- Type: text/x-diff, Size: 7795 bytes --]

From ce9c7aab9b883a7130ce1b350583c79365e82423 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fran=C3=A7ois=20Bobot?= <francois.bobot@cea.fr>
Date: Tue, 3 Mar 2015 23:50:00 +0100
Subject: [PATCH 1/2] [Dynlink] add utilities for dynamic linking packages

---
 src/findlib/META.in        | 10 ++++++++++
 src/findlib/Makefile       | 19 ++++++++++++++++---
 src/findlib/findlib.ml     |  4 ++++
 src/findlib/findlib.mli    |  7 +++++++
 src/findlib/fl_dynlink.ml  | 25 +++++++++++++++++++++++++
 src/findlib/fl_dynlink.mli |  9 +++++++++
 src/findlib/frontend.ml    | 13 ++++++++-----
 7 files changed, 79 insertions(+), 8 deletions(-)
 create mode 100644 src/findlib/fl_dynlink.ml
 create mode 100644 src/findlib/fl_dynlink.mli

diff --git a/src/findlib/META.in b/src/findlib/META.in
index e19bcad..859bcc7 100644
--- a/src/findlib/META.in
+++ b/src/findlib/META.in
@@ -6,3 +6,13 @@ archive(byte) = "findlib.cma"
 archive(byte,toploop) = "findlib.cma findlib_top.cma"
 archive(byte,create_toploop) = "findlib.cma findlib_top.cma"
 archive(native) = "findlib.cmxa"
+
+package "dynlink" (
+  description = "Package manager dynamic linker"
+  requires = "findlib dynlink"
+  archive(byte) = "findlib_dynlink.cma"
+  archive(native) = "findlib_dynlink.cmxa"
+#Even if it strange and discouraged to dynlink this package
+  archive(byte,plugin) = "findlib_dynlink.cma"
+  archive(native,plugin) = "findlib_dynlink.cmxs"
+)
\ No newline at end of file
diff --git a/src/findlib/Makefile b/src/findlib/Makefile
index bdd7f14..cbfec7b 100644
--- a/src/findlib/Makefile
+++ b/src/findlib/Makefile
@@ -28,14 +28,18 @@ TOBJECTS       = topfind.cmo
 
 XOBJECTS       = $(OBJECTS:.cmo=.cmx)
 
+OBJECTS_DYNLINK  = fl_dynlink.cmo
+XOBJECTS_DYNLINK = $(OBJECTS_DYNLINK:.cmo=.cmx)
+
 OCAMLFIND_OBJECTS = ocaml_args.cmo frontend.cmo
 OCAMLFIND_XOBJECTS = ocaml_args.cmx frontend.cmx
 
 NUMTOP_OBJECTS = num_top_printers.cmo num_top.cmo
 
-all: ocamlfind$(EXEC_SUFFIX) findlib.cma findlib_top.cma topfind num_top.cma
+all: ocamlfind$(EXEC_SUFFIX) findlib.cma findlib_top.cma topfind num_top.cma \
+     findlib_dynlink.cma
 
-opt: ocamlfind_opt$(EXEC_SUFFIX) findlib.cmxa topfind
+opt: ocamlfind_opt$(EXEC_SUFFIX) findlib.cmxa findlib_dynlink.cmxa topfind
 
 ocamlfind$(EXEC_SUFFIX): findlib.cma $(OCAMLFIND_OBJECTS)
 	$(OCAMLC) $(CUSTOM) -o ocamlfind$(EXEC_SUFFIX) -g findlib.cma unix.cma \
@@ -60,6 +64,15 @@ findlib.cmxa: $(XOBJECTS)
 	    $(OCAMLOPT) -shared -o findlib.cmxs $(XOBJECTS); \
 	fi
 
+findlib_dynlink.cma: $(OBJECTS_DYNLINK)
+	$(OCAMLC) -a -o $@ $(OBJECTS_DYNLINK)
+
+findlib_dynlink.cmxa: $(XOBJECTS_DYNLINK)
+	$(OCAMLOPT) -a -o $@ $(XOBJECTS_DYNLINK)
+	if [ $(HAVE_NATDYNLINK) -gt 0 ]; then \
+	    $(OCAMLOPT) -shared -o findlib_dynlink.cmxs $(XOBJECTS_DYNLINK); \
+	fi
+
 findlib_config.ml: findlib_config.mlp $(TOP)/Makefile.config
 	USE_CYGPATH="$(USE_CYGPATH)"; \
 	export USE_CYGPATH; \
@@ -100,7 +113,7 @@ install: all
 	mkdir -p "$(prefix)$(OCAML_SITELIB)/$(NAME)"
 	mkdir -p "$(prefix)$(OCAMLFIND_BIN)"
 	test $(INSTALL_TOPFIND) -eq 0 || cp topfind "$(prefix)$(OCAML_CORE_STDLIB)"
-	files=`$(TOP)/tools/collect_files $(TOP)/Makefile.config findlib.cmi findlib.mli findlib.cma topfind.cmi topfind.mli fl_package_base.mli fl_package_base.cmi fl_metascanner.mli fl_metascanner.cmi fl_metatoken.cmi findlib_top.cma findlib.cmxa findlib.a findlib.cmxs META` && \
+	files=`$(TOP)/tools/collect_files $(TOP)/Makefile.config findlib.cmi findlib.mli findlib.cma topfind.cmi topfind.mli fl_package_base.mli fl_package_base.cmi fl_metascanner.mli fl_metascanner.cmi fl_metatoken.cmi findlib_top.cma findlib.cmxa findlib.a findlib.cmxs findlib_dynlink.cma findlib_dynlink.cmxa findlib_dynlink.a findlib_dynlink.cmxs fl_dynlink.mli fl_dynlink.cmi META` && \
 	cp $$files "$(prefix)$(OCAML_SITELIB)/$(NAME)"
 	f="ocamlfind$(EXEC_SUFFIX)"; { test -f ocamlfind_opt$(EXEC_SUFFIX) && f="ocamlfind_opt$(EXEC_SUFFIX)"; }; \
 	cp $$f "$(prefix)$(OCAMLFIND_BIN)/ocamlfind$(EXEC_SUFFIX)"
diff --git a/src/findlib/findlib.ml b/src/findlib/findlib.ml
index 2b1aa2c..75e2ae1 100644
--- a/src/findlib/findlib.ml
+++ b/src/findlib/findlib.ml
@@ -427,3 +427,7 @@ let list_packages ?(tab = 20) ?(descr = false) ch =
     )
     packages_sorted
 ;;
+
+
+let linked_packages = ref []
+let linked_predicates = ref []
diff --git a/src/findlib/findlib.mli b/src/findlib/findlib.mli
index d435a84..6ef5cc9 100644
--- a/src/findlib/findlib.mli
+++ b/src/findlib/findlib.mli
@@ -193,3 +193,10 @@ val list_packages : ?tab:int -> ?descr:bool -> out_channel -> unit
    * @param tab The tabulator width, by default 20
    * @param descr Whether package descriptions are printed. Default: false
    *)
+
+val linked_packages : string list ref
+ (** The list of currently linked packages.
+     The packages can be absent of the database *)
+
+val linked_predicates : string list ref
+(** The list of predicates used during linking (eg. native, byte, mt, ...) *)
diff --git a/src/findlib/fl_dynlink.ml b/src/findlib/fl_dynlink.ml
new file mode 100644
index 0000000..b64b617
--- /dev/null
+++ b/src/findlib/fl_dynlink.ml
@@ -0,0 +1,25 @@
+
+(** Utilities for loading dynamically packages *)
+
+let load_pkg pkg =
+  if not (List.mem pkg !Findlib.linked_packages) then
+    (* Determine the package directory: *)
+    let d = Findlib.package_directory pkg in
+    (* Determine the 'archive(plugin,...)' property: *)
+    let archive =
+      try
+        Findlib.package_property ("plugin"::!Findlib.linked_predicates) pkg "archive"
+      with Not_found -> "" in
+    (* Split the 'archive' property and resolve the files: *)
+    let files = Fl_split.in_words archive in
+    List.iter (fun file ->
+        let file = Findlib.resolve_path ~base:d file in
+        Dynlink.loadfile file
+      ) files;
+    Findlib.linked_packages := pkg::!Findlib.linked_packages
+
+let load_packages pkgs =
+  let eff_pkglist =
+    Findlib.package_deep_ancestors !Findlib.linked_predicates pkgs in
+  List.iter load_pkg eff_pkglist
+
diff --git a/src/findlib/fl_dynlink.mli b/src/findlib/fl_dynlink.mli
new file mode 100644
index 0000000..1109442
--- /dev/null
+++ b/src/findlib/fl_dynlink.mli
@@ -0,0 +1,9 @@
+(* $Id$
+ * ----------------------------------------------------------------------
+ *
+ *)
+
+(** Utilities for loading dynamically packages *)
+
+val load_packages : string list -> unit
+(** Dynlink the given packages and all their dependencies *)
diff --git a/src/findlib/frontend.ml b/src/findlib/frontend.ml
index 2e00ccd..82b95b6 100644
--- a/src/findlib/frontend.ml
+++ b/src/findlib/frontend.ml
@@ -1148,8 +1148,11 @@ let ocamlc which () =
   let threads_dir = Filename.concat stdlibdir "threads" in
   let vmthreads_dir = Filename.concat stdlibdir "vmthreads" in
 
-  let initl_file_needed =
+  let create_toploop =
     List.mem "create_toploop" !predicates && List.mem "findlib" eff_link in
+  let initl_file_needed =
+    create_toploop || List.mem "findlib.dynlink" eff_link
+  in
 
   let initl_file_name =
     if initl_file_needed then
@@ -1170,19 +1173,19 @@ let ocamlc which () =
 		  initl_file_name in
     try
       output_string initl
-	("Topfind.don't_load [" ^
+	("Findlib.linked_packages := [" ^
 	 String.concat ";"
 	   (List.map
 	      (fun pkg -> "\"" ^ String.escaped pkg ^ "\"")
 	      eff_link) ^
 	 "];;\n");
+      let predicates = List.filter (fun p -> p <> "create_toploop") !predicates in
       output_string initl
-	("Topfind.predicates := [" ^
+	("Findlib.linked_predicates := [" ^
 	 String.concat ";"
 	   (List.map
 	      (fun pred -> "\"" ^ String.escaped pred ^ "\"")
-	      ("toploop" :: 
-		 (List.filter (fun p -> p <> "create_toploop") !predicates))) ^
+	      (if create_toploop then "toploop" :: predicates else predicates)) ^
 	 "];;\n");
       close_out initl;
     with
-- 
2.1.4


[-- Attachment #3: 0002-Dynlink-forbid-package-loading-when-we-are-currently.patch --]
[-- Type: text/x-diff, Size: 2868 bytes --]

From 5015566e762420fb4bfcdce1ba22f1bbbaff4212 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fran=C3=A7ois=20Bobot?= <francois.bobot@cea.fr>
Date: Wed, 4 Mar 2015 09:49:11 +0100
Subject: [PATCH 2/2] [Dynlink] forbid package loading when we are currently   
        initializing a package

---
 src/findlib/fl_dynlink.ml  | 28 ++++++++++++++++++++++++----
 src/findlib/fl_dynlink.mli | 13 ++++++++++++-
 2 files changed, 36 insertions(+), 5 deletions(-)

diff --git a/src/findlib/fl_dynlink.ml b/src/findlib/fl_dynlink.ml
index b64b617..78c688c 100644
--- a/src/findlib/fl_dynlink.ml
+++ b/src/findlib/fl_dynlink.ml
@@ -1,6 +1,15 @@
 
 (** Utilities for loading dynamically packages *)
 
+exception Package_loading
+
+let package_loading = ref false
+let is_package_loading () =
+  (** At least "findlib.dynlink" should be in the list if Findlib_initl has
+      been run. Otherwise we are running statically linked package  *)
+  !Findlib.linked_packages = []
+  || !package_loading
+
 let load_pkg pkg =
   if not (List.mem pkg !Findlib.linked_packages) then
     (* Determine the package directory: *)
@@ -8,7 +17,8 @@ let load_pkg pkg =
     (* Determine the 'archive(plugin,...)' property: *)
     let archive =
       try
-        Findlib.package_property ("plugin"::!Findlib.linked_predicates) pkg "archive"
+        Findlib.package_property
+          ("plugin"::!Findlib.linked_predicates) pkg "archive"
       with Not_found -> "" in
     (* Split the 'archive' property and resolve the files: *)
     let files = Fl_split.in_words archive in
@@ -19,7 +29,17 @@ let load_pkg pkg =
     Findlib.linked_packages := pkg::!Findlib.linked_packages
 
 let load_packages pkgs =
-  let eff_pkglist =
-    Findlib.package_deep_ancestors !Findlib.linked_predicates pkgs in
-  List.iter load_pkg eff_pkglist
+  if is_package_loading () then
+    raise Package_loading
+  else begin
+    package_loading := true;
+    try
+      let eff_pkglist =
+        Findlib.package_deep_ancestors !Findlib.linked_predicates pkgs in
+      List.iter load_pkg eff_pkglist ;
+      package_loading := false;
+    with exn ->
+      package_loading := false;
+      raise exn
+  end
 
diff --git a/src/findlib/fl_dynlink.mli b/src/findlib/fl_dynlink.mli
index 1109442..a615486 100644
--- a/src/findlib/fl_dynlink.mli
+++ b/src/findlib/fl_dynlink.mli
@@ -5,5 +5,16 @@
 
 (** Utilities for loading dynamically packages *)
 
+exception Package_loading
+(** Indicate that during the loading of a package the loading of another
+    package have been requested *)
+
 val load_packages : string list -> unit
-(** Dynlink the given packages and all their dependencies *)
+(** Dynlink the given packages and all their dependencies;
+    Call can't be nested.
+    @raise Package_loading
+*)
+
+
+val is_package_loading: unit -> bool
+(** Indicate if we are currently loading a package *)
-- 
2.1.4

Re: [Caml-list] Dependencies between plugins

[Ivan Gotovchits]

[-- Attachment #1: Type: text/plain, Size: 3877 bytes --]

Are there any movement in this direction, or this patches will die?

On Wed, Mar 4, 2015 at 4:58 AM, François Bobot <francois.bobot@cea.fr>
wrote:

> On 03/03/2015 15:55, Gerd Stolpmann wrote:
>
>> Am Dienstag, den 03.03.2015, 14:15 +0100 schrieb François Bobot:
>>
>>> Gerd, do you think that something that does that could be added to
>>> ocamlfind? One tricky thing is to
>>> know the library statically linked (ie. `Ocsigen_config.builtin_packages`),
>>> perhaps ocamlfind can
>>> during linking add this information.
>>>
>>
>> I think so. For toploops, there is already code that tracks libraries
>> already linked into the executable (i.e. if you ocamlmktop your
>> toploop).
>>
>
> All was already in place indeed! It was easy to add. I kept caml-list in
> CC for discussing the big picture in order to get comments from people. Is
> mail still your preferred way of receiving patch?
>
> I kept it simple, no hack (no automatic: cmxa -> cmxs) because I prefer
> problems in library META to be found than to be paper over.
>
> The first patch adds:
> - A library `findlib.dynlink` that depends on `findlib` and `dynlink`
> - During linking (using `ocamlfind ocaml*`) if `-package findlib.dynlink`
> and `-linkpkg` are used then a module `Findlib_initl...` is linked after
> all packages and it stores the names of packages linked in
> `Findlib.linked_packages`.
> - In the main program `Fl_dynlink.load_packages ["yojson"]` can be used to
> dynlink packages
>
> The second patch forbids to run `Fl_dynlink.load_packages` during the
> initialization of packages (statically or dynamically linked), because
> `Findlib_initl...` is not yet run and because if you want to load a package
> that depend on a statically linked package not yet initialized, there is no
> sane semantic.
>
> Problems:
> - The package is named `findlib.dynlink`, the archive `findlib_dynlink.cm*`
> and the module `Fl_dynlink` ...
> - If you don't use `-linkall` static packages could only be partially
> linked, and you can't link the remaining part later. So perhaps `-linkall`
> must be automatically added if `findlib.dynlink` is used.
> - If you define `archive(native)` and not `archive(native,plugin)` the
> error is not nice (in Dynlink.Error). Perhaps I should add a
> `Package_not_dynamically_loadable of string` error, that should catch the
> loading of something else than `*.cmxs`.
> - Often you link your binary with your own library without using
> `-package` (the library is not yet installed), and plugins for your tools
> depend on your library. Currently you should do before any
> `Fl_dynlink.load_packages`: `Findlib.linked_packages :=
> "mylib"::Findlib.linked_packages`. For simplicity, I don't know if I
> should add a function `Fl_dynlink.add_linked_packages`, or an option to
> ocamlfind `-additional-package-statically-linked `.
> - During the initialization of your own library (linked without -package)
> you should not use `Fl_dynlink.load_packages`, but the library doesn't
> protect you against this error.
>
> Choices:
> - If you don't link with `findlib.dynlink` or use `create_toploop`, the
> variables `Findlib.linked_packages` and `Findlib.linked_predicates` are
> empty because I don't wanted to add backward change by adding
> `Findlib_initl...` when `findlib` is linked.
>
>
> Remains to do:
> - Fix problems
> - Documentations (add `plugin` in the list of standard predicates, ...)
>
>
> Gerg, what do you think of this first version of the patch? Of the way to
> fix the problems?
>
> Thanks,
>
>
> Regards,
>
> --
> François
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

[-- Attachment #2: Type: text/html, Size: 4881 bytes --]

Re: [Caml-list] Dependencies between plugins

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 5086 bytes --]

Am Montag, den 13.04.2015, 15:27 -0400 schrieb Ivan Gotovchits:
> Are there any movement in this direction, or this patches will die?

Don't think so. Slowness on my side.

Gerd

> 
> On Wed, Mar 4, 2015 at 4:58 AM, François Bobot <francois.bobot@cea.fr>
> wrote:
>         On 03/03/2015 15:55, Gerd Stolpmann wrote:
>                 Am Dienstag, den 03.03.2015, 14:15 +0100 schrieb
>                 François Bobot:
>                         Gerd, do you think that something that does
>                         that could be added to ocamlfind? One tricky
>                         thing is to
>                         know the library statically linked (ie.
>                         `Ocsigen_config.builtin_packages`), perhaps
>                         ocamlfind can
>                         during linking add this information.
>                 
>                 I think so. For toploops, there is already code that
>                 tracks libraries
>                 already linked into the executable (i.e. if you
>                 ocamlmktop your
>                 toploop).
>         
>         All was already in place indeed! It was easy to add. I kept
>         caml-list in CC for discussing the big picture in order to get
>         comments from people. Is mail still your preferred way of
>         receiving patch?
>         
>         I kept it simple, no hack (no automatic: cmxa -> cmxs) because
>         I prefer problems in library META to be found than to be paper
>         over.
>         
>         The first patch adds:
>         - A library `findlib.dynlink` that depends on `findlib` and
>         `dynlink`
>         - During linking (using `ocamlfind ocaml*`) if `-package
>         findlib.dynlink` and `-linkpkg` are used then a module
>         `Findlib_initl...` is linked after all packages and it stores
>         the names of packages linked in `Findlib.linked_packages`.
>         - In the main program `Fl_dynlink.load_packages ["yojson"]`
>         can be used to dynlink packages
>         
>         The second patch forbids to run `Fl_dynlink.load_packages`
>         during the initialization of packages (statically or
>         dynamically linked), because `Findlib_initl...` is not yet run
>         and because if you want to load a package that depend on a
>         statically linked package not yet initialized, there is no
>         sane semantic.
>         
>         Problems:
>         - The package is named `findlib.dynlink`, the archive
>         `findlib_dynlink.cm*` and the module `Fl_dynlink` ...
>         - If you don't use `-linkall` static packages could only be
>         partially linked, and you can't link the remaining part later.
>         So perhaps `-linkall` must be automatically added if
>         `findlib.dynlink` is used.
>         - If you define `archive(native)` and not
>         `archive(native,plugin)` the error is not nice (in
>         Dynlink.Error). Perhaps I should add a
>         `Package_not_dynamically_loadable of string` error, that
>         should catch the loading of something else than `*.cmxs`.
>         - Often you link your binary with your own library without
>         using `-package` (the library is not yet installed), and
>         plugins for your tools depend on your library. Currently you
>         should do before any `Fl_dynlink.load_packages`:
>         `Findlib.linked_packages := "mylib"::Findlib.linked_packages`.
>         For simplicity, I don't know if I should add a function
>         `Fl_dynlink.add_linked_packages`, or an option to ocamlfind
>         `-additional-package-statically-linked `.
>         - During the initialization of your own library (linked
>         without -package) you should not use
>         `Fl_dynlink.load_packages`, but the library doesn't protect
>         you against this error.
>         
>         Choices:
>         - If you don't link with `findlib.dynlink` or use
>         `create_toploop`, the variables `Findlib.linked_packages` and
>         `Findlib.linked_predicates` are empty because I don't wanted
>         to add backward change by adding `Findlib_initl...` when
>         `findlib` is linked.
>         
>         
>         Remains to do:
>         - Fix problems
>         - Documentations (add `plugin` in the list of standard
>         predicates, ...)
>         
>         
>         Gerg, what do you think of this first version of the patch? Of
>         the way to fix the problems?
>         
>         Thanks,
>         
>         
>         Regards,
>         
>         -- 
>         François
>         
>         
> 
> 
-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] Dependencies between plugins

[François Bobot]

On 13/04/2015 21:29, Gerd Stolpmann wrote:
> Am Montag, den 13.04.2015, 15:27 -0400 schrieb Ivan Gotovchits:
>> Are there any movement in this direction, or this patches will die?
>
> Don't think so. Slowness on my side.

On my side, I haven't yet written the documentation. My main impediment is to choose which 
predicates to use for the cmxs in the META file:
1) to keep archive(plugin,native) because it is the defacto standard
2) to move to something that is semantically right: archive(native_plugin) or archive(shared).

Does anybody have a though about that? Gabriel, does my argument convinced you?

Follow up to "[Caml-list] META file standards for native plugins"

https://sympa.inria.fr/sympa/arc/caml-list/2015-04/msg00043.html

-- 
François

Re: [Caml-list] Dependencies between plugins

[Stéphane Glondu]

Le 14/04/2015 10:59, François Bobot a écrit :
>>> Are there any movement in this direction, or this patches will die?
>>
>> Don't think so. Slowness on my side.
> 
> On my side, I haven't yet written the documentation. My main impediment
> is to choose which predicates to use for the cmxs in the META file:
> 1) to keep archive(plugin,native) because it is the defacto standard
> 2) to move to something that is semantically right:
> archive(native_plugin) or archive(shared).

Sorry, I didn't follow the whole discussion but... this looks like
hardcoding a special treatment of plugins for the native case,
forgetting the bytecode case. Would you introduce byte_plugin (or a
bytecode counterpart to "shared" which looks bad to me) as well? Even
code using Dynlink should be as generic (w.r.t. native/bytecode) as
possible...

-- 
Stéphane

Re: [Caml-list] Dependencies between plugins

[François Bobot]

On 14/04/2015 11:47, Stéphane Glondu wrote:
> Le 14/04/2015 10:59, François Bobot a écrit :
>>>> Are there any movement in this direction, or this patches will die?
>>>
>>> Don't think so. Slowness on my side.
>>
>> On my side, I haven't yet written the documentation. My main impediment
>> is to choose which predicates to use for the cmxs in the META file:
>> 1) to keep archive(plugin,native) because it is the defacto standard
>> 2) to move to something that is semantically right:
>> archive(native_plugin) or archive(shared).
>
> Sorry, I didn't follow the whole discussion but... this looks like
> hardcoding a special treatment of plugins for the native case,
> forgetting the bytecode case. Would you introduce byte_plugin (or a
> bytecode counterpart to "shared" which looks bad to me) as well?

The fact is that native and bytecode are not symmetric:
              | byte  | native
static link  | cmo   | cmx
dynamic link | cmo   | cmxs

So for bytecode we can still use `archive(byte)`. If someone wants its library to be loaded 
differently in static linking and dynamic linking e can use `archive(byte,plugin)`.

You are right that I should give a full proposal (I'm going to go with `shared` instead of 
`native_plugin` because it is short and correspond to the ocamlopt option):

1. In META file:
   1.1 use `archive(byte)` and `archive(native)` for the files to statically link
   1.2 use `archive(byte,plugin)` for the files to dynamically link in bytecode if they are 
different from the static one
   1.3 use `archive(shared)` for the files that are dynamically linked in native code

2. During dynamic loading:
   2.1. in bytecode: look for variable `archive` with predicates `byte`,`plugin` and the other 
predicates used during compilation (`mt`, `mt_posix`, `mt_vm`, `gprof`, ...)
   2.2  in native: look for variable `archive` with predicates `shared`, `plugin` and the other 
predicates used during static linking except `native`


My goal is just that when you ask in native code "Does this library define files for dynamic 
linking" the answer is not "yes, it defines these cmx". There are other solutions (like asking that 
file to statically link are define with `archive(native,-plugin)`) but they seem to be less 
conservative.

 > Even
 > code using Dynlink should be as generic (w.r.t. native/bytecode) as
 > possible...


The examples of tools that use dependencies between plugins gathered at the start of the discussion 
are already not generic (w.r.t. native/bytecode) :

The following code makes a differences between bytecode and native code:
https://github.com/ocsigen/ocsigenserver/blob/master/src/baselib/ocsigen_loader.ml#L165
https://github.com/zoggy/stog/blob/e83c363c83465a7bfd1595816b3d9bc8331af560/stog_dyn.ml#L119-L146

This one works only for native code, it seems:
https://github.com/hammerlab/ketrew/blob/master/src/lib/pure/ketrew_plugin.ml#L52

The proposed modification is to replace (for example in ocsigen):

```ocaml
(if Ocsigen_config.is_native then "native" else "byte")
```

by

```ocaml
(if Ocsigen_config.is_native then "shared" else "byte")
```

-- 
François

Re: [Caml-list] Dependencies between plugins

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 5620 bytes --]

François,

I was thinking again about this issue. Introducing a third category
"shared" in addition to byte and native seems to be a bit odd. Actually,
what we really want to have is a second dimension plugin/executable in
addition to the already existing byte/native dimension, so that we can
have:

 - byte + executable (cmo)
 - native + executable (cmx)
 - byte + plugin (cmo, too)
 - native + plugin (cmxs)

That way we can have a separate cmo for byte+plugin, which may be useful
here and there. Also, byte and native are again symmetric.

A typical META file would now specify

archive(byte,executable) = "..."
archive(native,executable) = "..."

if it doesn't support plugins, and specify

archive(byte,executable) = "..."
archive(native,executable) = "..."
archive(byte,plugin) = "..."
archive(native,plugin) = "..."

if it does so. (NB. "executable" because these are the objects for
creating executables.)

The only remaining question is how to handle existing META files that
don't make this distinction. We don't have a version number in META
files, so we have to watch out for another criterion. I am thinking
about understanding

archive(native) = "..."

as

archive(native,executable) = "..."

if there is no other reference to the executable predicate. This would
be a special fixup after parsing META. After a transition phase (say,
two years from now on) we would consider archive(native) as an error.
The upcoming META lint will report this issue.

This way, the existing META files can still be used for some time,
including those specifying plugins. There is no hurry in changing this
detail. However, you are absolutely right that the current use of
"plugin" breaks the way the predicates are defined, and in the long term
this is worth fixing.

Gerd



Am Dienstag, den 14.04.2015, 14:45 +0200 schrieb François Bobot:
> On 14/04/2015 11:47, Stéphane Glondu wrote:
> > Le 14/04/2015 10:59, François Bobot a écrit :
> >>>> Are there any movement in this direction, or this patches will die?
> >>>
> >>> Don't think so. Slowness on my side.
> >>
> >> On my side, I haven't yet written the documentation. My main impediment
> >> is to choose which predicates to use for the cmxs in the META file:
> >> 1) to keep archive(plugin,native) because it is the defacto standard
> >> 2) to move to something that is semantically right:
> >> archive(native_plugin) or archive(shared).
> >
> > Sorry, I didn't follow the whole discussion but... this looks like
> > hardcoding a special treatment of plugins for the native case,
> > forgetting the bytecode case. Would you introduce byte_plugin (or a
> > bytecode counterpart to "shared" which looks bad to me) as well?
> 
> The fact is that native and bytecode are not symmetric:
>               | byte  | native
> static link  | cmo   | cmx
> dynamic link | cmo   | cmxs
> 
> So for bytecode we can still use `archive(byte)`. If someone wants its library to be loaded 
> differently in static linking and dynamic linking e can use `archive(byte,plugin)`.
> 
> You are right that I should give a full proposal (I'm going to go with `shared` instead of 
> `native_plugin` because it is short and correspond to the ocamlopt option):
> 
> 1. In META file:
>    1.1 use `archive(byte)` and `archive(native)` for the files to statically link
>    1.2 use `archive(byte,plugin)` for the files to dynamically link in bytecode if they are 
> different from the static one
>    1.3 use `archive(shared)` for the files that are dynamically linked in native code
> 
> 2. During dynamic loading:
>    2.1. in bytecode: look for variable `archive` with predicates `byte`,`plugin` and the other 
> predicates used during compilation (`mt`, `mt_posix`, `mt_vm`, `gprof`, ...)
>    2.2  in native: look for variable `archive` with predicates `shared`, `plugin` and the other 
> predicates used during static linking except `native`
> 
> 
> My goal is just that when you ask in native code "Does this library define files for dynamic 
> linking" the answer is not "yes, it defines these cmx". There are other solutions (like asking that 
> file to statically link are define with `archive(native,-plugin)`) but they seem to be less 
> conservative.
> 
>  > Even
>  > code using Dynlink should be as generic (w.r.t. native/bytecode) as
>  > possible...
> 
> 
> The examples of tools that use dependencies between plugins gathered at the start of the discussion 
> are already not generic (w.r.t. native/bytecode) :
> 
> The following code makes a differences between bytecode and native code:
> https://github.com/ocsigen/ocsigenserver/blob/master/src/baselib/ocsigen_loader.ml#L165
> https://github.com/zoggy/stog/blob/e83c363c83465a7bfd1595816b3d9bc8331af560/stog_dyn.ml#L119-L146
> 
> This one works only for native code, it seems:
> https://github.com/hammerlab/ketrew/blob/master/src/lib/pure/ketrew_plugin.ml#L52
> 
> The proposed modification is to replace (for example in ocsigen):
> 
> ```ocaml
> (if Ocsigen_config.is_native then "native" else "byte")
> ```
> 
> by
> 
> ```ocaml
> (if Ocsigen_config.is_native then "shared" else "byte")
> ```
> 
> -- 
> François
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] Dependencies between plugins

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 7232 bytes --]

Another (more general, less ad-hoc) way to have this semantics would be to
introduce a "default value" for predicates that do not explictly appear in
rules. Under the set of default values D, then a rule depending on
predicates P would be true if the boolean assignment (D,P=true) holds, with
the rightmost boolean assignment shadowing all others. I mean that the rule

  foo(x1,x2,x3) = ...

would be applied whenever x1=true, x2=true, x3=true with an empty default
environment, but with the default (y1=true, x1=false, y2=false) it would
only be applied whenever (y1=true, x1=true, y2=false, x2=true, x3=true).

Then setting "plugin=false" in the default environment (equivalently,
-plugin) would give the expected semantics, I believe. In particular
(thanks to François for his clear re-explanation on April 9) then the rule
archive(native) would not be matched by the query archive(native,plugin).

I can see a use for another default variable (ocaml=true) allowing people
that could want to use ocamlfind for non-ocaml stuff to disable it
explicitly, and then reuse the rule names for their own stuff. (In
ocamlbuild it is useful to be able to talk about compilation problems that
are not about OCaml).

On Mon, Apr 27, 2015 at 11:51 AM, Gerd Stolpmann <info@gerd-stolpmann.de>
wrote:

> François,
>
> I was thinking again about this issue. Introducing a third category
> "shared" in addition to byte and native seems to be a bit odd. Actually,
> what we really want to have is a second dimension plugin/executable in
> addition to the already existing byte/native dimension, so that we can
> have:
>
>  - byte + executable (cmo)
>  - native + executable (cmx)
>  - byte + plugin (cmo, too)
>  - native + plugin (cmxs)
>
> That way we can have a separate cmo for byte+plugin, which may be useful
> here and there. Also, byte and native are again symmetric.
>
> A typical META file would now specify
>
> archive(byte,executable) = "..."
> archive(native,executable) = "..."
>
> if it doesn't support plugins, and specify
>
> archive(byte,executable) = "..."
> archive(native,executable) = "..."
> archive(byte,plugin) = "..."
> archive(native,plugin) = "..."
>
> if it does so. (NB. "executable" because these are the objects for
> creating executables.)
>
> The only remaining question is how to handle existing META files that
> don't make this distinction. We don't have a version number in META
> files, so we have to watch out for another criterion. I am thinking
> about understanding
>
> archive(native) = "..."
>
> as
>
> archive(native,executable) = "..."
>
> if there is no other reference to the executable predicate. This would
> be a special fixup after parsing META. After a transition phase (say,
> two years from now on) we would consider archive(native) as an error.
> The upcoming META lint will report this issue.
>
> This way, the existing META files can still be used for some time,
> including those specifying plugins. There is no hurry in changing this
> detail. However, you are absolutely right that the current use of
> "plugin" breaks the way the predicates are defined, and in the long term
> this is worth fixing.
>
> Gerd
>
>
>
> Am Dienstag, den 14.04.2015, 14:45 +0200 schrieb François Bobot:
> > On 14/04/2015 11:47, Stéphane Glondu wrote:
> > > Le 14/04/2015 10:59, François Bobot a écrit :
> > >>>> Are there any movement in this direction, or this patches will die?
> > >>>
> > >>> Don't think so. Slowness on my side.
> > >>
> > >> On my side, I haven't yet written the documentation. My main
> impediment
> > >> is to choose which predicates to use for the cmxs in the META file:
> > >> 1) to keep archive(plugin,native) because it is the defacto standard
> > >> 2) to move to something that is semantically right:
> > >> archive(native_plugin) or archive(shared).
> > >
> > > Sorry, I didn't follow the whole discussion but... this looks like
> > > hardcoding a special treatment of plugins for the native case,
> > > forgetting the bytecode case. Would you introduce byte_plugin (or a
> > > bytecode counterpart to "shared" which looks bad to me) as well?
> >
> > The fact is that native and bytecode are not symmetric:
> >               | byte  | native
> > static link  | cmo   | cmx
> > dynamic link | cmo   | cmxs
> >
> > So for bytecode we can still use `archive(byte)`. If someone wants its
> library to be loaded
> > differently in static linking and dynamic linking e can use
> `archive(byte,plugin)`.
> >
> > You are right that I should give a full proposal (I'm going to go with
> `shared` instead of
> > `native_plugin` because it is short and correspond to the ocamlopt
> option):
> >
> > 1. In META file:
> >    1.1 use `archive(byte)` and `archive(native)` for the files to
> statically link
> >    1.2 use `archive(byte,plugin)` for the files to dynamically link in
> bytecode if they are
> > different from the static one
> >    1.3 use `archive(shared)` for the files that are dynamically linked
> in native code
> >
> > 2. During dynamic loading:
> >    2.1. in bytecode: look for variable `archive` with predicates
> `byte`,`plugin` and the other
> > predicates used during compilation (`mt`, `mt_posix`, `mt_vm`, `gprof`,
> ...)
> >    2.2  in native: look for variable `archive` with predicates `shared`,
> `plugin` and the other
> > predicates used during static linking except `native`
> >
> >
> > My goal is just that when you ask in native code "Does this library
> define files for dynamic
> > linking" the answer is not "yes, it defines these cmx". There are other
> solutions (like asking that
> > file to statically link are define with `archive(native,-plugin)`) but
> they seem to be less
> > conservative.
> >
> >  > Even
> >  > code using Dynlink should be as generic (w.r.t. native/bytecode) as
> >  > possible...
> >
> >
> > The examples of tools that use dependencies between plugins gathered at
> the start of the discussion
> > are already not generic (w.r.t. native/bytecode) :
> >
> > The following code makes a differences between bytecode and native code:
> >
> https://github.com/ocsigen/ocsigenserver/blob/master/src/baselib/ocsigen_loader.ml#L165
> >
> https://github.com/zoggy/stog/blob/e83c363c83465a7bfd1595816b3d9bc8331af560/stog_dyn.ml#L119-L146
> >
> > This one works only for native code, it seems:
> >
> https://github.com/hammerlab/ketrew/blob/master/src/lib/pure/ketrew_plugin.ml#L52
> >
> > The proposed modification is to replace (for example in ocsigen):
> >
> > ```ocaml
> > (if Ocsigen_config.is_native then "native" else "byte")
> > ```
> >
> > by
> >
> > ```ocaml
> > (if Ocsigen_config.is_native then "shared" else "byte")
> > ```
> >
> > --
> > François
> >
> >
>
> --
> ------------------------------------------------------------
> Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
> My OCaml site:          http://www.camlcity.org
> Contact details:        http://www.camlcity.org/contact.html
> Company homepage:       http://www.gerd-stolpmann.de
> ------------------------------------------------------------
>

[-- Attachment #2: Type: text/html, Size: 9158 bytes --]

Re: [Caml-list] Dependencies between plugins

[François Bobot]

On 27/04/2015 12:16, Gabriel Scherer wrote:
> Another (more general, less ad-hoc) way to have this semantics would be to introduce a "default
> value" for predicates that do not explictly appear in rules. Under the set of default values D, then
> a rule depending on predicates P would be true if the boolean assignment (D,P=true) holds, with the
> rightmost boolean assignment shadowing all others. I mean that the rule
>
>    foo(x1,x2,x3) = ...
>
> would be applied whenever x1=true, x2=true, x3=true with an empty default environment, but with the
> default (y1=true, x1=false, y2=false) it would only be applied whenever (y1=true, x1=true, y2=false,
> x2=true, x3=true).
>
> Then setting "plugin=false" in the default environment (equivalently, -plugin) would give the
> expected semantics, I believe. In particular (thanks to François for his clear re-explanation on
> April 9) then the rule archive(native) would not be matched by the query archive(native,plugin).
>

I'm not sure it is usable because "archive" is not the only variable we use. For example "requires" 
is also a variable. So in an environment with `plugin=false`, `requires = "foo"` is not taken into 
account when looking for the dependencies with predicate `native=true,plugin=true`.

Perhaps your idea can be refined by making the default environment linked to specific variables. 
`plugin=false` is used only for `archive`.

-- 
François

Re: [Caml-list] Dependencies between plugins

[Daniel Bünzli]

Le lundi, 27 avril 2015 à 14:16, François Bobot a écrit :
> Perhaps your idea can be refined by making the default environment linked to specific variables.
> `plugin=false` is used only for `archive`.

What about keeping this simple (i.e. the Gerd way) ?  

Daniel

Re: [Caml-list] Dependencies between plugins

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 9881 bytes --]

Well, I was quickly thinking about this obvious alternative, but I think
it is even a more ad-hoc solution. We are changing the rules until they
are matching the META files... Defaults have the disadvantage that they
are somewhat invisible to the user, and make it harder to understand
what's actually going on.

I have another idea. Maybe the wrong starting point of all previous
suggestions is to reuse the "archive" variable for plugins. We could
also have a separate variable for this, e.g. "plugin":

archive(byte) = "file.cma"
archive(native) = "file.cmxa"
plugin(byte) = "file.cma"
plugin(native) = "file.cmxs"

This is even a bit simpler than my previous suggestion. We would need a
little bit of magic for a transition phase, but I think this could be
somewhat more attractive in the long term.

Am Montag, den 27.04.2015, 12:16 +0200 schrieb Gabriel Scherer:
> Another (more general, less ad-hoc) way to have this semantics would
> be to introduce a "default value" for predicates that do not explictly
> appear in rules.
> ...
> I can see a use for another default variable (ocaml=true) allowing
> people that could want to use ocamlfind for non-ocaml stuff to disable
> it explicitly, and then reuse the rule names for their own stuff. (In
> ocamlbuild it is useful to be able to talk about compilation problems
> that are not about OCaml).

Basically you can already do that if you avoid the variables that are
used for ocaml. E.g. you have a script language "dofoo" and want to
manage deps with ocamlfind. You would just not reuse "archive", but a
different variable, let's say dofoo_script. There is already quite good
support for such alternate variables, e.g.

ocamlfind query -format '%+(dofoo_script)' -recursive pkg...

would print all such script names (with absolute path) of pkg, including
the predecessors.

Gerd



> 
> 
> On Mon, Apr 27, 2015 at 11:51 AM, Gerd Stolpmann
> <info@gerd-stolpmann.de> wrote:
>         François,
>         
>         I was thinking again about this issue. Introducing a third
>         category
>         "shared" in addition to byte and native seems to be a bit odd.
>         Actually,
>         what we really want to have is a second dimension
>         plugin/executable in
>         addition to the already existing byte/native dimension, so
>         that we can
>         have:
>         
>          - byte + executable (cmo)
>          - native + executable (cmx)
>          - byte + plugin (cmo, too)
>          - native + plugin (cmxs)
>         
>         That way we can have a separate cmo for byte+plugin, which may
>         be useful
>         here and there. Also, byte and native are again symmetric.
>         
>         A typical META file would now specify
>         
>         archive(byte,executable) = "..."
>         archive(native,executable) = "..."
>         
>         if it doesn't support plugins, and specify
>         
>         archive(byte,executable) = "..."
>         archive(native,executable) = "..."
>         archive(byte,plugin) = "..."
>         archive(native,plugin) = "..."
>         
>         if it does so. (NB. "executable" because these are the objects
>         for
>         creating executables.)
>         
>         The only remaining question is how to handle existing META
>         files that
>         don't make this distinction. We don't have a version number in
>         META
>         files, so we have to watch out for another criterion. I am
>         thinking
>         about understanding
>         
>         archive(native) = "..."
>         
>         as
>         
>         archive(native,executable) = "..."
>         
>         if there is no other reference to the executable predicate.
>         This would
>         be a special fixup after parsing META. After a transition
>         phase (say,
>         two years from now on) we would consider archive(native) as an
>         error.
>         The upcoming META lint will report this issue.
>         
>         This way, the existing META files can still be used for some
>         time,
>         including those specifying plugins. There is no hurry in
>         changing this
>         detail. However, you are absolutely right that the current use
>         of
>         "plugin" breaks the way the predicates are defined, and in the
>         long term
>         this is worth fixing.
>         
>         Gerd
>         
>         
>         
>         Am Dienstag, den 14.04.2015, 14:45 +0200 schrieb François
>         Bobot:
>         > On 14/04/2015 11:47, Stéphane Glondu wrote:
>         > > Le 14/04/2015 10:59, François Bobot a écrit :
>         > >>>> Are there any movement in this direction, or this
>         patches will die?
>         > >>>
>         > >>> Don't think so. Slowness on my side.
>         > >>
>         > >> On my side, I haven't yet written the documentation. My
>         main impediment
>         > >> is to choose which predicates to use for the cmxs in the
>         META file:
>         > >> 1) to keep archive(plugin,native) because it is the
>         defacto standard
>         > >> 2) to move to something that is semantically right:
>         > >> archive(native_plugin) or archive(shared).
>         > >
>         > > Sorry, I didn't follow the whole discussion but... this
>         looks like
>         > > hardcoding a special treatment of plugins for the native
>         case,
>         > > forgetting the bytecode case. Would you introduce
>         byte_plugin (or a
>         > > bytecode counterpart to "shared" which looks bad to me) as
>         well?
>         >
>         > The fact is that native and bytecode are not symmetric:
>         >               | byte  | native
>         > static link  | cmo   | cmx
>         > dynamic link | cmo   | cmxs
>         >
>         > So for bytecode we can still use `archive(byte)`. If someone
>         wants its library to be loaded
>         > differently in static linking and dynamic linking e can use
>         `archive(byte,plugin)`.
>         >
>         > You are right that I should give a full proposal (I'm going
>         to go with `shared` instead of
>         > `native_plugin` because it is short and correspond to the
>         ocamlopt option):
>         >
>         > 1. In META file:
>         >    1.1 use `archive(byte)` and `archive(native)` for the
>         files to statically link
>         >    1.2 use `archive(byte,plugin)` for the files to
>         dynamically link in bytecode if they are
>         > different from the static one
>         >    1.3 use `archive(shared)` for the files that are
>         dynamically linked in native code
>         >
>         > 2. During dynamic loading:
>         >    2.1. in bytecode: look for variable `archive` with
>         predicates `byte`,`plugin` and the other
>         > predicates used during compilation (`mt`, `mt_posix`,
>         `mt_vm`, `gprof`, ...)
>         >    2.2  in native: look for variable `archive` with
>         predicates `shared`, `plugin` and the other
>         > predicates used during static linking except `native`
>         >
>         >
>         > My goal is just that when you ask in native code "Does this
>         library define files for dynamic
>         > linking" the answer is not "yes, it defines these cmx".
>         There are other solutions (like asking that
>         > file to statically link are define with
>         `archive(native,-plugin)`) but they seem to be less
>         > conservative.
>         >
>         >  > Even
>         >  > code using Dynlink should be as generic (w.r.t.
>         native/bytecode) as
>         >  > possible...
>         >
>         >
>         > The examples of tools that use dependencies between plugins
>         gathered at the start of the discussion
>         > are already not generic (w.r.t. native/bytecode) :
>         >
>         > The following code makes a differences between bytecode and
>         native code:
>         >
>         https://github.com/ocsigen/ocsigenserver/blob/master/src/baselib/ocsigen_loader.ml#L165
>         >
>         https://github.com/zoggy/stog/blob/e83c363c83465a7bfd1595816b3d9bc8331af560/stog_dyn.ml#L119-L146
>         >
>         > This one works only for native code, it seems:
>         >
>         https://github.com/hammerlab/ketrew/blob/master/src/lib/pure/ketrew_plugin.ml#L52
>         >
>         > The proposed modification is to replace (for example in
>         ocsigen):
>         >
>         > ```ocaml
>         > (if Ocsigen_config.is_native then "native" else "byte")
>         > ```
>         >
>         > by
>         >
>         > ```ocaml
>         > (if Ocsigen_config.is_native then "shared" else "byte")
>         > ```
>         >
>         > --
>         > François
>         >
>         >
>         
>         --
>         
>         ------------------------------------------------------------
>         Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
>         My OCaml site:          http://www.camlcity.org
>         Contact details:        http://www.camlcity.org/contact.html
>         Company homepage:       http://www.gerd-stolpmann.de
>         ------------------------------------------------------------
>         
> 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] Dependencies between plugins

[François Bobot]

On 27/04/2015 11:51, Gerd Stolpmann wrote:
> François,
>
> I was thinking again about this issue. Introducing a third category
> "shared" in addition to byte and native seems to be a bit odd. Actually,
> what we really want to have is a second dimension plugin/executable in
> addition to the already existing byte/native dimension, so that we can
> have:
>
>   - byte + executable (cmo)
>   - native + executable (cmx)
>   - byte + plugin (cmo, too)
>   - native + plugin (cmxs)
>

This solution represent a more satisfying goal than mine, so even if more META will have to be 
modified at the end it is better.

>
> archive(native) = "..."
>
> as
>
> archive(native,executable) = "..."
>
> if there is no other reference to the executable predicate. This would
> be a special fixup after parsing META. After a transition phase (say,
> two years from now on) we would consider archive(native) as an error.

An error? archive(native) will remain a special case?

> The upcoming META lint will report this issue.

Nice.

>
> This way, the existing META files can still be used for some time,
> including those specifying plugins. There is no hurry in changing this
> detail. However, you are absolutely right that the current use of
> "plugin" breaks the way the predicates are defined, and in the long term
> this is worth fixing.
>

Great! Do you need some help for implementing all of that?

-- 
François

Re: [Caml-list] Dependencies between plugins

[François Bobot]

On 04/03/2015 10:58, François Bobot wrote:
> - If you define `archive(native)` and not `archive(native,plugin)` the error is not nice (in
> Dynlink.Error). Perhaps I should add a `Package_not_dynamically_loadable of string` error, that
> should catch the loading of something else than `*.cmxs`.

Does someone know why the archive for native plugin use `archive(native,plugin)`?

It seems that, according to the semantic of variable in META file, it would be better if it is 
`archive(native_plugin)`. The problem is that `(native,plugin)` means that `archive(native)` is okay 
if there is no `archive(native,plugin)`. Whereas `cmxs` and `cmxa` are as different as `cma` and `cmxa`.

-- 
François

Re: [Caml-list] Dependencies between plugins

[Gabriel Kerneis]

Le 2015-03-03 15:15, François Bobot a écrit :
> Even if many libraries compile a cmxs and have in their META file
> `archive(plugin,native)=...`. I know only one tool that uses this
> information: the ocsigen server. The code to gather all the cmxs or
> cma is not hard to write, thanks to findlib:
>
> 
> https://github.com/ocsigen/ocsigenserver/blob/master/src/baselib/ocsigen_loader.ml#L165

I'm very late to the party, but for the sake of completeness, note that 
I did something similar to add plugins to CIL:
https://github.com/cil-project/cil/blob/develop/src/feature.ml#L96

It was 1.5 years ago, and then I never updated the doc, and the release 
is still pending so I don't remember the details. But I do remember that 
it was a pain to make it work reliably, so a library would be much 
welcome :-)

Best regards,
-- 
Gabriel