[Caml-list] Create a deployable application unit from OPAM?

[Caml-list] Create a deployable application unit from OPAM?

[Hans Ole Rafaelsen]

[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]

Hi,

I have small application written using Ocsigen. Ocaml and the rest of the
development environment is installed using OPAM.

Now I want to install this application on a few other nodes, without having
to bring bring along the whole development process for deployment on each
node. What I want to end up with, is a single unit, e.g a tar-file, a
docker container or similar, that can be copied and deployed to the target
nodes. This unit should only contain the binaries needed to run the
application.

I have tried to ask on the Ocsigen mailing-list how do do this for Ocsigen
applications, but without success. But I guess the problem is more related
to how to deploy applications built using OPAM. I did not find any way to
do this with OPAM, but I might have missed something. I have tried to copy
all the binary files (.so, .cmxa .cmxs, etc) from the OPAM directory and
setting the environment variables to point the this new location, but
without much success. When starting ocsigenserver it was not able to solve
all symbols. Looking more into the problem, it looks like binaries such as
ocsigenserver has hard coded the location of the OPAM development
environment. Might be possible to override this using environment
variables, but I'm not sure how. So I gave up on this track.


Is there a way to generate such a deployable unit from OPAM development
environment?

Do anyone have some tips / best practice on how to deploy applications
built using OPAM?

(I'm developing and want to deploy on Ubuntu 16.04.)

Regards,

Hans Ole Rafaelsen

[-- Attachment #2: Type: text/html, Size: 1758 bytes --]

Re: [Caml-list] Create a deployable application unit from OPAM?

[David Scott]

[-- Attachment #1: Type: text/plain, Size: 2755 bytes --]

Hi,

I had a quick go at creating something with Docker: (and alpine since I was
more familiar with it, but I suspect a similar technique would work with
ubuntu)

https://github.com/djs55/experiments/pull/1

It uses `opam` to build a simple OCaml example (taken from cryptokit) which
needs `libgmp` at runtime. The `Dockerfile` has a slightly hacky attempt to
detect the runtime dependency:

opam depext -n cryptokit | grep -v '#' | grep '\-dev' | while read in; do
basename "$in" -dev; done > needed-packages.txt

It assumes that

- all external C libraries have an alpine depext called foo-dev
- foo-dev has a corresponding runtime package called foo

and installs these `foo` in an output minimal alpine image (i.e. one
without the development tools or other unnecessary stuff)

Another approach would be to use `ldd` or `objdump` recursively to find
libraries and copy only those binaries into the output (possibly using
`FROM scratch`?)

I don't know if this will help -- I realise my test case is a bit minimal
:-)

Cheers,
Dave


On Thu, Mar 29, 2018 at 5:31 PM, Hans Ole Rafaelsen <hrafaelsen@gmail.com>
wrote:

> Hi,
>
> I have small application written using Ocsigen. Ocaml and the rest of the
> development environment is installed using OPAM.
>
> Now I want to install this application on a few other nodes, without
> having to bring bring along the whole development process for deployment on
> each node. What I want to end up with, is a single unit, e.g a tar-file, a
> docker container or similar, that can be copied and deployed to the target
> nodes. This unit should only contain the binaries needed to run the
> application.
>
> I have tried to ask on the Ocsigen mailing-list how do do this for Ocsigen
> applications, but without success. But I guess the problem is more related
> to how to deploy applications built using OPAM. I did not find any way to
> do this with OPAM, but I might have missed something. I have tried to copy
> all the binary files (.so, .cmxa .cmxs, etc) from the OPAM directory and
> setting the environment variables to point the this new location, but
> without much success. When starting ocsigenserver it was not able to solve
> all symbols. Looking more into the problem, it looks like binaries such as
> ocsigenserver has hard coded the location of the OPAM development
> environment. Might be possible to override this using environment
> variables, but I'm not sure how. So I gave up on this track.
>
>
> Is there a way to generate such a deployable unit from OPAM development
> environment?
>
> Do anyone have some tips / best practice on how to deploy applications
> built using OPAM?
>
> (I'm developing and want to deploy on Ubuntu 16.04.)
>
> Regards,
>
> Hans Ole Rafaelsen
>



-- 
Dave Scott

[-- Attachment #2: Type: text/html, Size: 3675 bytes --]

Re: [Caml-list] Create a deployable application unit from OPAM?

[Hans Ole Rafaelsen]

[-- Attachment #1: Type: text/plain, Size: 3268 bytes --]

Hi David,

It helps a bit. But my biggest problem is that the Ocaml application
(main.exe in your example) has dependencies to other Ocaml libraries built
using OPAM. I need to find a way to find those dependencies and make sure
that the paths are resolved correctly once libraries have have been copied
to the image.

-- 
Hans Ole

On Thu, Mar 29, 2018 at 8:13 PM, David Scott <scott.dj@gmail.com> wrote:

> Hi,
>
> I had a quick go at creating something with Docker: (and alpine since I
> was more familiar with it, but I suspect a similar technique would work
> with ubuntu)
>
> https://github.com/djs55/experiments/pull/1
>
> It uses `opam` to build a simple OCaml example (taken from cryptokit)
> which needs `libgmp` at runtime. The `Dockerfile` has a slightly hacky
> attempt to detect the runtime dependency:
>
> opam depext -n cryptokit | grep -v '#' | grep '\-dev' | while read in; do
> basename "$in" -dev; done > needed-packages.txt
>
> It assumes that
>
> - all external C libraries have an alpine depext called foo-dev
> - foo-dev has a corresponding runtime package called foo
>
> and installs these `foo` in an output minimal alpine image (i.e. one
> without the development tools or other unnecessary stuff)
>
> Another approach would be to use `ldd` or `objdump` recursively to find
> libraries and copy only those binaries into the output (possibly using
> `FROM scratch`?)
>
> I don't know if this will help -- I realise my test case is a bit minimal
> :-)
>
> Cheers,
> Dave
>
>
> On Thu, Mar 29, 2018 at 5:31 PM, Hans Ole Rafaelsen <hrafaelsen@gmail.com>
> wrote:
>
>> Hi,
>>
>> I have small application written using Ocsigen. Ocaml and the rest of the
>> development environment is installed using OPAM.
>>
>> Now I want to install this application on a few other nodes, without
>> having to bring bring along the whole development process for deployment on
>> each node. What I want to end up with, is a single unit, e.g a tar-file, a
>> docker container or similar, that can be copied and deployed to the target
>> nodes. This unit should only contain the binaries needed to run the
>> application.
>>
>> I have tried to ask on the Ocsigen mailing-list how do do this for
>> Ocsigen applications, but without success. But I guess the problem is more
>> related to how to deploy applications built using OPAM. I did not find any
>> way to do this with OPAM, but I might have missed something. I have tried
>> to copy all the binary files (.so, .cmxa .cmxs, etc) from the OPAM
>> directory and setting the environment variables to point the this new
>> location, but without much success. When starting ocsigenserver it was not
>> able to solve all symbols. Looking more into the problem, it looks like
>> binaries such as ocsigenserver has hard coded the location of the OPAM
>> development environment. Might be possible to override this using
>> environment variables, but I'm not sure how. So I gave up on this track.
>>
>>
>> Is there a way to generate such a deployable unit from OPAM development
>> environment?
>>
>> Do anyone have some tips / best practice on how to deploy applications
>> built using OPAM?
>>
>> (I'm developing and want to deploy on Ubuntu 16.04.)
>>
>> Regards,
>>
>> Hans Ole Rafaelsen
>>
>
>
>
> --
> Dave Scott
>

[-- Attachment #2: Type: text/html, Size: 4549 bytes --]

Re: [Caml-list] Create a deployable application unit from OPAM?

[Gerd Stolpmann]

[-- Attachment #1.1.1: Type: text/plain, Size: 5440 bytes --]

On 29.03.18 20:50, Hans Ole Rafaelsen wrote:

> Hi David,
>
> It helps a bit. But my biggest problem is that the Ocaml application
> (main.exe in your example) has dependencies to other Ocaml libraries
> built using OPAM. I need to find a way to find those dependencies and
> make sure that the paths are resolved correctly once libraries have
> have been copied to the image.
As long as the files are in the findlib tree (i.e. .opam/lib), you can
try to create a copy of the tree with the required files, and set a few
environment variables before running your code:

 - CAML_LD_LIBRARY_PATH: set this to the stublibs directory
 - OCAMLPATH set this to the root of the tree.

AFAIK ocsigen respects OCAMLPATH at runtime when dynamically loading
libraries. However, I've never tried.

The question is then which files you need. I guess at least:
 - the .cmi and .cmxs files from the subdirectories in .opam/lib

You definitely do not need:
 - any bytecode files (.cmo, cma, .so in stublibs)
 - .cmx, .cmxa

For figuring out the minimal set of ocaml packages for running, you
could look into: ocamlfind query -recursive

As you are mostly interested in deploying, and not necessarily
relocating, another strategy might be to keep the file tree at the place
where the build happens, and to just remove files you don't need. (Or
just pick the right files in your Dockerfile, and ensure that the files
end up at the same place in the image.)

These are just some wild ideas. Ocsigen is in particular complicated
because of the dynamic loading.

Gerd
>
> -- 
> Hans Ole
>
> On Thu, Mar 29, 2018 at 8:13 PM, David Scott <scott.dj@gmail.com
> <mailto:scott.dj@gmail.com>> wrote:
>
>     Hi,
>
>     I had a quick go at creating something with Docker: (and alpine
>     since I was more familiar with it, but I suspect a similar
>     technique would work with ubuntu)
>
>     https://github.com/djs55/experiments/pull/1
>     <https://github.com/djs55/experiments/pull/1>
>
>     It uses `opam` to build a simple OCaml example (taken from
>     cryptokit) which needs `libgmp` at runtime. The `Dockerfile` has a
>     slightly hacky attempt to detect the runtime dependency:
>
>     opam depext -n cryptokit | grep -v '#' | grep '\-dev' | while read
>     in; do basename "$in" -dev; done > needed-packages.txt
>
>     It assumes that
>
>     - all external C libraries have an alpine depext called foo-dev
>     - foo-dev has a corresponding runtime package called foo
>
>     and installs these `foo` in an output minimal alpine image (i.e.
>     one without the development tools or other unnecessary stuff)
>
>     Another approach would be to use `ldd` or `objdump` recursively to
>     find libraries and copy only those binaries into the output
>     (possibly using `FROM scratch`?)
>
>     I don't know if this will help -- I realise my test case is a bit
>     minimal :-)
>
>     Cheers,
>     Dave
>
>
>     On Thu, Mar 29, 2018 at 5:31 PM, Hans Ole Rafaelsen
>     <hrafaelsen@gmail.com <mailto:hrafaelsen@gmail.com>> wrote:
>
>         Hi,
>
>         I have small application written using Ocsigen. Ocaml and the
>         rest of the development environment is installed using OPAM.
>
>         Now I want to install this application on a few other nodes,
>         without having to bring bring along the whole development
>         process for deployment on each node. What I want to end up
>         with, is a single unit, e.g a tar-file, a docker container or
>         similar, that can be copied and deployed to the target nodes.
>         This unit should only contain the binaries needed to run the
>         application.
>
>         I have tried to ask on the Ocsigen mailing-list how do do this
>         for Ocsigen applications, but without success. But I guess the
>         problem is more related to how to deploy applications built
>         using OPAM. I did not find any way to do this with OPAM, but I
>         might have missed something. I have tried to copy all the
>         binary files (.so, .cmxa .cmxs, etc) from the OPAM directory
>         and setting the environment variables to point the this new
>         location, but without much success. When starting
>         ocsigenserver it was not able to solve all symbols. Looking
>         more into the problem, it looks like binaries such as
>         ocsigenserver has hard coded the location of the OPAM
>         development environment. Might be possible to override this
>         using environment variables, but I'm not sure how. So I gave
>         up on this track.
>
>
>         Is there a way to generate such a deployable unit from OPAM
>         development environment?
>
>         Do anyone have some tips / best practice on how to deploy
>         applications built using OPAM?
>
>         (I'm developing and want to deploy on Ubuntu 16.04.)
>
>         Regards,
>
>         Hans Ole Rafaelsen
>
>
>
>
>     -- 
>     Dave Scott
>
>

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #1.1.2: Type: text/html, Size: 11113 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Caml-list] Create a deployable application unit from OPAM?

[Louis Gesbert]

[-- Attachment #1: Type: text/plain, Size: 3942 bytes --]

There is no perfect solution at the moment, as there are still some hiccups with relocation of quite a few opam packages, but there are a few options:

1. Not precisely what you were asking for, since it involves re-compiling, but opam-bundle¹ can generate an all-in-one package that can easily be deployed to a different system and will bootstrap OCaml, opam, the packages and their dependencies, and can put wrappers for the binaries to your PATH. This takes a while, but avoids any relocation problems and is quite reliable.
The bundle could also be used as base for a container, or even copied across similar hosts (to the same location), and just the wrappers reinstalled.

2. opam 2.0 has an `opam install --destdir` option, that you could use to copy the installed artefacts of the required packages to some prefix (be sure to reinstall all the dependencies, or start from an empty switch). You could then tar/gz that prefix, and copy it across hosts.

3. in a simpler way, you could just copy the opam switch across hosts. With opam 2, all metadata is gathered in <switch-prefix>/.opam-switch/, so omit that subdirectory and you just get a "normal" Linux prefix (i.e. with bin/, lib/, etc. subdirectories).

Of course, 2. and 3. can hit relocation issues, depending on what the packages you are interested in use (e.g. ocsigenserver seems to have specific needs). As Gerd mentionned, specifying `CAML_LD_LIBRARY_PATH` and `OCAMLPATH` (and maybe a few others²) may help.
And none of these reduce the end result to "just the binaries needed", although 2. might help a little by letting you choose which packages get into the dest-prefix. I am not aware of a way to do this automatically. A vast super-appoximation could be to just ignore opam build-dependencies: `opam reinstall --destdir SOME_TEMP_PREFIX $(opam list --installed --rec --required-by PACKAGES --nobuild --short)`, on the origin host, then package and ship SOME_TEMP_PREFIX, possibly with wrappers to set the required PATH variables.

These are just a few clues, but hope this helps!

Louis Gesbert — OCamlPro



¹ https://github.com/AltGr/opam-bundle
² https://github.com/ocaml/opam-repository/issues/10863#issuecomment-347174662

> - Hans Ole Rafaelsen, 29/03/2018 18:31 -
> Hi,
> 
> I have small application written using Ocsigen. Ocaml and the rest of the
> development environment is installed using OPAM.
> 
> Now I want to install this application on a few other nodes, without having
> to bring bring along the whole development process for deployment on each
> node. What I want to end up with, is a single unit, e.g a tar-file, a
> docker container or similar, that can be copied and deployed to the target
> nodes. This unit should only contain the binaries needed to run the
> application.
> 
> I have tried to ask on the Ocsigen mailing-list how do do this for Ocsigen
> applications, but without success. But I guess the problem is more related
> to how to deploy applications built using OPAM. I did not find any way to
> do this with OPAM, but I might have missed something. I have tried to copy
> all the binary files (.so, .cmxa .cmxs, etc) from the OPAM directory and
> setting the environment variables to point the this new location, but
> without much success. When starting ocsigenserver it was not able to solve
> all symbols. Looking more into the problem, it looks like binaries such as
> ocsigenserver has hard coded the location of the OPAM development
> environment. Might be possible to override this using environment
> variables, but I'm not sure how. So I gave up on this track.
> 
> 
> Is there a way to generate such a deployable unit from OPAM development
> environment?
> 
> Do anyone have some tips / best practice on how to deploy applications
> built using OPAM?
> 
> (I'm developing and want to deploy on Ubuntu 16.04.)
> 
> Regards,
> 
> Hans Ole Rafaelsen
> 
> 

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Caml-list] Create a deployable application unit from OPAM?

[Hans Ole Rafaelsen]

[-- Attachment #1: Type: text/plain, Size: 6334 bytes --]

I managed to get it working.

First I tried to copy files from the lib folder (.cmxs etc), but did not
manage to solve all dependencies. It would probably have worked if I had
spent some more time figuring out what was still missing. So I installed
opam2 and installed the packages using the destdir folder (/opt/myapp)

for p in `opam2 list | cut -d ' ' -f 1`; do opam2 install --destdir
/opt/myapp/ $p ; done
(This is all, so I still have a job to do to figure out which I can do
without)

Also installed the Osigen app under /opt/myapp/

The whole folder was then copied a target node under /opt/myapp/
To solve paths I had to crate a new findlib.conf file and put it under
/opt/myapp/etc/
$ cat /opt/myapp/etc/findlib.conf
destdir="/opt/myapp/lib"
path="/opt/myapp/lib"
ocamlc="ocamlc.opt"
ocamlopt="ocamlopt.opt"
ocamldep="ocamldep.opt"
ocamldoc="ocamldoc.opt"

To run it, have have this script setting the environment
$ cat /opt/myapp/run.sh
#!/bin/bash

MYTOP=/opt/myapp/

CAML_LD_LIBRARY_PATH=$MYTOP/lib/stublibs
MANPATH=:$MYTOP/man
PERL5LIB=$MYTOP/lib/perl5:
OCAML_TOPLEVEL_PATH=/lib/toplevel
PATH=$MYTOP/bin:$PATH
OCAMLPATH=$MYTOP

export OCAMLFIND_CONF=$MYTOP/etc/findlib.conf

ocsigenserver.opt -c $MYTOP/etc/myapp/myapp.conf

Also, oscigenserver seems to use the ~/.opam2/4.06.1/ paths form extensions
loaded, so these have to be configured.
Got this kind of log messages
ocsigenserver.opt: ocsipersist:dbm: Launching a new Ocsidbm process:
/home/hans/.opam2/4.06.1/lib/ocsigenserver/extensions/ocsidbm.opt on
directory /opt/myapp/var/data/myapp/ocsipersist.

Needed to change the configuration of the module to use the right path for
the binary
vi /opt/myapp/etc/myapp/myapp.conf
...
    <extension findlib-package="ocsigenserver.ext.ocsipersist-dbm">
      <ocsidbm name="/opt/myapp/lib/ocsigenserver/extensions/ocsidbm.opt"/>
    </extension>
...
Still have a problem with mime-types, but it's probably a similar config
change.

So it's not totally smooth, but it seems to be doable.

Regards,

Hans Ole Rafaelsen




On Fri, Mar 30, 2018 at 4:11 PM, Louis Gesbert <louis.gesbert@ocamlpro.com>
wrote:

> There is no perfect solution at the moment, as there are still some
> hiccups with relocation of quite a few opam packages, but there are a few
> options:
>
> 1. Not precisely what you were asking for, since it involves re-compiling,
> but opam-bundle¹ can generate an all-in-one package that can easily be
> deployed to a different system and will bootstrap OCaml, opam, the packages
> and their dependencies, and can put wrappers for the binaries to your PATH.
> This takes a while, but avoids any relocation problems and is quite
> reliable.
> The bundle could also be used as base for a container, or even copied
> across similar hosts (to the same location), and just the wrappers
> reinstalled.
>
> 2. opam 2.0 has an `opam install --destdir` option, that you could use to
> copy the installed artefacts of the required packages to some prefix (be
> sure to reinstall all the dependencies, or start from an empty switch). You
> could then tar/gz that prefix, and copy it across hosts.
>
> 3. in a simpler way, you could just copy the opam switch across hosts.
> With opam 2, all metadata is gathered in <switch-prefix>/.opam-switch/, so
> omit that subdirectory and you just get a "normal" Linux prefix (i.e. with
> bin/, lib/, etc. subdirectories).
>
> Of course, 2. and 3. can hit relocation issues, depending on what the
> packages you are interested in use (e.g. ocsigenserver seems to have
> specific needs). As Gerd mentionned, specifying `CAML_LD_LIBRARY_PATH` and
> `OCAMLPATH` (and maybe a few others²) may help.
> And none of these reduce the end result to "just the binaries needed",
> although 2. might help a little by letting you choose which packages get
> into the dest-prefix. I am not aware of a way to do this automatically. A
> vast super-appoximation could be to just ignore opam build-dependencies:
> `opam reinstall --destdir SOME_TEMP_PREFIX $(opam list --installed --rec
> --required-by PACKAGES --nobuild --short)`, on the origin host, then
> package and ship SOME_TEMP_PREFIX, possibly with wrappers to set the
> required PATH variables.
>
> These are just a few clues, but hope this helps!
>
> Louis Gesbert — OCamlPro
>
>
>
> ¹ https://github.com/AltGr/opam-bundle
> ² https://github.com/ocaml/opam-repository/issues/10863#
> issuecomment-347174662
>
> > - Hans Ole Rafaelsen, 29/03/2018 18:31 -
> > Hi,
> >
> > I have small application written using Ocsigen. Ocaml and the rest of the
> > development environment is installed using OPAM.
> >
> > Now I want to install this application on a few other nodes, without
> having
> > to bring bring along the whole development process for deployment on each
> > node. What I want to end up with, is a single unit, e.g a tar-file, a
> > docker container or similar, that can be copied and deployed to the
> target
> > nodes. This unit should only contain the binaries needed to run the
> > application.
> >
> > I have tried to ask on the Ocsigen mailing-list how do do this for
> Ocsigen
> > applications, but without success. But I guess the problem is more
> related
> > to how to deploy applications built using OPAM. I did not find any way to
> > do this with OPAM, but I might have missed something. I have tried to
> copy
> > all the binary files (.so, .cmxa .cmxs, etc) from the OPAM directory and
> > setting the environment variables to point the this new location, but
> > without much success. When starting ocsigenserver it was not able to
> solve
> > all symbols. Looking more into the problem, it looks like binaries such
> as
> > ocsigenserver has hard coded the location of the OPAM development
> > environment. Might be possible to override this using environment
> > variables, but I'm not sure how. So I gave up on this track.
> >
> >
> > Is there a way to generate such a deployable unit from OPAM development
> > environment?
> >
> > Do anyone have some tips / best practice on how to deploy applications
> > built using OPAM?
> >
> > (I'm developing and want to deploy on Ubuntu 16.04.)
> >
> > Regards,
> >
> > Hans Ole Rafaelsen
> >
> >
>

[-- Attachment #2: Type: text/html, Size: 7488 bytes --]

Re: [Caml-list] Create a deployable application unit from OPAM?

[Francois BERENGER]

On 03/30/2018 11:11 PM, Louis Gesbert wrote:
> There is no perfect solution at the moment, as there are still some hiccups with relocation of quite a few opam packages, but there are a few options:
> 
> 1. Not precisely what you were asking for, since it involves re-compiling, but opam-bundle¹ can generate an all-in-one package that can easily be deployed to a different system and will bootstrap OCaml, opam, the packages and their dependencies, and can put wrappers for the binaries to your PATH. This takes a while, but avoids any relocation problems and is quite reliable.
> The bundle could also be used as base for a container, or even copied across similar hosts (to the same location), and just the wrappers reinstalled.

Is opam-bundle production ready?

For example, is it mature enough so that the coccinelle people can use
it to ship a source-based version of coccinelle to their users, without
requiring users to install opam first?

Regards,
F.