[Caml-list] In-memory assembly

[Caml-list] In-memory assembly

[Christoph Höger]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

Dear all,

I want to trigger OCaml compilation programmatically from a C++ application
and avoid the usage of subprocesses. It is fairly straightforward to
generate an OCaml AST, but I cannot compile it.

1. The relevant modules/libraries do not load

  utop # #require "compiler-libs.optcomp";;
  Error: Reference to undefined global `Location'

  X86_masm.generate_asm stdout [] ;;


2. The relevant modules seem to insist on file i/o for communication (i.e.,
they use out_channel objects)


Does anyone know a simple way around these issues?

a) Can I load the optcomp library without modifying the compiler (or with
some simple modifications)

b) Is there a way to create an out_channel  from a memory buffer?

thanks,

Christoph

ps: If the answer to both questions is "no", I would be willing to work
with the community and enhance the compiler if I could get some advice on
how to do it best and if such a modification is wanted.

[-- Attachment #2: Type: text/html, Size: 1169 bytes --]

Re: [Caml-list] In-memory assembly

[Gabriel Scherer]

# 1

Loading compiler-libs module from the toplevel requires some black
magic because the toplevel is already built by linking the same
libraries. (see MPR#6704,
https://caml.inria.fr/mantis/view.php?id=6704). The "ocaml" toplevel
does more black magic than "utop" and it works better there, but in
general I would recommend experimenting with non-toplevel programs for
this specific purpose.

# 2

The various x86_* modules seem to internally define a

  print_line : Buffer.t -> X86_ast.asm_line -> unit

which does not involve filesystem i/o. I think that the simplest
solution to your question would be to expose this function in addition
to the global one. (If I were you I would feel free to send a Pull
Request performing this change for all x86 printer implementations,
although I don't know whether the maintainers of this part of the
codebase would find it desirable.)

That said, what you would get is an in-memory representation of the
assembler *text*; the compiler calls an assembler and linker as
external programs. If for some reason you wanted to avoid this, you
would have to link to an assembler as a library, and then it would
possibly be easier to directly feed the (X86_ast.asm_line list)
abstract representation -- with some conversion into the format this
assembler library expects. For this you don't need to modify the
interface of the printer modules.

On Wed, Jan 10, 2018 at 11:20 AM, Christoph Höger
<christoph.hoeger@celeraone.com> wrote:
> Dear all,
>
> I want to trigger OCaml compilation programmatically from a C++ application
> and avoid the usage of subprocesses. It is fairly straightforward to
> generate an OCaml AST, but I cannot compile it.
>
> 1. The relevant modules/libraries do not load
>
>   utop # #require "compiler-libs.optcomp";;
>   Error: Reference to undefined global `Location'
>
>   X86_masm.generate_asm stdout [] ;;
>
>
> 2. The relevant modules seem to insist on file i/o for communication (i.e.,
> they use out_channel objects)
>
>
> Does anyone know a simple way around these issues?
>
> a) Can I load the optcomp library without modifying the compiler (or with
> some simple modifications)
>
> b) Is there a way to create an out_channel  from a memory buffer?
>
> thanks,
>
> Christoph
>
> ps: If the answer to both questions is "no", I would be willing to work with
> the community and enhance the compiler if I could get some advice on how to
> do it best and if such a modification is wanted.

Re: [Caml-list] In-memory assembly

[Evgeny Roubinchtein]

[-- Attachment #1: Type: text/plain, Size: 3569 bytes --]

With respect to compiler calling the assembler, there _was_ work done to
allow the compiler to generate object code directly, see:
https://github.com/bmeurer/ocamljit2, as well as
https://arxiv.org/pdf/1110.1029.pdf and https://arxiv.org/pdf/1011.6223.pdf.
Unfortunately, it looks like that effort stopped tracking the OCaml
mainline as of version 3.12.  I don't have a good idea of how much effort
it would be to resurrect that work for version 4.0.x of the OCaml
compiler.  IIRC, Benedikt Meurer reads this list, so perhaps he can
comment. :-)

-- 
Best,
Evgeny

On Wed, Jan 10, 2018 at 12:41 PM, Gabriel Scherer <gabriel.scherer@gmail.com
> wrote:

> # 1
>
> Loading compiler-libs module from the toplevel requires some black
> magic because the toplevel is already built by linking the same
> libraries. (see MPR#6704,
> https://caml.inria.fr/mantis/view.php?id=6704). The "ocaml" toplevel
> does more black magic than "utop" and it works better there, but in
> general I would recommend experimenting with non-toplevel programs for
> this specific purpose.
>
> # 2
>
> The various x86_* modules seem to internally define a
>
>   print_line : Buffer.t -> X86_ast.asm_line -> unit
>
> which does not involve filesystem i/o. I think that the simplest
> solution to your question would be to expose this function in addition
> to the global one. (If I were you I would feel free to send a Pull
> Request performing this change for all x86 printer implementations,
> although I don't know whether the maintainers of this part of the
> codebase would find it desirable.)
>
> That said, what you would get is an in-memory representation of the
> assembler *text*; the compiler calls an assembler and linker as
> external programs. If for some reason you wanted to avoid this, you
> would have to link to an assembler as a library, and then it would
> possibly be easier to directly feed the (X86_ast.asm_line list)
> abstract representation -- with some conversion into the format this
> assembler library expects. For this you don't need to modify the
> interface of the printer modules.
>
> On Wed, Jan 10, 2018 at 11:20 AM, Christoph Höger
> <christoph.hoeger@celeraone.com> wrote:
> > Dear all,
> >
> > I want to trigger OCaml compilation programmatically from a C++
> application
> > and avoid the usage of subprocesses. It is fairly straightforward to
> > generate an OCaml AST, but I cannot compile it.
> >
> > 1. The relevant modules/libraries do not load
> >
> >   utop # #require "compiler-libs.optcomp";;
> >   Error: Reference to undefined global `Location'
> >
> >   X86_masm.generate_asm stdout [] ;;
> >
> >
> > 2. The relevant modules seem to insist on file i/o for communication
> (i.e.,
> > they use out_channel objects)
> >
> >
> > Does anyone know a simple way around these issues?
> >
> > a) Can I load the optcomp library without modifying the compiler (or with
> > some simple modifications)
> >
> > b) Is there a way to create an out_channel  from a memory buffer?
> >
> > thanks,
> >
> > Christoph
> >
> > ps: If the answer to both questions is "no", I would be willing to work
> with
> > the community and enhance the compiler if I could get some advice on how
> to
> > do it best and if such a modification is wanted.
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

[-- Attachment #2: Type: text/html, Size: 4913 bytes --]

Re: [Caml-list] In-memory assembly

[Basile Starynkevitch]

On 01/10/2018 11:20 AM, Christoph Höger wrote:
> Dear all,
>
> I want to trigger OCaml compilation programmatically from a C++ 
> application and avoid the usage of subprocesses. It is fairly 
> straightforward to generate an OCaml AST, but I cannot compile it.


You should explain what kind and size is the generated code, and why do 
you want to avoid the usage of subprocesses.
My impression is that you should not be afraid of using subprocesses and 
temporary files.

Some hints and feelings (which I did not got from generating Ocaml code, 
but C code, in my past http://starynkevitch.net/Basile/gcc-melt/ project)


Generating Ocaml code is a good idea (I am very fond of metaprogramming 
in general). Perhaps you should into MetaOcaml (see 
http://okmij.org/ftp/ML/MetaOCaml.html for more)

I'm guessing you might generate a non-tiny amount of Ocaml code, and 
since you want to use the native Ocaml compiler, that code will run for 
a non-short period of time. Notice that if both assumptions are false 
(tiny amount of generated code, running for a short period of time) 
you'll be happy by embedding the Ocaml bytecompiler in your C++ 
application (and there is no real point in using the native Ocaml 
compiler), which I believe is a solved problem.

So, I am assuming you generate a non-small Ocaml file (e.g. a thousand 
lines), and it runs for a non-tiny amount of time (e.g. at least half a 
second, or perhaps 0.1 second).

Then you probably want to ask your ocamlopt compiler to optimize, e.g. 
with its -O2 flag. This probably takes a few tenths of seconds (on a 
file of a few thousand lines of Ocaml), or more. That delay (even 0.1 
seconds of compilation time by ocaml) is long enough to make the usage 
of subprocesses insignificant: a modern system can run hundreds of 
processes every second.

If you care about filesystem delays (which are not that important), 
consider putting the generated Ocaml source code on some fast 
filesystem, e.g. some tmpfs on Linux.

Cleaning up temporary files and running short processes from C++ code is 
a solved problem (and a very common practice).

Cheers.


Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
opinions are mine only - les opinions sont seulement miennes
Bourg La Reine, France