Re: [Caml-list] Accelerating compilation

[Fabrice Le Fessant <Fabrice.Le_fessant@inria.fr>]

On Fri, Sep 6, 2013 at 5:20 PM, Romain Bardou <romain.bardou@inria.fr> wrote:
>>> 3) Parallel compilation in Ocamlbuild
>>>
>>> Of course it would help but it is not easy to implement so I'm just
>>> putting it there to be exhaustive.
>>
>> I'm not sure what you are referring to, OCamlBuild does already
>> support parallel builds.
>
> Does it? I actually thought the -j option was ignored.
>
> I just did a quick test and I gain about 5 seconds with -j on a 1min15
> build (I had cleaned, recompiled and recleaned before so that caching by
> the file system would not impact the result too much), so it does seem
> to be a *little* faster :)

FWIW, I recently compiled Merlin with both ocamlbuild and ocp-build,
on a quad-core with "-j 10" for both (the link to the ocp-build
description file is in the latest OCamlPro's report), ocamlbuild needs
13s where ocp-build only needs 4s to compile everything.

As Xavier suggested, I think the main problem is that ocamlbuild must
discover the files to compile dynamically, starting from the last
module, so it cannot parallelize much. On the contrary, ocp-build has
the list of files in the description of the project, so for example,
it can call ocamldep on all of them almost at the same time.

ocp-build has an option to choose between digest and (timestamp +
inode), the default. When I benchmarked the two modes, I could notice
a small difference, but not more then 5%, so the digests are unlikely
to be responsible for the problem.

Finally, another optimization that could be considered for ocamlbuild
is to call camlp4 only once per file, and then call ocamldep, ocamlc
and ocamlopt on the preprocessed file. I could notice a huge
difference on Core, for example, when this optimization was
implemented first in ocp-build (about a 50% speed-up, if I remember
well).

--Fabrice