[Caml-list] factor 5 speed increase for natively compiled camlp4 parsers

[Hendrik Tews <tews@tcs.inf.tu-dresden.de>]

Dear all,

yesterday I managed to natively compile a custom camlp4
application. The speed increase was dramatic: from 10 seconds
down to 1.9 seconds for 1.4 MB of Ocaml code in 121 files. Below
I describe howto compile your camlp4 parser to native-code.
Further below there are some questions for the camlp4
maintainers.


HOW TO GET ``camlp4 pa_o.cmo pa_op.cmo foo.cmo'' DOWN TO NATIVE-CODE?

1. Prerequisites: 

   Some of the files you need are not installed in the standard
   ocaml distribution. Therefore you need a fully configured
   ocaml source tree. Say its located at ${ocamlsource}. Do a
   ``make world.opt'' there. (But don't clean!)

2. Compile the camlp4 extensible grammars for ocaml to native-code.

   cp ${ocamlsource}/camlp4/etc/pa_o.ml .
   ocamlopt -c -I `camlp4 -where` -pp "camlp4r pa_extend.cmo q_MLast.cmo" \
	-o pa_o.cmx pa_o.ml
   cp ${ocamlsource}/camlp4/etc/pa_op.ml .
   ocamlopt -c -I `camlp4 -where` -pp "camlp4r pa_extend.cmo q_MLast.cmo" \
	-o pa_op.cmx pa_op.ml

   You might want to copy the two cmx files to some ${lib}
   directory. If this differs from ``ocamlc -where'' you have to
   add ``-I ${lib}'' in step 4 below.

3. foo.cmx: Compile all the modules you want to load to camlp4 to
   native-code.

4. Do

   ocamlopt -linkall -I `camlp4 -where` \
      odyl.cmxa camlp4.cmxa \
      pa_o.cmx pa_op.cmx \
      pr_dump.cmx \
      foo.cmx \
      ${ocamlsource}/camlp4/odyl/odyl.cmx

   Note the -linkall! The second line is for the camlp4 runtime
   system. The third line is for standard ocaml syntax (with
   optimized parsers). The fourth line contains the standard
   printer (leave it out if you use your own printer). The fivth
   line is your camlp4 extension. The last line is for the main
   program.

   If you get wired errors about non-matching interfaces: Make
   sure your ocaml source tree is in sync with our ocaml
   installation!


4. Try it! I tried it on Otags, the TAGS generator from
   Jean-Francois Monin. Otags just uses camlp4 parsers and some
   customized printers (_not_ the usual camlp4 preprocessor to
   ocamlc approach). My biggest ocaml repository contains 51000
   lines of code in 121 files (1.4 MB). TAGS generation time
   drops from 10.4 seconds to 1.9 seconds. (I'll post an Otags
   patch in a separate email in the near future.)


WHAT IF YOU DON'T NEED EXTENSIBLE GRAMMARS?

   Substitute ``${ocamlsource}/camlp4/compile/pa_o_fast.cmx'' for
   ``pa_o.cmx pa_op.cmx'' in line 3 in step 4 above. 

   Apparently there exists some way to compile camlp4 EXTEND
   expressions to plain stream parser source code. This is used
   during the ocaml build to generate pa_o_fast.ml from pa_o.ml
   and pa_op.ml. camlp4o.opt contains pa_o_fast.cmx.

   Applied to Otags this increases its speed by another factor 
   of 2. Now it takes mearly 1.2 second to generate the TAGS.
   This is almost 10 times faster than the original!


WHAT ABOUT THE REVISED SYNTAX?

   Leave out step two and substiture ``pa_r.cmx pa_rp.cmx'' in
   line 3 in step 4 above. 

   There is no pa_r_fast.ml. camlp4r.opt contains extensible
   grammars. 


QUESTIONS TO EXPERTS AND CAMLP4 MAINTAINERS:

1. Why is the -linkall neccessary? Its clear to me that one needs
   the -linkall when building camlp4o. However, I don't
   understand why camlp4o.opt needs the -linkall.

2. Is the process that generates pa_o_fast.ml applicable to all
   camlp4 grammars? Why is it not used for the revised syntax?
   What is the option -meta_action of pa_extend.cmo doing?

3. Would it be possible to include a ``mknativecamlp4'' script
   that automates the described procedure in the next
   distribution? 

   Would it be possible to build and install pa_o.cmx, pa_op.cmx,
   pa_o_fast.cmx, and odyl.cmx? 


Bye,

Hendrik

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners