[Caml-list] Attn: Development Editor, Latest OCaml Weekly News

Alan Schmitt
2021-04-27T14:26:24Z

[-- Attachment #1: Type: text/plain, Size: 26103 bytes --]

Hello

Here is the latest OCaml Weekly News, for the week of April 20 to 27,
2021.

Table of Contents
─────────────────

docs.ocaml.pro : an OCaml Documentation Hub
Decompress 1.4.0
elliptic curves - maintainable and verified (full stack, from primitives to TLS)
First release of Docteur, an opiniated read-only file-system for MirageOS
Ocaml-solidity, a new OCaml library for Solidity
Migrating to floatarray (blog post)
Old CWN

docs.ocaml.pro : an OCaml Documentation Hub
═══════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/ann-docs-ocaml-pro-an-ocaml-documentation-hub/7718/1>

Fabrice Le Fessant announced
────────────────────────────

We are pleased to announce that we just published the first version of
the OCaml Documentation Hub on:

<https://docs.ocaml.pro>

The OCaml Documentation Hub can be used to browse the sources and the
documentations of more than 2000 opam packages, following links
between them when useful. This is a work-in-progress, and we are
working on improving it with many more features, such as source
annotations with types, full-text and type-driven searches,
improvements in the general readability of documentation, etc.

The site is generated using an open-source tool called digodoc,
available on:

<https://github.com/OCamlPro/digodoc>

Digodoc is able to build a map of an opam switch, with links between
files, opam packages, ocaml libraries, meta packages and ocaml
modules. It is also able to generate documentation using odoc with
cross-indexes between all these kinds of packages.

We welcome feedback and contributions! Enjoy !

Simon Cruanes said and Anil Madhavapeddy added
──────────────────────────────────────────────

Great work on this site, and I love the domain name as well ;-)

The cross linking between packages is fantastic.

As a bit of background on why documentation cross-linking has taken so
long, there is a lonnnggg history intertwined with many people's
contributions to opam, build systems (ocamlbuild and dune),
conventions (findlib and odig) and of course [odoc] itself. The major
milestones along the way have been:

• [odoc 1.0], first began in 2014 as a quick project to pull together
typing information from cmt[i] files, but which ran into the problem
that it needs a consistent set of compiled cmt files to actually
work, and so needs help from external tools to pull that set of
compiled libraries together.
• [odig], which pulls together multiple opam packages (and a
filesystem layout for metadata) and runs odoc on then. This allowed
for the creation of <https://docs.mirage.io> a few years ago which
cross-references a smaller number of packages
• opam-repo itself has had better and better bulk builds over the
years to ensure that we can actually automatically compile all the
artefacts needed for docs builds, thanks to efforts like [health
check] and [ocurrent].
• odoc 2.0, which featured a multi-year [rewrite] of the OCaml module
resolver and introduced a new [output IR]. This forthcoming release
was presented in this [OCaml 2020 talk] by @jonludlam.

And now with all these pieces in place, the OCaml documentation spring
has arrived! The OCamlPro one posted here as the first of the "new
batch" of mass documentation indexers, and I'm aware of concurrent
efforts by the odoc/ocaml.org maintainer teams to push a central one
out to ocaml.org, as well as by the MirageOS team who are refreshing
docs.mirage.io with the latest and greatest. I'm sure when the dust
has settled on all these indexers we can look for common pieces, but
for now it's lovely to see so much innovation happening at pace.

For the community: now is the time to fix your docstrings in your
libraries, as there will many cool tools parsing and processing them,
and rendering them into all kinds of output formats!

To the [odoc contributors], thank you! The journey to get to this
documentation site started here seven years ago:

┌────
│ commit ef91571cab31d9ece7af965ed52eaaff57a12efc
│ Author: Leo White <lpw25@cl.cam.ac.uk>
│ Date: Thu Oct 16 19:20:18 2014 +0100
│
│ Initial commit
└────

@lefessan one thing I'm not sure about in your site is the "copyright
library authors" claim. That's murky legal ground – it's worth
establishing if the odoc HTML has gone through a compilation process
and so is no longer copyright the authors (just as a binary output is
not copyright the original source code). If the output _is_ copyright
the authors, then they have reasonable grounds to claim that you
should also reproduce the copyright notice and other license
restrictions. Personally, I prefer to claim that there is no copyright
to the original authors in odoc output, and sidestep this issue.

[odoc] <https://github.com/ocaml/odoc>

[odoc 1.0] <https://github.com/ocaml/odoc>

[odig] <https://github.com/dbuenzli/odig>

[health check] <https://github.com/ocurrent/opam-health-check>

[ocurrent] <https://github.com/ocurrent/overview>

[rewrite] <https://github.com/ocaml/odoc/pull/439>

[output IR] <https://github.com/ocaml/odoc/pull/423>

[OCaml 2020 talk] <https://www.youtube.com/watch?v=wVyZ-KveN-w&t=3s>

[odoc contributors] <https://github.com/ocaml/odoc/graphs/contributors>

Fabrice Le Fessant replied
──────────────────────────

Thanks @avsm , all these projects were indeed important milestones
towards the creation of this site. However, I wouldn't want this
history perspective to give the wrong feeling that building this site
was easy, it is the result of a very good, long and hard work by the
team at OCamlPro to make it work despite a road paved with many
obstacles. It also benefited from OCamlPro's long history of
innovative projects for the OCaml community, that lead for example in
the past to Opam, [Try-OCaml], Memprof/[Memthol,] [Opam-builder],
[Learn-OCaml], the Typerex tools (ocp-indent, ocp-index, ocp-build,
etc.) and more recently [opam-bin] and [drom].

As I said, this is a work-in-progress, and there are many features
that we will be adding in the next months to make this website much
easier to navigate, for users to rapidely reach the information that
matters for them. We hope it will be inspirational for all the other
developers who are working on similar projects, and we are looking
forward to using their projects soon too!

[Try-OCaml] <https://try.ocamlpro.com/>

[Memthol,]
<https://www.ocamlpro.com/2020/12/01/memthol-exploring-program-profiling/>

[Opam-builder] <https://hal.inria.fr/hal-01352008>

[Learn-OCaml] <https://github.com/ocaml-sf/learn-ocaml>

[opam-bin] <https://github.com/OCamlPro/opam-bin>

[drom] <https://github.com/OCamlPro/drom/>

Daniel Bünzli said
──────────────────

I'd just like to stress that `odig' documents OCaml package installs
regardless of the package manager used as long the install structure
follows [these conventions] (which are automatically followed by [dune
installs]) .

Also for people using my packages, I'd just like to mention they may
miss important documentation bits on [https://docs.ocaml.pro] until
[that issue] is resolved.

[these conventions]
<https://erratique.ch/software/odig/doc/packaging.html>

[dune installs]
<https://dune.readthedocs.io/en/stable/opam.html#odig-conventions>

[https://docs.ocaml.pro] <https://docs.ocaml.pro/>

[that issue] <https://github.com/OCamlPro/digodoc/issues/33>

Much later in the thread, Kiran Gopinathan said
───────────────────────────────────────────────

It's not quite the same as hoogle, but merlin has a functionality to
search for functions by type signature - the feature doesn't seem to
get much attention apparently - probably the interface is a little
lacking, but with some extra elisp tuning, it can work quite smoothly:

<https://aws1.discourse-cdn.com/standard11/uploads/ocaml/original/2X/3/3c2d1c63fac7cbd7dd1bb5b9a406589e031cb795.gif>

Yawar Amin then added
─────────────────────

The command line for this:

┌────
│ ocamlmerlin single search-by-polarity -position 0 -query '-int +string'
└────

(To search for values of type `int -> string'.)

Decompress 1.4.0
════════════════

Archive: <https://discuss.ocaml.org/t/ann-decompress-1-4-0/7724/1>

Charles Edouard Lecat announced
───────────────────────────────

Greetings everyone,
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

I am happy to announce the new release of [decompress 1.4.0],
available for installation via OPAM. Decompress is a library
containing a pure OCaml implementation of several compression
algorithms:
• RFC1951
• Zlib
• Gzip
• LZO

It's goal is to provide several algorithms for both the inflation and
the deflation of objects, in the form of a stream API allowing to call
the chosen algorithm one bit at a time. Such behavior allows for an
easy use of decompress in situations where we would not be able to
have the input in one go, or where we would like to output the result
in a non blocking way. This new release comes with several
improvements to the documentation and bug fixes, but even more, with a
whole new implementation for the rfc 1951 and zlib algorithms.

[decompress 1.4.0]
<https://github.com/mirage/decompress/releases/tag/v1.4.0>

Non-stream implementation for rfc 1951 and zlib
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

Up to this day, decompress was used in several projects like
ocaml-git. However, as time passed by, it appeared that in some cases,
the current implementation of decompress was not the optimal solution:
As useful as a stream implementation is, it requires to save a lot of
information about the state of the compression, in order to resume it
once we have enough input.

This is why, in some cases where we would be sure that we have our
whole input in one go, we might want to avoid all of these side-costs,
and directly go to the point.

State of the art: libdeflate
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄

This new problematic in mind, we have started thinking about the
existing implementations of these algorithms which were also bypassing
the stream behavior. One implementation that proved to be a suitable
example for our problem, was the library `libdeflate', an
implementation in C. It's main advantages being: a better compression
ratio than zlib and with faster runtime.

It was used as the solid base for the OCaml implementation provided by
this new release.

OCaml version of libdeflate, performances and use cases
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄

Inheriting the logic of libdeflate, the new implementation now has a
better compression ratio, while being slightly faster at it. On the
other side, the decompression is way faster, with `33% of speed
increase in most tested cases: On the ~book2' (from the Calgary
corpus) file:
• `decompress' (stream): 15 Mb/s (deflation), 76 Mb/s (inflation),
ratio: 42.46 %
• `decompress' (non-stream): 17 Mb/s (deflation), 105 Mb/s
(inflation), ratio: 34.66 %

Now that this is in place, the users of decompress will be able to
choose between the two versions, according to their needs. In the case
of ocaml-git, the vast majority of the git objects are small and will
be compressed in one go. This is why we updated with the new
implementation when possible.

Writing optimized code and profiling it
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄

One of the biggest concerns of this release was to be able to produce
optimized code. The base code being coded in C, a lot of sub-optimal
behavior where ported in the OCaml version: `for' and `while' loops,
references everywhere, mixes of `struct' and `union.', it needed a lot
of clean up.

This is why once the main iteration was done, we have spent several
weeks profiling the code base, using the OCaml library `landmarks',
`flamegraph' or simply the linux binary `perf'. This work, sometimes
tedious, proved to be helpful and healthy for both the harmonization
of the code and it's performances.

Decompress & MirageOS
┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄

Compression algorithms are a really important piece in many projects,
and operating systems do not avoid this. `decompress' was coded from
the start with the idea of being used in the much larger project
MirageOS.

This release is another opportunity to broaden MirageOS’s reach, by
providing one more algorithm to it’s stack, allowing us to specialise
even more the unikernels that would have a need for
inflation/deflation algorithms. This more restrictive implementation,
as we need to have the whole input in one go, will allow us to take
advantage of the situation and give more flexibility for the user.

The positive aspects of this release will most likely show up soon
enough, as we make use of decompress to its full potential

elliptic curves - maintainable and verified (full stack, from primitives to TLS)
════════════════════════════════════════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/ann-elliptic-curves-maintainable-and-verified-full-stack-from-primitives-to-tls/7729/1>

Hannes Mehnert announced
────────────────────────

over the last month I worked on upgrading the cryptography stack for
OCaml and MirageOS. I just published a [blog post]. Enhancments of
[OCaml-TLS] ([usenix security paper from 2015]) and [X.509] are in
place.

The main achievement after TLS 1.3 support (since May 2020, 0.12.0) is
that elliptic curve certificates are now supported. Elliptic curve
cryptography uses [fiat]. The X509 implementation now supports PKCS 12
(used by browsers and other software (e.g. OpenVPN) to bundle
certificates and private keys).

Get mirage-crypto-ec, x509 0.13.0 and tls 0.13.1 (all available in the
opam-repository). Discussion and feedback appreciated.

[blog post] <https://hannes.robur.coop/Posts/EC>

[OCaml-TLS] <https://github.com/mirleft/ocaml-tls>

[usenix security paper from 2015] <https://usenix15.nqsb.io>

[X.509] <https://github.com/mirleft/ocaml-x509>

[fiat] <https://github.com/mit-plv/fiat-crypto>

First release of Docteur, an opiniated read-only file-system for MirageOS
═════════════════════════════════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/ann-first-release-of-docteur-an-opiniated-read-only-file-system-for-mirageos/7743/1>

Calascibetta Romain announced
─────────────────────────────

I'm glad to announce the first release of [`docteur'], a simple tool
to make and use (in read-only) a "file-system" for [MirageOS]. As you
know, with MirageOS, we don't have _sockets_, _kernel space_ or even
_file-descriptor_. It's not possible to manipulate files
_standalonely_ and many _primitives_ commonly available with the
`unix' module don't exists in our space.

Therefore, it is difficult to imagine making a website that displays
local files or a database system. But in our spirit of separation of
services, it becomes possible for your unikernel to communicate over
the network to a "file system" or a database.

For quite some time we have been experimenting with a file system
external to our unikernel called Git. This is the case of [`pasteur']
which saves the pastes in a Git repository. It is also the case of
[`unipi'] or [Canopy] which display the content of a Git repository
and can resynchronize with it using a hook. Or the case of [our
primary DNS server] whose zone file comes from a Git repository - we
can then trace all the changes on this file.

However, we have several limitations:
1) it requires the Git repository to load into memory in your
unikernel
2) it requires a communication (external with GitHub or internal in a
private network)

The persistent aspect is very important. We should always be able to
launch a unikernel and not lose the data if our system shuts down.

The mutable aspect (modify a file) is useful in some cases but not in
others. As for `unipi' for example (a simple static web site), the
difference between resynchronizing with a hook or restarting the
unikernel with a new version of your filesystem is minor.

[`docteur'] <https://github.com/dinosaure/docteur>

[MirageOS] <https://mirage.io/>

[`pasteur'] <https://github.com/dinosaure/pasteur>

[`unipi'] <https://github.com/roburio/unipi>

[Canopy] <https://github.com/Engil/Canopy>

[our primary DNS server] <https://github.com/roburio/dns-primary-git>

Docteur as a second solution
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌

This is where Doctor comes in. It solves both of our problems by
offering the generation of a file system from scratch:
• a Git repository (local or available on a service)
• a specific folder

Doctor is able to create a complete representation of a folder and to
compress it at such a ratio that a generation of the documentation of
several OPAM packages with all their versions making 14 Gb is reduced
to an image of only 280 Mb!

Such a high compression ratio is in particular due to a double level
of compression by [`decompress'] and [`duff']. For more details,
Docteur just generates a slightly modified PACK file with [carton].

Then, Docteur proposes a simple library which makes available 2 ways
to manipulate this image for your unikernel:
1) a way that is fast but with a consequent boot time
2) a slower way but with no cost to the boot time

The first way will simply "analyze" the image to re-extract the layout
of your file system. Then it uses the [ART data-structure] to save
this layout. So, whenever you want a specific file and according to
[ART benchmarks], you have access to the content very quickly.

The problem remains the analysis which takes place at boot time and
which can take a very long time (it depends essentially on the number
of files you have). There can also be an impact on memory usage as the
ART data structure is in memory - the more files there are, the bigger
the structure is.

The second method is more "silly". Each time you request a file, we
will have to rebuild the entire path and therefore deserialize several
objects (like folders). The advantage is that we don't analyze the
image and we don't try to maintain a layout of your file system.

[`decompress'] <https://github.com/mirage/decompress>

[`duff'] <https://github.com/mirage/duff>

[carton] <https://github.com/mirage/ocaml-git/tree/master/src/carton>

[ART data-structure] <https://github.com/dinosaure/art>

[ART benchmarks] <https://dinosaure.github.io/art/bench/find.html>

Example
╌╌╌╌╌╌╌

Docteur is meant to be simple. The generation of the image is done
very simply by the command `make':
┌────
│ $ docteur.make -b refs/heads/main https://github.com/dinosaure/docteur disk.img
│ $ docteur.make -b refs/heads/main git@github.com:dinosaure/docteur disk.img
│ $ docteur.make -b refs/heads/main git://github.com/dinosaure/docteur disk.img
│ $ docteur.make -b refs/heads/main file://$(pwd)/dev/docteur disk.img
└────

Then, Docteur proposes 2 supports: Unix & [Solo5]. For Unix, you just
have to name explicitly the image file to use. For the case of Solo5
(and thus of virtualization). You just have to find a name for a
"block device" and to reuse this name with the Solo5 "tender"
specifying where the image is.
┌────
│ $ cd unikernel
│ $ mirage configure -t unix --disk disk.img
│ $ make depends
│ $ mirage build
│ $ ./simple --filename README.md
└────

┌────
│ $ cd unikernel
│ $ mirage configure -t hvt --disk docteur
│ $ make depends
│ $ mirage build
│ $ solo5-hvt --block:docteur=disk.img -- simple.hvt --filename README.md
└────

Finally, Docteur proposes another tool that checks (and analyzes) an
image to give you the version of the commit used (if the image comes
from a Git repository) or the hash of your file system produced by the
calculation of a [Merkle tree].
┌────
│ $ docteur.verify disk.img
│ commit : ad8c418635ca6683177c7ff3b583e1ea5afea78f
│ author : "Calascibetta Romain" <romain.calascibetta@gmail.com>
│ root : bea10b6874f51e3f6feb1f9bcf3939933b2c4540
│
│ Merge pull request #11 from dinosaure/fix-tree-expanding
│
│ Fix how we expand our file-system
└────

[Solo5] <https://github.com/Solo5/solo5>

[Merkle tree] <https://en.wikipedia.org/wiki/Merkle_tree>

Conclusion
╌╌╌╌╌╌╌╌╌╌

Many times people ask me for a purpose in MirageOS such as a website
or a particular service. I think that Docteur shows one essential
thing about MirageOS, it is a tool and an ecosystem. But it's not an
endpoint that is concretized in a specific application.

Docteur is not THE solution to our problems and answers a specific use
case. What is important to note is not what Docteur does but the
possibility for our ecosystem and our tools to allow the development
of Docteur. As it allows the development of a trillion applications!

As such, I say to those people to "play" with MirageOS if they want to
learn more. Our goal is not to show you applications that you could
then deploy easily (even if we are working on this aspect too) but to
give you the possibility to imagine your OS (independently from our
vision)!

And if you try, we'll be happy to help you!

Ocaml-solidity, a new OCaml library for Solidity
════════════════════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/ocaml-solidity-a-new-ocaml-library-for-solidity/7746/1>

OCamlPro announced
──────────────────

We are pleased to announce our new OCaml library, ocaml-solidity !
[Ocaml-solidity] is a program manipulation library that provides a
Solidity parser and typechecker.

Our library is made for developers on Solidity code analysis, it
builds a typechecked AST that can be analyzed with a provided
visitor. Please note that our parser and typecheck conforms mostly to
Solidity 0.7, inline assembly is not supported. Take a look at [our
documentation].

You can test it and report bugs just [here]!

[Ocaml-solidity] <https://github.com/OCamlPro/ocaml-solidity>

[our documentation] <https://ocamlpro.github.io/ocaml-solidity/>

[here] <https://github.com/OCamlPro/ocaml-solidity/issues>

Migrating to floatarray (blog post)
═══════════════════════════════════

Archive:
<https://discuss.ocaml.org/t/migrating-to-floatarray-blog-post/7749/1>

Nicolás Ojeda Bär announced
───────────────────────────

At LexiFi we recently migrated our codebase to use `floatarray' in
place of `float array' in order to disable the "flat float array" mode
in the compiler. If you are interested in finding out more about how
we did it, we wrote a blog post about it
<https://www.lexifi.com/blog/ocaml/floatarray-migration/>. Enjoy!

Old CWN
═══════

If you happen to miss a CWN, you can [send me a message] and I'll mail
it to you, or go take a look at [the archive] or the [RSS feed of the
archives].

If you also wish to receive it every week by mail, you may subscribe
[online].

[Alan Schmitt]

[send me a message] <mailto:alan.schmitt@polytechnique.org>

[the archive] <https://alan.petitepomme.net/cwn/>

[RSS feed of the archives] <https://alan.petitepomme.net/cwn/cwn.rss>

[online] <http://lists.idyll.org/listinfo/caml-news-weekly/>

[Alan Schmitt] <https://alan.petitepomme.net/>

[-- Attachment #2: Type: text/html, Size: 38337 bytes --]