[Caml-list] [ANN] Core Suite 112.01.00 + rpc_parallel

[Jeremie Dimino <jdimino@janestreet.com>]

[-- Attachment #1: Type: text/plain, Size: 10887 bytes --]

I am pleased to announce the 112.01.00 release of the Core suite.
Starting with this releases core packages require OCaml 4.02.

We are also releasing a new package: rpc_parallel. From the package
description:

  Rpc_parallel offers an API to define various workers and protocols,
  spawn workers as separate processes, and communicate with them using
  Async Rpc.

The following packages were upgraded:

- async
- async_extended
- async_extra
- async_kernel
- async_unix
- bignum
- bin_prot
- core_bench
- core
- core_extended
- core_kernel
- custom_printf
- jenga
- ocaml_plugin
- rpc_parallel
- sexplib
- textutils
- type_conv

Files for this release are available on our website and all packages
are in opam:

  https://ocaml.janestreet.com/ocaml-core/112.01.00/individual/

Unfortunately the documentation is not there yet due to opam-doc not
working with 4.02.

Here is list of changes for this version:

# 112.01.00

## async

- update tests

## async_extended

- Clarified an error in `Rpc_proxy`.

## async_extra

- Changed `Persistent_rpc_client.connected` to avoid returning a
  connection that is closed at the time it was called.
- Optimized `Rpc.implement` so that if a server's implementation
  returns a determined deferred, then the output is immediately
  serialized and written out for the client.

  This reduces memory consumption, improves throughput and latency.
  Measurements with the `pipe_rpc_test program` showed that a server
  went from processing 600\_000 msg/sec, to 2\_200\_000 msg/sec before
  pegging the CPU.
- Changed `Log`'s output processor's batch size from `1_000` to `100`.
- Added `Persistent_rpc_client.close` and `close_finished`.
- In `Rpc.Connection.client` and `with_client`, used the
  `handshake_timeout` as the `timeout` passed to `Tcp.connect`.

  `handshake_timeout` was previously used only for the `Rpc` module's
  handshake timeout.
- Changed `Rpc.create`'s `on_unknown_rpc` argument, renaming
  `\`Ignore` as `\`Close_connection`, and requiring `\`Call` to return
  `\`Close_connection` or `\`Continue`.

  `\`Ignore` was renamed because it was a poor name, since in fact it
  closed the connection.

  Added a `\`Continue` option, whic allows one to keep the connection
  open.

  Changed `\`Call` to return `\`Continue` or `\`Close_connection`,
  where the old `unit` return value meant `\`Close_connection`.
- In `Versioned_typed_tcp`, enabled the use of "credentials" in the
  "Hello" message.

  Propagate credentials to the user code when it arrives on the wire.

## async_kernel

- Optimized `Monitor.try_with ~run:\`Now f` to return a determined
  deferred if `f ()` returns a determined deferred.

  Previously, `Monitor.try_with ~run:\`Now f` always introduced a
  `Deferred.map`, which made it impossible to do some optimizations
  that bypass the scheduler overhead.
- Added an `ASYNC_CONFIG` field that causes the program to dump core
  if Async jobs are delayed too much.

  The new field is `dump_core_on_job_delay`.
- Switched `Async_kernel` from using `Core.Sys` to `Pervasives.Sys`
  eliminating one of the dependencies on `Core`.

## async_unix

- Changed `Writer.transfer write pipe` to close `pipe` when the
  `writer`, is closed.

  Previously, `Writer.transfer` did not close the pipe when the
  underlying writer is closed.  This was strange because:

  1. Callers would have to consistently check for the writer being
     closed and close the `Pipe.Reader`t= themselves
  2. The analogous function `Pipe.transfer` closes the reader on
     similar circumstances.

  The absence of the close was noticed as a bug in `Rpc`, which
  assumed that `Writer.transfer` did the close.
- Fixed a bug in `Scheduler.yield` that caused it to pause for 50ms if
  there is no other pending work and no I/O.
- Exposed type equivalence between `Unix.Passwd.t` and
  `Core.Std.Unix.Passwd.t`.
- Changed `Writer.write_bin_prot` to use the new
  `Bigstring.write_bin_prot`.

## bignum

- Added `Bignum.Bigint` module, with arbitrary-precision integers
  based on `Zarith`, which is significantly faster than the
  `Num.Big_int` library.

## bin_prot

- In `Write`, improved some OCaml macros to name values and avoid
  calling C functions multiple times.

## core

- Removed vestigial code supporting OCaml 4.00.
- Added `Command` support for flags that are passed one or more times.

  Added `Command.Spec.one_or_more` and
  `Command.Spec.non_empty_sequence` to deal with the cases where you
  expect a flag or anonymous argument (respectively) to be passed one
  or (optionally) more times.  This is common enough and distinct from
  the case where you want the argument passed zero or more times that
  it seems like we should canonize it in the library.
- In `Lock_file`, made stale lock detection more robust.

  Made `Lock_file.create foo` succeed if `foo` is absent and
  `foo.nfs_lock` file is present and stale.  Previously, it would
  fail.
- Removed `Syslog.syslog`'s `add_stderr` argument; use the `PERROR`
  option instead.
- Fixed `unix_stubs.c` compilation on NetBSD

  Closes #45
- Added `Filename` operators `/^` and `/@`, and `of_parts`, like the
  same functions for Catalog paths.
- Changed `Iobuf` functions that advance the iobuf to not also return
  a redundant number of bytes processed.

  This avoids a small allocation (in the case of the `int option`
  functions) and normalizes the result (so the same information isn't
  returned two ways).  Actually, it doesn't yet avoid the allocation in
  the implementation, as the corresponding `Bigstring` functions must
  still return the number of bytes processed, and currently do so as an
  option.  We hope to eventually change that.

  In the future I expect we will change `unit` to some `error` variant
  to also avoid the exception construction for `EWOULDBLOCK/EAGAIN`.  We
  can even make Unix syscalls `noalloc` if we're careful.
- In `Unix` module, added unit tests for `Cidr.does_match`.

## core_bench

- fixed legacy format string

## core_extended

- Added `Float_ref` module, which is like `float ref` but faster for
  sets due to bypassing the write barrier.

  Benchmark results on Sandy Bridge:

  | [float\_ref.ml:] float ref set  |    2\_886.94ns |   8.00w |
 |
  | [float\_ref.ml:] Float\_ref.set |       355.76ns |   6.00w |
 |
  | [float\_ref.ml:] float ref get  |       415.52ns |   6.00w |
 |
  | [float\_ref.ml:] Float_ref.get  |       416.19ns |   6.00w |
 |
- Added `Bin_io_utils.Wrapped.t`, which defines an `'a t with bin_io`
  that supports size-prefixed serialization and deserialization.

  `Wrapped` has two useful submodules, `Opaque` and `Ignored`, for
  efficient handling of size-prefixed bin-io values in cases where
  serialization can be bypassed.  See the comments in the module for
  more details.

## core_kernel

- Removed vestigial code supporting OCaml 4.00.
- Used `{Hashable,Comparable}.S_binable` in `Day_of_week` and `Month`.
- Improved the performance of `Set_once.set`.
- Added `Type_equal.Lift3` functor.
- Replaced occurrences of `Obj.magic 0` with `Obj.magic None`.

  With the former the compiler might think the destination type is
  always an integer and instruct the GC to ignore references to such
  values.  The latter doesn't have this problem as options are not
  always integers.
- Made `String_id.of_string` faster.
- Added `Bigstring` functions for reading and writing the
  size-prefixed bin-io format.

  - `bin_prot_size_header_length`
  - `write_bin_prot`
  - `read_bin_prot`
  - `read_bin_prot_verbose_errors`
- Added `{Info,Error}.to_string_mach` which produces a single-line
  sexp from an `Error.t`.
- Added `{Info,Error}.createf`, for creation from a format string.
- Added new `Perms` module with phantom types for managing access
  control.

  This module supersedes the `read_only`, `read_write`, and
  `immutable` phantom types, which are now deprecated, and will be
  removed in the future.  This module uses a different approach using
  sets of polymorphic variants as capabilities, and contravariant
  subtyping to express dropping capabilities.

  This approach fixes a bug with the current phantom types used for
  `Ref.Permissioned` in which `immutable` types aren't guaranteed to
  be immutable:

  ```ocaml
  let r = Ref.Permissioned.create 0
  let r_immutable = (r :  (int, immutable) Ref.Permissioned.t)
  let () = assert (Ref.Permissioned.get r_immutable = 0)
  let () = Ref.Permissioned.set r 1
  let () = assert (Ref.Permissioned.get r_immutable = 1)
  ```

  The bug stems from the fact that the phantom-type parameter is
  covariant, which allows OCaml's relaxed value restriction to kick
  in, which allows one to create a polymorphic value, which can then
  be viewed as both immutable and read write.  Here's a small
  standalone example to demonstrate:

  ```ocaml
  module F (M : sig
              type +'z t
              val create : int -> _ t
              val get : _ t -> int
              val set : read_write t -> int -> unit
            end) : sig
    val t : _ M.t
  end = struct
    let t = M.create 0
    let t_immutable = (t :  immutable M.t)
    let () =
      assert (M.get t_immutable = 0);
      M.set t 1;
      assert (M.get t_immutable = 1);
    ;;
  end
  ```

  The new approach fixes the problem by making the phantom-type
  parameter contravariant, and using polymorphic variants as
  capabilities to represent what operations are allowed.
  Contravariance allows one to drop capabilities, but not add them.
- Added `Int.Hex` module, which has hexadecimal sexp/string
  conversions.
- Added `Gc.major_plus_minor_words`, for performance reasons.

## custom_printf

- Fixed uses of `printf=-style`format strings that have unspecified
  behavior in OCaml 4.02 and will become errors.
- Support substitution in format string (lost with 4.02 compatibility)

## jenga

- Don't show noisy `glob..changed` messages except with
`-show-glob-changed` flag.
- Support shared build rules via `${jenga}/share`.
- Detect cycle in dep scheme instead of hanging.
- Made standalone actions atomic, just like actions associated with
  target files.

  Running actions and recording the result in the persistent
  `.jenga.db` should be performed atomically for standalone actions,
  as it is for actions which are associated with target files

## ocaml_plugin

- Changed to not use `rm -r` when it is expected to remove one file.

## rpc_parallel

Initial import.

## sexplib

- Replaced occurrences of `Obj.magic 0` with `Obj.magic None`.

  With the former the compiler might think the destination type is
  always an integer and instruct the GC to ignore references to such
  values.  The latter doesn't have this problem as options are not
  always integers.

## textutils

- Added `Ascii_table.Table_char`, to expose the special table border
  characters.

## type_conv

- Updated ast matching for 4.02

-- 
Jeremie Dimino, for the Core team

[-- Attachment #2: Type: text/html, Size: 14982 bytes --]