[Caml-list] [ANN] Core Suite 109.23.00 + async_parallel

[Caml-list] [ANN] Core Suite 109.23.00 + async_parallel

[Jeremie Dimino]

I am pleased to announce the 109.23.00 release of the Core suite.

The following packages were upgraded:

- async_parallel
- core
- core_extended
- jenga

async_parallel is a new library.  It is for running tasks in other
processes on a cluster of machines.  A detailed description is
included in the API documentation:

  https://ocaml.janestreet.com/ocaml-core/109.23.00/doc/parallel/Std.html

Files and documentation for this release are available on our website
and all packages are in opam:

  https://ocaml.janestreet.com/ocaml-core/109.14.00/individual/
  https://ocaml.janestreet.com/ocaml-core/109.14.00/doc/

Here are the changelogs for versions 109.15.00 to 109.23.00:

# 109.15.00

## async_extra

- In `Rpc.client` and `Rpc.with_client`, allowed the client to
  implement the rpcs.

  Added a new optional argument: `?implementations:_ Client_implementations.t`.
- Added new module `Versioned_rpc.Both_convert` to allow the caller
  and callee to independently upgrade to a new rpc.

  This is a new flavor of `Versioned_rpc` in which both sides do some
  type coercions.

## async_unix

- The `epoll`-based scheduler now supports sub-millisecond timeouts,
  using `Linux_ext.Timerfd`.

  Async still uses the `select`-based scheduler by default.  We plan
  to switch the default to `epoll` in a few weeks, once we have done
  more testing.
- Eliminated module `Work_group`, which was for limiting the number of
  threads used by jobs.

  This was a little-used module that significantly complicated the
  implementation of the Async thread pool.

  One should consider using a `Throttle` instead.

  Along the way, fixed a bug in Async helper threads in which the
  finalizer could fire too early, causing an unhandled exception.  The
  fix involves relaxing the requirements on when
  `Thread_pool.finished_with_helper_thread` functions can be called,
  allowing it to be called while the helper thread still has work, but
  so long as no future work will be added.

## core

- Changed the tolerance of `Time.Robustly_compare` functions from
  `1E-7` to `1E-6`.
- Fixed the names of some toplevel pretty-printers, which referred to
  nonexistent modules.

  Fix some of the `pp`'s for Core which are used to install printers
  in the top-level.  Some of the toplevel printers refer to
  non-existent modules like `Core.Nativeint.pp`; this feature changed
  to the correct name, like `Core.Std.Nativeint.pp`.
- Added to module `Unix` functionality for getting and setting flags
  in the open-file-descriptor table.

  ```ocaml
  module Open_flags : sig type t include Flags.S with type t :` t  ...  end
  val fcntl_getfl : File_descr.t -> Open_flags.t
  val fcntl_setfl : File_descr.t -> Open_flags.t -> unit
  ```
- Added module `Linux_ext.Timerfd`.

  This allows one to create a file descriptor that can be monitored by
  `epoll` or `select` and notify them at a certain time.  It makes it
  possible to use `epoll` with sub-millisecond timeouts.
- Added `Version_util.application_specific_fields`, which allows
  custom build-time information to be included in an executable.

## zero

This is a new library with general data structures like `Core` but
that emphasizes performance over usability.

- Added module `Pool`.

  This is an implementation of a heap-block allocator that provides
  the API of a set of tuples.  The point is to allocate a single
  long-lived block of memory (the pool) that lives in the OCaml major
  heap, and then to reuse the block, rather than continually
  allocating blocks on the minor heap, and then promoting them to the
  major heap.

# 109.17.00

## async_extra

- Added an option to `Async.Log.Rotation` to include the date in
  logfile names.

  This is mostly for archiving purposes.
- Made `Versioned_rpc.Callee_converts.Pipe_rpc.implement_multi` agree
  with `Rpc.Pipe_rpc.implement` on the type of pipe rpc
  implementations.
- Improved the performance of `Versioned_typed_tcp`.

  Avoided creating deferreds while reading the incoming messages.

## core

- Fixed `Random.self_init`, which was broken since 109.00.00 with the
  upgrade to OCaml 4.0

  The fix changed the type signature expressed in `core_random.ml` of
  the standard OCaml `caml_sys_random_seed` C function from `unit ->
  int` from `unit -> int array`.  That C function changed between
  OCaml 3.12 and 4.0.
- Moved module `Core_extended.Unix.Cidr` into `Core.Unix`.
- Wrapped `Unix.wordexp` into an `Or_error.t` to handle systems that
  does not implement it in the libc.
- Fixed two other printer names
- Added `Array.int_blit` and `Array.float_blit`, which are specialized
  fast blits for `int array` and `float array`.

  For motivation underlying this change and other design alternatives
  please see Section 3 "Fast, Slow and Incorrect Array blits" of
  http://janestreet.github.com/ocaml-perf-notes.html
- Added `Unpack_buffer.Unpack_one.sexp` for parsing sexps using the
  `Unpack_buffer` interface.

## zero

- Fixed `Obj_array.blit` to safely handle overlapping blits.

# 109.18.00

## async_unix

- added `Async.Unix.fcntl_{get,set}fl`.

  Made `Reader` and `Writer` detect if they are passed a file
  descriptor with incorrect permissions (`O_WRONLY` for `Reader`,
  `O_RDONLY` for `Writer`).

## core

- changed implementation of `Array.sort` to use introsort.

  See http://en.wikipedia.org/wiki/Introsort.
- tweaked a unit test in `Core.Flags` to not print a message to
  stderr.

## pa_ounit

- a number of improvements to `inline_tests_runner`, including a
  `-verbose` flag.

  1. Made pa_ounit errors more readable.
  2. Added `-verbose` flag.
  3. Made the `-only-test` locations compatible with those displayed
     by the `-verbose` flag.
  4. Renamed `-display` as `-show-counts` to avoid confusion with
     `-verbose`.
  5. Improved errors when parsing the command line.
  6. Updated the readme.
  7. Added a `-list-test-names` which shows what tests would be run,
     if this option was not given.

# 109.19.00

## async_extra

- Added function `Versioned_typed_tcp.Client.flushed : t ->
  [ `Flushed | `Pending of Time.t Deferred.t ]`.

  This exposes whether the underlying `Writer.t` has been flushed.

## async_unix

- Reworked a number of `Reader` functions to improve performance by
  avoiding deferreds.

  This is a followup to the `Reader` improvements in 109.14, and
  eliminates some last vestiges of performance degradation that had
  been introduced in 109.04.
- Added function `Reader.lseek : t -> int64 -> mode:[< `Set | `End] ->
  int64 Deferred.t`.

  `lseek t offset ~mode` clears `t`'s buffer and calls `Unix.lseek` on
  `t`'s file descriptor.
- Added function `Writer.bytes_received : t -> int`.
- Added function `Unix.mkfifo : ?perm:file_perm -> string -> unit
  Deferred.t`, which was mistakenly missing.

  This is a simple wrapper around `Core.Unix.mkfifo`.

## core

- Changed `Time.to_string` and `Time.sexp_of_t` to include the
  timezone.

  This is an incompatible change with very old programs in which
  `Time.of_string` and `Time.t_of_sexp` did not support the timezone.

  If you have programs that are:

  * very old and do Time string/sexp handling
  * rely on reading in time values without using `Time.of_string` and
    `Time.t_of_sexp`.
  * rely on chains of writing/reading/writing times across machines
    and timezones where the time is always intended to be taken as the
    local time on the currently reading machine

  you should recompile/review your code to make sure you won't have
  issues.
- Added function `List.remove_consecutive_duplicates : 'a t ->
  equal:('a -> 'a -> bool) -> 'a t`.

  This returns the input list with consecutive duplicates removed, and
  doesn't change the order of the remaining elements.
- Added module `User_and_group`, which is a pair of a unix username
  and primary unix group.

  The string/sexp converters follow the usual unix convention of
  `<user>:<group>`.
- Added function `Date.first_strictly_after : t -> on:Weekday.t -> t`.

  `first_strictly_after t ~on:day_of_week` returns the first
  occurrence of `day_of_week` strictly after `t`.
- Added functor `Type_equal.Lift`.

  It is always safe to conclude that if type `a` equals `b`, then type
  `a X.t` equals `b X.t`, for any type `X.t`.  The OCaml type checker
  uses this fact when it can.  However, sometimes, e.g. when using
  `Type_equal.conv`, one needs to explicitly use this fact to
  construct an appropriate `Type_equal.t`.  The `Type_equal.Lift*`
  functors do this.

  ```ocaml
  module Type_equal : sig
    type ('a, 'b) t
    ...
    module Lift (X : T1) : sig
      val lift : ('a, 'b) t -> ('a X.t, 'b X.t) t
    end
  end
  ```

## fieldslib

- Made `with fields` generate the same functions in the `Fields` and
  `Fields_of_*` modules whether the type is called `t` or not.

## zero

- Removed a performance test from `Pool` that was nondeterministically
  failing.

# 109.20.00

## async_core

- Added the ability for a `Throttle` to have resources that are
exclusively available to running jobs.

## async_unix

- Set `close-on-exec` for both ends of the pipe used to wake up the scheduler.

## core

- Wrapped `Unix.wordexp` in an `Or_error.t` since it is not available
on all systems.

- Added function `Process_env.parse_ssh_client`.
  This gets the address from which you're currently ssh'd in.

- Added to `Unix` module the ability to get and set
`IP_MULTICAST_LOOP` and `IP_MULTICAST_TTL`.

- Exposed module `Core.Std.Ref`, which was previously only available
via `Core.Ref`.

- Remove `Mutex.am_holding_mutex` and changed the type of `Mutex.try_lock`.

  With NPTL it is impossible to determine which thread is holding the
  lock.  So, `Mutex.am_holding_mutex` is unimplementable.  Also,
  `Mutex.try_lock` was incorrect because it claimed to raise if one was
  attempting to recursively lock.  Since it's not possible to
  distinguish between recursive locking and the lock being held by
  another thread, we changed the type to make this clear:

  ```ocaml
  val try_lock : t -> [ `Already_held_by_me_or_other | `Acquired ]
  ```

- Removed our custom version of the OCaml runtime's `core_sys_open` function.

  There used to be a bug in the OCaml runtime, PR#5069, in which
  `open_{in,out}_gen` could block while holding the OCaml lock, because
  they made a call to `fcntl` outside the blocking section.  We had our
  own C code with the bug fix and re-exposed the fixed versions of the
  functions in `Core`.

  The bug in OCaml has been fixed, so we have removed our patched
  function from `Core`.

- In `unix_stubs.c`, switched from using `FNM_FILE_NAME` to `FNM_PATHNAME`.

  The GNU project introduced FNM_FILE_NAME as a non-portable synonym for
  FNM_PATHNAME.

  We were using pre-processor macros to define FNM_FILE_NAME as
  FNM_PATHNAME if unavailable, but it is simpler to just use the more
  portable FNM_PATHNAME everywhere.

## ocaml_plugin

- Removed a test that (rarely) failed nondeterministically.

## sexplib

- Renamed converter generated by `with sexp` for polymorphic variants
so it is hidden from the toplevel.

  `of_sexp` created a value named `<type>_of_sexp__` to handle
  polymorphic variants.  To hide it from the toplevel, we renamed it as
  `__<type>_of_sexp__`.  We kept the `__` suffix to avoid any confusion
  with a type named `__<type>`.

## type_conv

- Removed some warnings caused by generated signatures.

  1. In signatures on local modules.
  2. When there are duplicate signature items like in this example:

    ```ocaml
    module Warnings : sig
      type t = private { foo : int } with fields (** used to say
unused value foo *)
      val foo : string
    end = struct
      type t = { foo : int } with fields
      let foo = "a"
    end
    ```

  3. In the signatures of all the parameters of functors that take multiple
     parameters; this used to work only for the last parameter.

## zero

- Added module `Timing_wheel`, a new priority-queue implementation
specialized for sets of time-based alarms.

# 109.21.00

## async_unix

- Added `Unix.remove`.

## core

- Massively improved the signatures of `Map` and `Set`, both for
  readability and ocamldoc, as well as improved type error messages.

  For instance the type of `Int.Set.singleton` was:

  ```ocaml
  ('a, 'comparator, 'a Core.Std.Int.Set.elt_ -> ('a, 'comparator)
Core.Std.Int.Set.t_) Core.Core_set_intf.without_comparator
  ```

  Now it is simply:

  ```ocaml
  int -> Int.Set.t
  ```
- Added an optional argument to `Command.run` that can be used to
  specify default flags from a user config file.

  The optional argument can extend the command line based on the path
  to the command.
- Rename module `Weekday` as `Day_of_week`.

  The name `Weekday` conflicted with ordinary usage of "weekday" to
  mean Monday through Friday.
- Changed `sexp_of_t` for `{Month,Ofday,Time,Time.Span}.{Set,Map}` to
  use the nice sexp format of the underlying atomic type.

  Previously, the converter had used thes raw type (`float`, `int`,
  etc.).  `t_of_sexp` still accepts both formats; we will remove the
  ability to accept the raw format in the distant future.

  This output-format change was planned when we originally in 108.06b
  improved those `t_of_sexp` functions to accept both formats.
- Added `Unix.remove`.
- Removed some `IFDEF`'s connected to OCaml <4 support.

## zero

- Added module `Flat_tuple_array` for tuples packed into an array.
- Added function `Pool.length : _ t -> int`, which returns the number
  of live tuples in a pool.

# 109.23.00

## core

- Exposed `Core.Std.Flags` module.
- Made the `Heap` module implement `Container.S1`.
- Added module `Ref.Permissioned`, which is a ref with `read_only` /
  `read_write` access control.
- Exposed the unique id in `Type_equal.Id`.

  This allows, e.g. hash tables indexed by unique ids.
- Removed the use of `Obj.magic` from the implementation of
  `Type_equal.Id.same`.

  It is not needed because the `Id.t` contains a `Uid.t` and we can
  just use `Uid.equal`.

-- 
Jeremie Dimino, for the Core team