I am pleased to announce the 112.01.00 release of the Core suite. Starting with this releases core packages require OCaml 4.02. We are also releasing a new package: rpc_parallel. From the package description: Rpc_parallel offers an API to define various workers and protocols, spawn workers as separate processes, and communicate with them using Async Rpc. The following packages were upgraded: - async - async_extended - async_extra - async_kernel - async_unix - bignum - bin_prot - core_bench - core - core_extended - core_kernel - custom_printf - jenga - ocaml_plugin - rpc_parallel - sexplib - textutils - type_conv Files for this release are available on our website and all packages are in opam: https://ocaml.janestreet.com/ocaml-core/112.01.00/individual/ Unfortunately the documentation is not there yet due to opam-doc not working with 4.02. Here is list of changes for this version: # 112.01.00 ## async - update tests ## async_extended - Clarified an error in `Rpc_proxy`. ## async_extra - Changed `Persistent_rpc_client.connected` to avoid returning a connection that is closed at the time it was called. - Optimized `Rpc.implement` so that if a server's implementation returns a determined deferred, then the output is immediately serialized and written out for the client. This reduces memory consumption, improves throughput and latency. Measurements with the `pipe_rpc_test program` showed that a server went from processing 600\_000 msg/sec, to 2\_200\_000 msg/sec before pegging the CPU. - Changed `Log`'s output processor's batch size from `1_000` to `100`. - Added `Persistent_rpc_client.close` and `close_finished`. - In `Rpc.Connection.client` and `with_client`, used the `handshake_timeout` as the `timeout` passed to `Tcp.connect`. `handshake_timeout` was previously used only for the `Rpc` module's handshake timeout. - Changed `Rpc.create`'s `on_unknown_rpc` argument, renaming `\`Ignore` as `\`Close_connection`, and requiring `\`Call` to return `\`Close_connection` or `\`Continue`. `\`Ignore` was renamed because it was a poor name, since in fact it closed the connection. Added a `\`Continue` option, whic allows one to keep the connection open. Changed `\`Call` to return `\`Continue` or `\`Close_connection`, where the old `unit` return value meant `\`Close_connection`. - In `Versioned_typed_tcp`, enabled the use of "credentials" in the "Hello" message. Propagate credentials to the user code when it arrives on the wire. ## async_kernel - Optimized `Monitor.try_with ~run:\`Now f` to return a determined deferred if `f ()` returns a determined deferred. Previously, `Monitor.try_with ~run:\`Now f` always introduced a `Deferred.map`, which made it impossible to do some optimizations that bypass the scheduler overhead. - Added an `ASYNC_CONFIG` field that causes the program to dump core if Async jobs are delayed too much. The new field is `dump_core_on_job_delay`. - Switched `Async_kernel` from using `Core.Sys` to `Pervasives.Sys` eliminating one of the dependencies on `Core`. ## async_unix - Changed `Writer.transfer write pipe` to close `pipe` when the `writer`, is closed. Previously, `Writer.transfer` did not close the pipe when the underlying writer is closed. This was strange because: 1. Callers would have to consistently check for the writer being closed and close the `Pipe.Reader`t= themselves 2. The analogous function `Pipe.transfer` closes the reader on similar circumstances. The absence of the close was noticed as a bug in `Rpc`, which assumed that `Writer.transfer` did the close. - Fixed a bug in `Scheduler.yield` that caused it to pause for 50ms if there is no other pending work and no I/O. - Exposed type equivalence between `Unix.Passwd.t` and `Core.Std.Unix.Passwd.t`. - Changed `Writer.write_bin_prot` to use the new `Bigstring.write_bin_prot`. ## bignum - Added `Bignum.Bigint` module, with arbitrary-precision integers based on `Zarith`, which is significantly faster than the `Num.Big_int` library. ## bin_prot - In `Write`, improved some OCaml macros to name values and avoid calling C functions multiple times. ## core - Removed vestigial code supporting OCaml 4.00. - Added `Command` support for flags that are passed one or more times. Added `Command.Spec.one_or_more` and `Command.Spec.non_empty_sequence` to deal with the cases where you expect a flag or anonymous argument (respectively) to be passed one or (optionally) more times. This is common enough and distinct from the case where you want the argument passed zero or more times that it seems like we should canonize it in the library. - In `Lock_file`, made stale lock detection more robust. Made `Lock_file.create foo` succeed if `foo` is absent and `foo.nfs_lock` file is present and stale. Previously, it would fail. - Removed `Syslog.syslog`'s `add_stderr` argument; use the `PERROR` option instead. - Fixed `unix_stubs.c` compilation on NetBSD Closes #45 - Added `Filename` operators `/^` and `/@`, and `of_parts`, like the same functions for Catalog paths. - Changed `Iobuf` functions that advance the iobuf to not also return a redundant number of bytes processed. This avoids a small allocation (in the case of the `int option` functions) and normalizes the result (so the same information isn't returned two ways). Actually, it doesn't yet avoid the allocation in the implementation, as the corresponding `Bigstring` functions must still return the number of bytes processed, and currently do so as an option. We hope to eventually change that. In the future I expect we will change `unit` to some `error` variant to also avoid the exception construction for `EWOULDBLOCK/EAGAIN`. We can even make Unix syscalls `noalloc` if we're careful. - In `Unix` module, added unit tests for `Cidr.does_match`. ## core_bench - fixed legacy format string ## core_extended - Added `Float_ref` module, which is like `float ref` but faster for sets due to bypassing the write barrier. Benchmark results on Sandy Bridge: | [float\_ref.ml:] float ref set | 2\_886.94ns | 8.00w | | | [float\_ref.ml:] Float\_ref.set | 355.76ns | 6.00w | | | [float\_ref.ml:] float ref get | 415.52ns | 6.00w | | | [float\_ref.ml:] Float_ref.get | 416.19ns | 6.00w | | - Added `Bin_io_utils.Wrapped.t`, which defines an `'a t with bin_io` that supports size-prefixed serialization and deserialization. `Wrapped` has two useful submodules, `Opaque` and `Ignored`, for efficient handling of size-prefixed bin-io values in cases where serialization can be bypassed. See the comments in the module for more details. ## core_kernel - Removed vestigial code supporting OCaml 4.00. - Used `{Hashable,Comparable}.S_binable` in `Day_of_week` and `Month`. - Improved the performance of `Set_once.set`. - Added `Type_equal.Lift3` functor. - Replaced occurrences of `Obj.magic 0` with `Obj.magic None`. With the former the compiler might think the destination type is always an integer and instruct the GC to ignore references to such values. The latter doesn't have this problem as options are not always integers. - Made `String_id.of_string` faster. - Added `Bigstring` functions for reading and writing the size-prefixed bin-io format. - `bin_prot_size_header_length` - `write_bin_prot` - `read_bin_prot` - `read_bin_prot_verbose_errors` - Added `{Info,Error}.to_string_mach` which produces a single-line sexp from an `Error.t`. - Added `{Info,Error}.createf`, for creation from a format string. - Added new `Perms` module with phantom types for managing access control. This module supersedes the `read_only`, `read_write`, and `immutable` phantom types, which are now deprecated, and will be removed in the future. This module uses a different approach using sets of polymorphic variants as capabilities, and contravariant subtyping to express dropping capabilities. This approach fixes a bug with the current phantom types used for `Ref.Permissioned` in which `immutable` types aren't guaranteed to be immutable: ```ocaml let r = Ref.Permissioned.create 0 let r_immutable = (r : (int, immutable) Ref.Permissioned.t) let () = assert (Ref.Permissioned.get r_immutable = 0) let () = Ref.Permissioned.set r 1 let () = assert (Ref.Permissioned.get r_immutable = 1) ``` The bug stems from the fact that the phantom-type parameter is covariant, which allows OCaml's relaxed value restriction to kick in, which allows one to create a polymorphic value, which can then be viewed as both immutable and read write. Here's a small standalone example to demonstrate: ```ocaml module F (M : sig type +'z t val create : int -> _ t val get : _ t -> int val set : read_write t -> int -> unit end) : sig val t : _ M.t end = struct let t = M.create 0 let t_immutable = (t : immutable M.t) let () = assert (M.get t_immutable = 0); M.set t 1; assert (M.get t_immutable = 1); ;; end ``` The new approach fixes the problem by making the phantom-type parameter contravariant, and using polymorphic variants as capabilities to represent what operations are allowed. Contravariance allows one to drop capabilities, but not add them. - Added `Int.Hex` module, which has hexadecimal sexp/string conversions. - Added `Gc.major_plus_minor_words`, for performance reasons. ## custom_printf - Fixed uses of `printf=-style`format strings that have unspecified behavior in OCaml 4.02 and will become errors. - Support substitution in format string (lost with 4.02 compatibility) ## jenga - Don't show noisy `glob..changed` messages except with `-show-glob-changed` flag. - Support shared build rules via `${jenga}/share`. - Detect cycle in dep scheme instead of hanging. - Made standalone actions atomic, just like actions associated with target files. Running actions and recording the result in the persistent `.jenga.db` should be performed atomically for standalone actions, as it is for actions which are associated with target files ## ocaml_plugin - Changed to not use `rm -r` when it is expected to remove one file. ## rpc_parallel Initial import. ## sexplib - Replaced occurrences of `Obj.magic 0` with `Obj.magic None`. With the former the compiler might think the destination type is always an integer and instruct the GC to ignore references to such values. The latter doesn't have this problem as options are not always integers. ## textutils - Added `Ascii_table.Table_char`, to expose the special table border characters. ## type_conv - Updated ast matching for 4.02 -- Jeremie Dimino, for the Core team