Re: [Caml-list] Long-term storage of values

["Jake Donham" <jake.donham@skydeck.com>]

[-- Attachment #1: Type: text/plain, Size: 1943 bytes --]

On Fri, Feb 29, 2008 at 6:45 AM, Martin Jambon <martin.jambon@ens-lyon.org>
wrote:

> if we restrict the transition to a certain subset of operations, it can be
> possible to define a mapping using some Camlp4 tool such as Deriving
> (well, that's what I was told, assuming I interpreted correctly).
>
> For instance:
> - adding a record field: a default value is injected
> - removing a record field: just remove it
> - adding a variant: do nothing since it doesn't exist in the old data
> - changing a type arbitrarily (such as changing a type foo1 into foo2
>   everywhere): provide a map function that would override the default map
>   function for such nodes.
> - functional and abstract values: left as-is
>

We do this at Skydeck. We modified Deriving's Typeable module to expose the
structure of types, and we marshal values along with a version number (for
upgrades) and a type description from Typeable (so you get an error if you
change the type and forget to change the version number).

Our modified Typeable also has reflection functions so if you have a dynamic
(a value along with a Typeable.TypeRep) you can examine its parts
dynamically (implemented with Obj.magic). On top of that we have a generic
function for changing a value of one type into a value of another, as Martin
describes--we try to do the translation automatically as far as we can, and
provide a way to pass in mapping functions for particular types.

For the most part this works well. Compared to a SQL database it is very
nice to have the OCaml type system for data representation. The upgrade
mechanism is much safer than hand-coding SQL schema upgrades. On the other
hand, there are many things you get for free with a SQL db that you have to
do yourself: e.g. putting IDs on objects so you can refer to them
externally, easy hand-inspection of the data (it's annoying navigating big
structures in the top-level), to name a couple of small ones.

Jake

[-- Attachment #2: Type: text/html, Size: 2262 bytes --]