Re: [Caml-list] The state of ML: multi-threaded, saved state and support platforms

[Jon Harrop <jon@ffconsultancy.com>]

On Thursday 11 June 2009 22:44:08 Lucas Dixon wrote:
> Richard Jones wrote:
> > OCaml has native threads with shared memory, _but_ OCaml threads
> > nevertheless cannot run simultaneously because there's a lock
> > protecting the garbage collector.  The standard solution is to use
> > fork-based parallelism or MPI (which will work better on new multicore
> > machines which are all NUMA nowadays, and in any case is simpler to
> > program).  See extensive discussions on this list about garbage
> > collectors with threads.
>
> Thanks, I just found an old post:
>
> http://caml.inria.fr/pub/ml-archives/caml-list/2002/11/64c14acb90cb14bedb2c
>acb73338fb15.en.html
>
> So PolyML is the only version of ML that supports SMP? (although it
> locks all threads on a major collection - it does support parallelism on
> shared-memory multiprocessors - so you really can max out 4 processor
> machines - but the speedup tails off only 5 times faster for 8 core
> machines)

Wow! Virtually none of the surviving open source standalone FPL 
implementations are able to leverage multicores effectively. There is an 
obscure Scheme implementation called Ypsilon that is also theoretically 
capable of leveraging multicores but it is apparently too slow to be useful 
as such.

A 2008 Jane St. Summer Project funded Emmanuel Chailloux's team to build a 
parallel GC (not state-of-the-art but vastly better than anything available 
from OCaml today). They do occasionally make noises about it but, AFAIK, 
nothing has been released.

> >> 3. Lastly, can you help me get a good picture of the state of support
> >> for ML systems:
> >>
> >> OCaml: is this still supported by INRIA? what is the level of support?
> >
> > What level of support are you looking for?
>
> I was wondering who pays for the documentation to be kept up to date,
> bug fixes, and implements new features. I saw that there is an OCaml
> Consortium: http://caml.inria.fr/consortium/index.en.html
> Is this INRIA or something else? Is this the source of funding that
> supports OCaml? or is there INRIA backing also? Does anyone know how
> many people are employed?

I don't know about the earlier questions but, IIRC, the money from the 
consortium funds one half-time developer.

> I'm interested in how to support functional programming environments. What
> level of financial support is needed?  

That depends entirely upon your goals. You might be able to develop FPL 
implementations from an academic position but it is insecure and restricted 
in terms of not being able to devote a lot of time to polishing practically 
important features because they do not constitute research.

On the other hand, you might copy Stephen Wolfram's business model with 
Mathematica: productize an FPL implementation with useful features, a decent 
stdlib and a next generation user interface. That earned him about $1bn and 
he can now cover his own support costs.

Personally, I would opt for the latter. However, I cannot see how to do that 
without expending a six figure sum developing a viable first version. Hence I 
am not gambling my own money on that (yet).

There is also the problem that going cross platform is hard, Windows opens you 
up to strong competition (like F#) and the non-Windows market has little 
money.

> > OCaml users will give you
> > support through the ocaml-beginners mailing list which is active and
> > most questions are answered within a few hours.  If you want "call
> > someone on the phone and drop everything" support then you'll have to
> > pay (a lot).
>
> Are there organisations that offer OCaml support over the phone? Or
> would that be consultancy rates for academics who specialise in OCaml?

There are several OCaml consultants including myself.

> >> F#: Miscrosoft
> >
> > Are you sure about this one?
>
> no idea :)
> I just saw that there is a MSDN F# Developer Center, and assume
> Microsoft field the responsibility to fix bugs etc. Any experience with
> this?

Microsoft have provided me with excellent F# support since we started using it 
commercially over 2 years ago.

> >> 4. I'm also interested in interoperability: what systems can OCaml
> >> easily interoperate with? What about the other systems?
> >
> > This question is very unclear.  OCaml offers lots of opportunities for
> > interoperating, eg. RPC mechanisms, web services, direct linking,
> > passing structures through data files (XML, sexprs etc), pipes,
> > databases.  What are you looking for?  What other systems?
>
> Sorry, I will try to clarify:
> I was taking for granted the ability to open sockets, have pipes, and
> read files. Web-services, links to databases, parsers, and anything else
> at that level are not what I meant.
>
> I meant things at the level of linking compiled code/modules.

OCaml can interoperate easily via file formats and benefits from some great 
parsing tools. There are also some RPC libraries of mixed quality, e.g. the 
most popular XML library for OCaml is far from standards compliant and most 
OCaml libraries do not support unicode. OCaml can interoperate with C but it 
requires stubs to be written and compiled separately. Consequently, most of 
OCaml's C bindings are buggy. Interoperability with other languages like C++ 
is prohibitively difficult, e.g. see OCaml's (lack of) Qt bindings. 
Interoperating with other high-level languages either introduces problems 
with GCs or is very inefficient because there is no CLR.

> F# website 
> says it has "a cross-compiling core shared with the popular OCaml
> language", so I presume that modules written in one can be used in the
> other.

Only if they are written in the (tiny) intersection of the two language. Also, 
F# is diverging from OCaml as it drops legacy features in preparation for its 
first full product release in 2010.

> This is the level of interoperability I was wondering about. Is 
> there similar things for libraries writtin in Haskell, or MLton, or
> MLKit, or something similar for OCaml with SML/NJ?

Are you asking if OCaml code can interoperate with libraries written in SML 
and Haskell, and vice-versa? If so, I believe it is not at all easy. I recall 
one person asking for advice about this and my advice was to use the loosest 
bindings possible (e.g. XMLRPC) because tight bindings are just too fragile 
in these kinds of languages.

> I know the HLVM project is hoping to provide some of these features. I
> was wondering what the current situation is.

HLVM is a hobby project of mine and the recession is currently forcing me to 
devote my time to profit-making activities. If my new book "F# for technical 
computing" sells as well as I hope, I'll get back to HLVM development.

-- 
Dr Jon Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/?e