[Caml-list] Great Programming Language Shootout Revived

[Caml-list] Great Programming Language Shootout Revived

[Brian Hurt]

Thought I'd point this article out for the people who've stopped reading 
slashdot:
http://developers.slashdot.org/article.pl?sid=04/06/16/1622211&mode=nested&tid=126&tid=156&tid=185&tid=90

The original GPLS was the best advertising Ocaml ever had.  It's why I 
learned Ocaml- I discovered the original GPLS and started noticing Ocaml 
ranking among the top scorers for fastest execution, least memory used, 
and fewest lines of code, and said to myself "I've got to check that 
language out- obviously there's something going on there."

What's starting to happen, now that the project has started up again, is 
advocates/supporters of other languages have started to submit improved 
versions of the code for their languages.  For example, I notice that 
Ocaml has dropped from it's #1 place in least lines of code to #2, with 
Ruby taking the lead.  

I don't think we need to make a big thing out of this (hey, #2 across the 
board is pretty damned good- we're beating both Java and C++ across the 
board, in fact the only other language that comes close to Ocaml's 
performance is, unsurprisingly, version of SML- MLton and SML/NJ).  But if 
you have some spare time, and want to help out Ocaml's marketing, wander 
over.

-- 
"Usenet is like a herd of performing elephants with diarrhea -- massive,
difficult to redirect, awe-inspiring, entertaining, and a source of
mind-boggling amounts of excrement when you least expect it."
                                - Gene Spafford 
Brian

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[Yaron Minsky]

On Thu, 17 Jun 2004 13:05:12 -0500 (CDT), Brian Hurt <bhurt@spnz.org> wrote:

> What's starting to happen, now that the project has started up again, is
> advocates/supporters of other languages have started to submit improved
> versions of the code for their languages.  For example, I notice that
> Ocaml has dropped from it's #1 place in least lines of code to #2, with
> Ruby taking the lead.

For what it's worth, the ocaml line-counts in the shootout are
particularly bogus.  There are lots of examples where the ocaml code
does things like putting entire for loops (or even nested for loops!)
on a single line.  Here's an example:

let _ =
  let n = if Array.length Sys.argv > 1 then int_of_string Sys.argv.(1) else 1 in
  let lix = n - 1 and x = Array.make n 0 and y = Array.make n 0 in
  for i = 0 to lix do x.(i) <- i + 1 done;
  for k = 0 to 999 do for i = lix downto 0 do y.(i) <- x.(i) + y.(i) done done;
  Printf.printf "%d %d\n" y.(0) y.(lix)

The code is perfectly reasonable, but it would be a hell of a lot more
readable and idiomatic if it was on 16 lines instead of 6.  The other
languages in the shootout appear to engage in less of this tomfoolery.
 Personally, I think the ocaml entries should be fixed to have more
lines of code.  As it is, the examples make ocaml look much more
obscure that it is.

y

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[Sebastien Ferre]

Yaron Minsky wrote:
> On Thu, 17 Jun 2004 13:05:12 -0500 (CDT), Brian Hurt <bhurt@spnz.org> wrote:
> 
> 
>>What's starting to happen, now that the project has started up again, is
>>advocates/supporters of other languages have started to submit improved
>>versions of the code for their languages.  For example, I notice that
>>Ocaml has dropped from it's #1 place in least lines of code to #2, with
>>Ruby taking the lead.
> 
> 
> For what it's worth, the ocaml line-counts in the shootout are
> particularly bogus.  There are lots of examples where the ocaml code
> does things like putting entire for loops (or even nested for loops!)
> on a single line.  Here's an example:

What about counting characters instead (counting several spaces in
a row as only one character) ? As most languages just don't mind
where you cut the lines, this seems to me a more fair measure.
Also, this would give complete freedom on the indentation style
without resulting in any disadvantage for the language rank.

Just a suggestion :)

Sébastien


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[Brian Hurt]

On Fri, 18 Jun 2004, Sebastien Ferre wrote:

> What about counting characters instead (counting several spaces in
> a row as only one character) ? As most languages just don't mind
> where you cut the lines, this seems to me a more fair measure.
> Also, this would give complete freedom on the indentation style
> without resulting in any disadvantage for the language rank.
> 
> Just a suggestion :)
> 

Actually, the idea I like is to pick a standard compression algorithm- say 
gzip, or bzip2- and compress the files.  This actually gives you a decent 
approximation of the amount of information the soure code contains- and 
entire sidesteps a lot of issues.  In addition to newlines and indenting, 
it also minimizes the effect of variable name length.  For example, the 
average name length for ocaml programs is probably a lot shorter than the 
average name length for java programs.

-- 
"Usenet is like a herd of performing elephants with diarrhea -- massive,
difficult to redirect, awe-inspiring, entertaining, and a source of
mind-boggling amounts of excrement when you least expect it."
                                - Gene Spafford 
Brian

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[Eray Ozkural]

On Friday 18 June 2004 18:45, Brian Hurt wrote:
> On Fri, 18 Jun 2004, Sebastien Ferre wrote:
> > What about counting characters instead (counting several spaces in
> > a row as only one character) ? As most languages just don't mind
> > where you cut the lines, this seems to me a more fair measure.
> > Also, this would give complete freedom on the indentation style
> > without resulting in any disadvantage for the language rank.
> >
> > Just a suggestion :)
>
> Actually, the idea I like is to pick a standard compression algorithm- say
> gzip, or bzip2- and compress the files.  This actually gives you a decent
> approximation of the amount of information the soure code contains- and
> entire sidesteps a lot of issues.  In addition to newlines and indenting,
> it also minimizes the effect of variable name length.  For example, the
> average name length for ocaml programs is probably a lot shorter than the
> average name length for java programs.

A note. This is an approximation of kolmogorov complexity which is regarded as 
a universal measure of information content. In recent research, it was 
observed that statistical compressors (PPM) do a little better than bzip2 in 
terms of discriminatory power (ie. information distance).

Best Regards,

-- 
Eray Ozkural (exa) <erayo@cs.bilkent.edu.tr>
Comp. Sci. Dept., Bilkent University, Ankara  KDE Project: http://www.kde.org
http://www.cs.bilkent.edu.tr/~erayo  Malfunction: http://malfunct.iuma.com
GPG public key fingerprint: 360C 852F 88B0 A745 F31B  EA0F 7C07 AE16 874D 539C

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] Great Programming Language Shootout Revived

[Brandon J. Van Every]

Brian Hurt wrote:
>
> The original GPLS was the best advertising Ocaml ever had.

Indeed, the Shootout + the victories in the ICFP are what caused me to
become interested in the language as a successor to C++ for 3D graphics
and AI problems.  Not that I've realized that vision yet.


Cheers,                     www.indiegamedesign.com
Brandon Van Every           Seattle, WA

"The pioneer is the one with the arrows in his back."
                          - anonymous entrepreneur

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.693 / Virus Database: 454 - Release Date: 5/31/2004

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[Ville-Pertti Keinonen]

On Jun 17, 2004, at 9:05 PM, Brian Hurt wrote:

> board is pretty damned good- we're beating both Java and C++ across the
> board, in fact the only other language that comes close to Ocaml's
> performance is, unsurprisingly, version of SML- MLton and SML/NJ).  
> But if

I looked at it a bit earlier (when it was posted on LtU), and MLton and 
SML/NJ were both ahead of OCaml in the overall scorecard for CPU.  
There was probably a bug in computing the totals, as the raw scores 
don't seem to have changed.

Anyhow, the shootout seems, like most benchmarks, to be misleading and 
arbitrary.  The language features compared are not really equivalent.  
E.g. C, C++ and Ada should be approximately the same in performance for 
code that doesn't compare their libraries or exception models.  Despite 
doing things like disabling array index checks, it seems some of the 
Ada benchmarks don't even use types equivalent to the C versions.

OCaml does have good performance characteristics - decent overall code 
generation, fast exceptions, fast memory allocation and a simple, 
lightweight standard library - but most of the benchmarks in the 
shootout are simply bogus for many of the languages.  They can be 
useful if you compare the performance of specific languages in specific 
benchmarks, as long as you look at the code as well to see what it is 
that's really being compared.

A more interesting way to compare programming languages might be to see 
what programming techniques are possible (and efficient) in different 
languages and how well suited they are for different tasks.  As you've 
noted, no one language is good at everything.  One reason I like OCaml 
is that it makes a reasonable range of techniques efficient, and it 
doesn't have DSLish characteristics.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[skaller]

On Fri, 2004-06-18 at 17:56, Ville-Pertti Keinonen wrote:
>  One reason I like OCaml 
> is that it makes a reasonable range of techniques efficient, and it 
> doesn't have DSLish characteristics.

I don't entirely 'agree' with the last point.

Ocaml not only has 'DSL' characteristics, it is easily
the best DSL around for a wide range of tasks: 
more precisely, Ocaml is such a good DSL it doesn't 
need any built-in DSLs, it uses a DSL generator instead.

Of course I'm mainly talking about camlp4, but one
cannot leave out its strong ability to create
parsers and interpreters and translators by other
means as another DSL integration technique.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[Ville-Pertti Keinonen]

On Jun 18, 2004, at 11:59 AM, skaller wrote:

> Ocaml not only has 'DSL' characteristics, it is easily
> the best DSL around for a wide range of tasks:
> more precisely, Ocaml is such a good DSL it doesn't
> need any built-in DSLs, it uses a DSL generator instead.

What I think of as DSL characteristics are language features targeted 
at specific tasks which aren't generally useful outside that area, yet 
are integrated in the language (e.g. file and text processing in Perl).

> Of course I'm mainly talking about camlp4, but one
> cannot leave out its strong ability to create
> parsers and interpreters and translators by other
> means as another DSL integration technique.

I was ignoring camlp4...and while OCaml is well-suited for creating 
parsers, interpreters and translators, can you think of any specific 
non-general language feature (apart from camlp4) facilitating this?  I 
think things like variant types and pattern matching are extremely 
general language features.

Of course it's difficult to define what a "general" language feature 
is, but I'm curious as to what OCaml features someone might consider 
domain-specific.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Implementing DSLs in OCaml/CamlP4 (was: Re: [Caml-list] Great Programming Language Shootout Revived)

[Richard Jones]

Are there any decent tutorials on using OCaml/Camlp4 as a
Domain-Specific Language (DSL)?  I was intrigued by
http://www.venge.net/graydon/talks/mkc/html/index.html but it was fair
to say that I didn't really understand the finer points of what he was
talking about.

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
If I have not seen as far as others, it is because I have been
standing in the footprints of giants.  -- from Usenet

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: Implementing DSLs in OCaml/CamlP4 (was: Re: [Caml-list] Great Programming Language Shootout Revived)

[Walid Taha]

Hi Rich,

I like that talk, and especially that discussion on safety.  MetaOCaml
provides safer support essentially the same approach.  Good starting 
points would be:

	http://www.cs.rice.edu/~taha/publications/journal/dspg04b.pdf
and	http://www.cs.rice.edu/~taha/publications/journal/dspg04a.pdf

Walid.

On Fri, 18 Jun 2004, Richard Jones wrote:

|Are there any decent tutorials on using OCaml/Camlp4 as a
|Domain-Specific Language (DSL)?  I was intrigued by
|http://www.venge.net/graydon/talks/mkc/html/index.html but it was fair
|to say that I didn't really understand the finer points of what he was
|talking about.
|
|Rich.
|
|

-- 


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[Brian Hurt]

On Fri, 18 Jun 2004, Ville-Pertti Keinonen wrote:

> Anyhow, the shootout seems, like most benchmarks, to be misleading and 
> arbitrary.  The language features compared are not really equivalent.  
> E.g. C, C++ and Ada should be approximately the same in performance for 
> code that doesn't compare their libraries or exception models.  Despite 
> doing things like disabling array index checks, it seems some of the 
> Ada benchmarks don't even use types equivalent to the C versions.

I agree.  Even worse, the majority exercise only small amounts of the 
language.  The tests array accesses, exception handling, method calls, 
nested loops, object instantiations, and string concatenation all 
explicitly test specific operations- with no regard for how often those 
options are actually implemented.  Take, for example, exception handling.  
The C version of this code uses setjmp()/longjmp().  Two functions which 
are rarely actually used in C code- the standard language pattern is to 
return an error code instead.  And other tests- notably ackerman's 
function and fibonacci numbers, are mainly dependent upon a similiarly 
small set of features (mainly speed of function calls).

"Artificial" doesn't begin to cover it.

As a side note, there is a reason why C++ is slower than C, even if you're 
compiling the exact same code.  C++ functions need to be able to handle 
exceptions and unroll the stack, even if the function itself doesn't throw 
or catch exceptions.  Even if unrolling the stack is a no-op, the function 
generally needs to be able to both be handed off from another function's 
stack unrolling, and hand off to another function's stack unrolling.  To 
remove this cost, the compiler needs to do a full program analysis, and 
basically detect "wait- you could compile this code with a C compiler, so 
I will compile this code with a C compiler!"  While this may be doable in 
theory, it's not going to happen in the real world.

Which raises another point.  Many languages, and C++ and Ocaml both
qualify, have "preferred" styles of programming, even if they admit other
styles.  For example, Ocaml "prefers" functional programming, it's the
default style, even as it allows imperitive and OO programming constructs.  
C++ "prefers" a template/OO style of coding, even as it admits C as
(basically) a proper subset.

If I were to sit down and start writting an application in Ocaml, 99% or 
more of it would be in a functional style.  How fast Ocaml is in 
imperitive or OO would not likely be all that important to the overall 
performance of my application.  Likewise, were I to write an application 
in C++, it's use templates, classes, virtual methods, etc.  If I wanted to 
write the application in C, I'd write it in C, not C++ and not Ocaml.  
Even if they involve a performance hit, they have other advantages (easier 
to write/debug code in, easier to use more advanced data structures, 
etc.).  This then raises the question of how much performance am I 
sacrificing for this increased power at the high level?

Plus there's the whole efficiency in the small vr.s efficiency in the 
large issue I mentioned earlier.  The GPLS measures almost entirely 
efficiency in the small.  Which isn't necessarily bad, just that it 
doesn't necessarily map to larger projects.

-- 
"Usenet is like a herd of performing elephants with diarrhea -- massive,
difficult to redirect, awe-inspiring, entertaining, and a source of
mind-boggling amounts of excrement when you least expect it."
                                - Gene Spafford 
Brian

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[David Brown]

On Fri, Jun 18, 2004 at 10:38:45AM -0500, Brian Hurt wrote:

> As a side note, there is a reason why C++ is slower than C, even if you're 
> compiling the exact same code.  C++ functions need to be able to handle 
> exceptions and unroll the stack, even if the function itself doesn't throw 
> or catch exceptions.  Even if unrolling the stack is a no-op, the function 
> generally needs to be able to both be handed off from another function's 
> stack unrolling, and hand off to another function's stack unrolling.

That depends on how exceptions are implemented.  There is a cost to
exceptions that can be paid at different times:

  - At each exception catch (try).
  - At each exception occurrence (raise).
  - At every call.

I've not actually seen the third case implemented.  For some test cases,
the actual function code generated by gcc and g++ are identical.  g++
contains an extra descriptor to facilitate stack unwinding.  It appears
that g++ pays most of the exception cost on raise (which is probably
best for C++ where exceptions are truly, exceptional).

So, aside from cache differences due to the layout of the structures,
the C-type code should execute at the same speed with C++ and C.

Most gnat (Ada) targets also implement these "zero cost" exceptions,
where the raise pays the cost, rather than catch.

Dave

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[Nicolas FRANCOIS]

Le Fri, 18 Jun 2004 10:56:38 +0300 Ville-Pertti Keinonen <will@exomi.com>
a écrit :

> 
> On Jun 17, 2004, at 9:05 PM, Brian Hurt wrote:
> 
> > board is pretty damned good- we're beating both Java and C++ across
> > the board, in fact the only other language that comes close to Ocaml's
> > performance is, unsurprisingly, version of SML- MLton and SML/NJ).  
> > But if
> 
> I looked at it a bit earlier (when it was posted on LtU), and MLton and 
> SML/NJ were both ahead of OCaml in the overall scorecard for CPU.  
> There was probably a bug in computing the totals, as the raw scores 
> don't seem to have changed.

They are ahead of the bytecode, not the native code.
 
> Anyhow, the shootout seems, like most benchmarks, to be misleading and 
> arbitrary.  The language features compared are not really equivalent.  
> E.g. C, C++ and Ada should be approximately the same in performance for 
> code that doesn't compare their libraries or exception models.  Despite 
> doing things like disabling array index checks, it seems some of the 
> Ada benchmarks don't even use types equivalent to the C versions.

Read the disclaimers :

"Disclaimer No. 1: I'm just a beginner in many of these languages, so if
you can help me improve any of the solutions, please drop me an email.
Thanks.

Disclaimer No. 2: These pages are provided for novelty purposes only. Any
other use voids the manufacturer's warranty. Do not mix with alchohol.
Some contents may consist of recycled materials. Contents packaged by
weight. Some settling of volume may occur.

Disclaimer No. 3: ditto.

Disclaimer No. 4: Please read the pages on Methodology, the FAQ, and
Conclusions before you flame."

Don't misunderstand te purpose of the site : _compilers_ are compared, not
languages. The confusion comes from the fact that most languages have only
one (or several, but on different platforms) implementation. That's for
the interpretation of the results.

As for the benchmarks provided, they don't pretend to be the best, and I
think that if you know a better way to do something, just contact the
author, who will certainly be glad to correct and improve its tests.

It's just a tool, and CERTAINLY not an impartial judgement, or a way to
choose a language. It just shows facts, you use them the way you want.

> OCaml does have good performance characteristics - decent overall code 
> generation, fast exceptions, fast memory allocation and a simple, 
> lightweight standard library - but most of the benchmarks in the 
> shootout are simply bogus for many of the languages.  They can be 
> useful if you compare the performance of specific languages in specific 
> benchmarks, as long as you look at the code as well to see what it is 
> that's really being compared.

Again, it's not a promotion site for OCaml. So don't expect it to present
Caml better than any other language.

And don't be too sad : second place just after a huge work like gcc is not
so bad :-) And remember : on certain circumpstances, gcc is RIDICULOUS
compared to the native INTEL compiler (on running on I86 architectures, of
course). COnsidering all this, I would say Pierre, Xavier and the others'
work is OUTSTANDING !
 
> A more interesting way to compare programming languages might be to see 
> what programming techniques are possible (and efficient) in different 
> languages and how well suited they are for different tasks.  As you've 
> noted, no one language is good at everything.  One reason I like OCaml 
> is that it makes a reasonable range of techniques efficient, and it 
> doesn't have DSLish characteristics.

So why don't you start your own site, or BETTER : contribute to the work
allready started ?

\bye

-- 

Nicolas FRANCOIS
http://nicolas.francois.free.fr

We are the Micro$oft.
Resistance is futile.
You will be assimilated.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Benchmark suggestion (Was: Programming Language Shootout)

[Wolfgang Müller]

Well, I knew OCaml before the shootout :-) , but I loved the shootout
results a lot.

OCaml helped me once in a project I had to code on a small slow portable
computer, where c++ was too slow to compile, JAVA needed too much RAM, and
Python/Perl Scripting was out of the question for execution speed reasons.

We then recently did some benchmarking at our place comparing JAVA against
c++ for an indexing application (inverted files). I also did some initial
tests with OCaml, however, as it was out of the question to convince my
peers to use OCaml I abandoned :-(

What was emphasized by the results of that benchmark is that you can
always do fancy optimisations, but what matters in projects that have to
be finished in time is the relation between programmer time and (execution
time || functionality).

JAVA took in our precise scenario about twice the time of c++, which we
deemed in our context fast enough, as we can save developer time when
using JAVA (e.g. faster debug-build-execute). In other contexts this might
be different.

In general you might approximate programmer time via number of lines or
the suggested compressed number of bytes (in absence of a better measure),
and the execution time you can measure, as well as the functionality.

What I would dream of, would be two classes of shootout-like benchmarks:

1. Gurus in every language solve given standard problems, like e.g.
parsing of web pages, indexing text, etc.etc. (suitable problems would
need to be found).

Here we would learn how fast and/or short someone really good can make it,
some upper bound of the performance reachable.

2. Graduate students with the usual experience in functional and
imperative/OO programming solve the problem given ample documentation of
the language and three days of time (like the ICFP contest?).

This would give an idea if one reaches reasonably quickly useful results
when switching to a given language. In my experience OCaml is also very
good at that.

What do you think about these ideas?

Cheers,
Wolfgang


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Benchmark suggestion (Was: Programming Language Shootout)

[Ville-Pertti Keinonen]

On Jun 19, 2004, at 12:04 PM, Wolfgang Müller wrote:

> 1. Gurus in every language solve given standard problems, like e.g.
> parsing of web pages, indexing text, etc.etc. (suitable problems would
> need to be found).

This sounds a bit like Lutz Prechelt's study (see 
http://page.mi.fu-berlin.de/~prechelt/Biblio/jccpprt_computer2000.pdf), 
except that there was only one problem, the implementers were 
self-selected and not necessarily gurus.

Unsurprisingly, scripting languages took the least time to write and 
had the lowest LOC.  OCaml wasn't part of the study, but the 
methodology was interesting.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] Benchmark suggestion

[Brandon J. Van Every]

Wolfgang Müller wrote:
>
> 2. Graduate students with the usual experience in functional and
> imperative/OO programming solve the problem given ample
> documentation of
> the language and three days of time (like the ICFP contest?).
>
> This would give an idea if one reaches reasonably quickly
> useful results
> when switching to a given language. In my experience OCaml is
> also very good at that.
>
> What do you think about these ideas?

I think one has to define the motivation for the benchmark.  Is it
language evangelism, i.e. making people aware of OCaml?  or of FP
languages generally?  If language evangelism is the point, then one
should look at *all* the factors that go into picking new languages.
The devil's in the details.

For instance, the Shootout led me to OCaml, but further inquiry is
leading me towards Standard ML because it has better int / 32-bit float
/ C interfacing.  These efficiency concerns are rather important to me,
they affect the most basic ways that I personally program.  It's way
easier for me to learn to blow off OO than to try to add such things to
OCaml.  I'm still going to organize a Seattle SIG, but I'm going to
re-Christen it as an "ML" SIG.  I'm reasonably convinced that ML is the
right language family for my problems, it's just down to which ML is
best for my specific needs.

Actually I think comparative benchmarking for some goal other than
evangelism is kinda pointless.  If you want your own app to go faster at
something, you write a benchmark for your own app.  You only look at
app-app or snippet-snippet comparisons when you're trying to decide
which option to pick.

Otherwise why are you bothering?  You are wasting your time.  I do mean
*time*, with my OpenGL device driver background I'm very conscious of
how much production time benchmarking consumes.  Even when you have all
the tests already written and you're just running them.  It takes time
to run and analyze the results of a benchmark.  I've learned not to sit
around running benchmarks all day, as that's not coding.  It's better to
develop instincts about what optimizations are worth it, and sanity
check those instincts with benchmarks a few times in a week.

I'd say the same for picking languages.  I have a lot of instincts about
language evangelism now, having looked at so many of them in the past
year.


Cheers,                         www.indiegamedesign.com
Brandon Van Every               Seattle, WA

20% of the world is real.
80% is gobbledygook we make up inside our own heads.

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.708 / Virus Database: 464 - Release Date: 6/18/2004

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] Benchmark suggestion

[Brian Hurt]

On Sat, 19 Jun 2004, Brandon J. Van Every wrote:

> I'd say the same for picking languages.  I have a lot of instincts about
> language evangelism now, having looked at so many of them in the past
> year.

The problem is that most people can't differentiate between their
instincts and their prejudices.  For example, most people's instinct is
that C++ should be as fast as C.  But the one measurement I've seen says
otherwise.  Show me another measurement, and I'll admit the issue is open
to debate.  But until you do, my response to all the logic and reasoning
in the world will still be "and yet it moves."  One measurement- no matter
how flawed and questionable- is worth a million opinions.  Otherwise we're
just stumbling around Plato's cave.

-- 
"Usenet is like a herd of performing elephants with diarrhea -- massive,
difficult to redirect, awe-inspiring, entertaining, and a source of
mind-boggling amounts of excrement when you least expect it."
                                - Gene Spafford 
Brian

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] Benchmark suggestion

[Wolfgang Müller]

> I think one has to define the motivation for the benchmark.  Is it
> language evangelism, i.e. making people aware of OCaml?  or of FP
> languages generally?  If language evangelism is the point, then one
> should look at *all* the factors that go into picking new languages. The
> devil's in the details.

No, for me personally language evangelism is not the point. Language
evaluation is the point. If it comes out that the best language for
solving my problems is actually JAVA, I swallow and say "yes, thanks for
this information" (this is the current state of affairs, by the way). If
it turns out that Python/SWIG is what would do best what I need and I can
motivate people to do that, I'd do Python/SWIG.

> Actually I think comparative benchmarking for some goal other than
> evangelism is kinda pointless.

How do you define "evangelism"? The word means "spreading the good
message". So this means benchmarking is good for finding out if a language
is useful? Yes, I think this is what a benchmark is about, right?

And yes, ease of mixed-language programming could be a useful parameter.

> Otherwise why are you bothering?  You are wasting your time.  I do mean
> *time*, with my OpenGL device driver background I'm very conscious of
> how much production time benchmarking consumes.

I certainly do not have your OpenGL device driver background :-) and I
spare you with my definition of what "real work" is :-D . In any case, I
would be interested in looking at a set of benchmarks that tells me if
what I want to achieve can be done easily and efficiently in a given
language or not, without me having to write too many comparative
benchmarks myself. One per week is clearly too many for me. Ballpark
figures are useful already.

Cheers,
Wolfgang


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

[Caml-list] Evangelism

[Brandon J. Van Every]

What's this post of mine about?  Really, it's about Evangelism.  How do
people make decisions on what language to use?  Which languages survive
in industry and which ones die?  Herein I discuss general principles.
Extrapolation to OCaml is left as an exercise to the reader.

Wolfgang Müller wrote:
>
> No, for me personally language evangelism is not the point. Language
> evaluation is the point. If it comes out that the best language for
> solving my problems is actually JAVA, I swallow and say "yes,
> thanks for this information"

The problem space is too complex to believe you're swallowing anything
on 'objective merit'.  What you're really doing is iterating your
subjective criteria over and over again.  You probably have 10 variables
you care about.  It takes a lot of time, looking at a lot of options, to
identify which of those 10 variables aren't so important, which ones are
dealbreakers, which ones would subjectively improve your life if only
you had feature X, etc.

If you lay out all the permutations and/or combinations mathematically,
and assign some kind of fungible constant amount of time to each
iteration, i.e. "3 days," you will probably conclude that the decision
gets made when you run out of time.  To fully evaluate the problem
always takes more time than you have.  So at some point, you close your
eyes and make a leap.

Or else, someone more important than you are tells you what to do.

Companies like Microsoft thrive mainly on the limited amount of time
that most managers are willing to dedicate to these problems.

> Brandon Van Every wrote:
> >
> > Actually I think comparative benchmarking for some goal other than
> > evangelism is kinda pointless.
>
> How do you define "evangelism"?

I define it as "promoting the benefits of a technology in the face of
competition."  We have many programming languages we could choose to
use.  When we say that OCaml is better than other options for Problem X,
we are *evangelizing* OCaml.

> In any case, I
> would be interested in looking at a set of benchmarks that tells me if
> what I want to achieve can be done easily and efficiently in a given
> language or not, without me having to write too many comparative
> benchmarks myself. One per week is clearly too many for me. Ballpark
> figures are useful already.

My experience over the past year is:

- all compiled languages are within a similar ballpark of performance
- the performance divide is between compiled and interpreted
- small open source projects are useless for production coding
- only the large open source projects are any good
- if someone's language reeks of smallness, skip it
- if it doesn't have a VC++ build it's useless on Windows
- has someone solved *your problems* using the language?

My driving problems are:

- how do I make commercially viable games with less work?
- can I turn around and consult these skills to others?

I think it's important to define one's driving problems, and also
resolve conflicts between them.  You say you want language benchmarks.
Well, why?  What are they going to help you decide?  "Anything you might
want to achieve" is too vague.


Cheers,                         www.indiegamedesign.com
Brandon Van Every               Seattle, WA

"We live in a world of very bright people building
crappy software with total shit for tools and process."
                                - Ed Mckenzie
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.708 / Virus Database: 464 - Release Date: 6/18/2004

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[Ville-Pertti Keinonen]

On Jun 19, 2004, at 3:26 AM, Nicolas FRANCOIS (AKA El Bofo) wrote:

> Le Fri, 18 Jun 2004 10:56:38 +0300 Ville-Pertti Keinonen 
> <will@exomi.com>
> a écrit :

>> I looked at it a bit earlier (when it was posted on LtU), and MLton 
>> and
>> SML/NJ were both ahead of OCaml in the overall scorecard for CPU.
>> There was probably a bug in computing the totals, as the raw scores
>> don't seem to have changed.
>
> They are ahead of the bytecode, not the native code.

They were ahead of the native code, which was between C++ and SML/NJ.  
I even clicked the link and scratched my head over the raw scores, 
which seemed to indicate that it should've had a better overall score.

> Again, it's not a promotion site for OCaml. So don't expect it to 
> present
> Caml better than any other language.

Actually, since the original shootout author is a fan of OCaml, I 
suspect the tests have something of a pro-OCaml bias. ;-)

> So why don't you start your own site, or BETTER : contribute to the 
> work
> allready started ?

Every time I start writing a comparison of programming languages, it 
seems to require the length of a book...

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[Nicolas Janin]

Contributions are actually encouraged.

I recall that the site contains a wiki, so that anyone can add his own
comments/suggestions in order to improve the implementations.

Concerning the idea of zipping the source codes as a measure of code length,
I find it quite good, as the zipped code is less sensitive to code layout
than a pure line count and is a decent measure of the information conveyed.
In the opposite, just counting the number of characters is probably worse
than line count, as it would lead to extremely compact unreadable code for
languages like C or Perl (we all know  Obfuscated C and Perl contests).


_____________________________________________________________________
Envie de discuter en "live" avec vos amis ? Télécharger MSN Messenger
http://www.ifrance.com/_reloc/m la 1ère messagerie instantanée de France

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[Marcin 'Qrczak' Kowalczyk]

W liście z sob, 19-06-2004, godz. 13:56 +0200, Nicolas Janin napisał(a):

> Concerning the idea of zipping the source codes as a measure of code length,
> I find it quite good, as the zipped code is less sensitive to code layout
> than a pure line count and is a decent measure of the information conveyed.

I don't like this as a measure of the amount of code which must be
written and maintained, because copy&paste might actually come shorter
or equal to introducing appropriate named abstractions, while being much
worse design.

I think token count is the most objective measure from measures which
are reasonably easy to take automatically. If not token count, then
line count, and finally character count.

-- 
   __("<         Marcin Kowalczyk
   \__/       qrczak@knm.org.pl
    ^^     http://qrnik.knm.org.pl/~qrczak/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] Great Programming Language Shootout Revived

[Brandon J. Van Every]

Marcin 'Qrczak' Kowalczyk
>
> I think token count is the most objective measure from measures which
> are reasonably easy to take automatically. If not token count, then
> line count, and finally character count.

I don't suppose you guys would consider moving this discussion to
comp.benchmarks?  Not so much that it's off-topic on caml-list - I don't
care.  Rather, there's this whole community that deals with all these
"are tokens or LOCs better?" issues in exceeding detail, and you're only
repeating what's been done to death in their archives.  If you actually
want to design a 'better' language benchmarker, this would be your ideal
watering hole.

Personally I think the idea of 'token counts' is silly, just not as
silly as LOC.  How would you ever objectively define the semantic value
of a 'token' ?  What if my tokens are more dense with information than
your tokens?  Anyways, my question is rhetorical.  I suggest
comp.benchmarks.


Cheers,                         www.indiegamedesign.com
Brandon Van Every               Seattle, WA

20% of the world is real.
80% is gobbledygook we make up inside our own heads.

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.708 / Virus Database: 464 - Release Date: 6/18/2004

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] Great Programming Language Shootout Revived

[Brian Hurt]

On Sat, 19 Jun 2004, Brandon J. Van Every wrote:

> I don't suppose you guys would consider moving this discussion to
> comp.benchmarks?  

Or better yet, the shootout's own mailing list:
http://lists.alioth.debian.org/mailman/listinfo/shootout-list

I just posted something to that list, and I recommend everyone interested 
in continuing this discussion move it over there instead.

-- 
"Usenet is like a herd of performing elephants with diarrhea -- massive,
difficult to redirect, awe-inspiring, entertaining, and a source of
mind-boggling amounts of excrement when you least expect it."
                                - Gene Spafford 
Brian

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[Nicolas Janin]

Thinking about it a little more, the good thing with LOC count is, as I've
read somewhere else, in large softwares (say > 50,000 LOC), LOC count was
quite a fairly good measure of the size of a project in terms of spent
resources. In other words, according to surveys, the effort involved (as
measured in man-months) was fairly proportional to the LOC count, which is
why LOC count still prevails as a measurement of a software project.
However there has never been any survey measuring code size in terms of
zipped kilobytes unfortunately, although one might suspect the correlation
between this measurement and the effort involved would be stronger than with
LOC count.

Also, it is not obvious at all how this translates to small programs of <200
lines like in the shootout.


___[ Pub ]____________________________________________________________
Inscrivez-vous gratuitement sur Tandaime, Le site de rencontres !
http://rencontre.rencontres.com/index.php?origine=4

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

RE: [Caml-list] Great Programming Language Shootout Revived

[John Hughes]

As someone who used to write some code in APL, I was
always a fan of LOC as a measure of *something*, because
APL always wins. It used to be said that a line of APL
was like 40 lines of FORTRAN...but that because of that,
you should expect each to take the same amount of time
to read. 

The point of APL being compact was not that the 
programs were smaller, but that idioms were easier to
recognize/learn/use, and the structure of the language 
helped the programmer structure his/her thought. 

I actually believe that those are two of the advantages of
ML as well. 

--John 

> -----Original Message-----
> From: owner-caml-list@pauillac.inria.fr 
> [mailto:owner-caml-list@pauillac.inria.fr] On Behalf Of Nicolas
Janin
> Sent: Saturday, June 19, 2004 2:10 PM
> To: Nicolas FRANCOIS (AKA El Bofo); caml-list@inria.fr
> Subject: Re: [Caml-list] Great Programming Language Shootout Revived
> 
> Thinking about it a little more, the good thing with LOC 
> count is, as I've read somewhere else, in large softwares 
> (say > 50,000 LOC), LOC count was quite a fairly good measure 
> of the size of a project in terms of spent resources. In 
> other words, according to surveys, the effort involved (as 
> measured in man-months) was fairly proportional to the LOC 
> count, which is why LOC count still prevails as a measurement 
> of a software project.
> However there has never been any survey measuring code size 
> in terms of zipped kilobytes unfortunately, although one 
> might suspect the correlation between this measurement and 
> the effort involved would be stronger than with LOC count.
> 
> Also, it is not obvious at all how this translates to small 
> programs of <200 lines like in the shootout.
> 
> 
> ___[ Pub
]____________________________________________________________
> Inscrivez-vous gratuitement sur Tandaime, Le site de rencontres !
> http://rencontre.rencontres.com/index.php?origine=4
> 
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: 
> http://caml.inria.fr Bug reports: 
> http://caml.inria.fr/bin/caml-bugs FAQ: 
> http://caml.inria.fr/FAQ/ Beginner's list: 
> http://groups.yahoo.com/group/ocaml_beginners
> 

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Great Programming Language Shootout Revived

[Brian Hurt]

On Sat, 19 Jun 2004, Nicolas Janin wrote:

> Thinking about it a little more, the good thing with LOC count is, as I've
> read somewhere else, in large softwares (say > 50,000 LOC), LOC count was
> quite a fairly good measure of the size of a project in terms of spent
> resources. In other words, according to surveys, the effort involved (as
> measured in man-months) was fairly proportional to the LOC count, which is
> why LOC count still prevails as a measurement of a software project.
> However there has never been any survey measuring code size in terms of
> zipped kilobytes unfortunately, although one might suspect the correlation
> between this measurement and the effort involved would be stronger than with
> LOC count.

I beleive it was Fred Brooks who first opinioned that programmers produced
about the same number of working lines of code per month no matter what
the language was- and that therefor higher level languages (by which he
meant Fortran and PL/1) that did more work in fewer lines of code were
more productive than lower level languages (by which he meant assembly 
language).

-- 
"Usenet is like a herd of performing elephants with diarrhea -- massive,
difficult to redirect, awe-inspiring, entertaining, and a source of
mind-boggling amounts of excrement when you least expect it."
                                - Gene Spafford 
Brian

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners