In the "How can I get a list containing..." thread in the Forums, TerrenceBrannon wrote:
I'm sure many of you are suspecting that I am doing a naive implementation the Sieve of Eratosthenes benchmark at the Language Shootout.
I think Roger's solution is optimized and hence not acceptable.
What is the point of comparing languages if you're not allowed to use the advantages intrinsic to each language? If you have to write all programs in C, spoken with a lisp, why not just use C in the first place?
If the idea is to compare the implementations of various "constructs fundamental to a programming language", such as iteration and recursion, significant(ly more) thought needs to be put into the definition of "constructs fundamental to a programming language".
If the idea is to compare languages independent the programmer, a better metric would be how a programmer intimate with the language but naive of the algorithm would code the solution. A programming language, like J, might present a whole new way of thinking about problems, which would lead to an entirely different algorithm.
The assumption in this shootout appears to be that "all languages are pretty much the same" (so why compare them?). That is, everyone's going to write the same answer, but in a different language. That seems as silly to me as trying to compare "take the sum of the first N positive integers" in a language which only supports additition:
+/ 1 + i. N
vs one that also supports multiplication and division:
(N * 1+N) % 2
Knowing about, being familiar with, and having experience using, multiplication and division might cause one to frame the question differently than someone who's stuck with merely addition. And those tools may lead to fresh insights which are independent of the tools that provided them. Put another way: Eratosthenes didn't know any programming languages, and I know several. Yet, I am not confident that, if he had not invented it, I would have.
But, of course, there's no way to compare "language as a tool of thought". So we're stuck with benchmarks that force us to write every program in C.
If we could somehow enforce honesty, perhaps a superior metric would be "how long for the a programmer familiar with language L to produce a correct, working, comprehensive solution to the problem?". Then, partitioning the times-to-solution into orders of magnitude, prizes could be awarded to the fastest interpreters within each stratum.
I think the fundamental problem with this benchmark, like all benchmarks, are that its goals are nebulous. It wants to determine "the best language", but it doesn't define what "best" means (never mind "language"!).
If the best language is the one that the average human can use to instruct a computer to take an action, without training, why have programmers? Or, equivalently, if that's the definition of best, then does it matter if the language makes it progressively more difficult to instruct a computer to take progressively more complex actions?
If the best language is the one that a manager chooses to maximize productivity on a project, and it turns out that performance is much more dependent on the individual programmer than the individual language *, perhaps that manager's time would be best spent choosing a programmer rather than a language.
Or perhaps the benchmark isn't designed to help anyone choose a language at all. I certainly doubt anyone will choose to learn a new language based on its outcome (who cares how long language L takes to calculate the naive implementation of the Sieve of Eratosthenes?). I think the benchmark tries to optimize the ability of the constituency of a language to retroactively support their previously-made decision to use that language. And, of course, crow about it.
-- DanBron 2007-05-21 16:33:37
Of course, if that's true, this benchmark specifically seeks to abstract away from it. After all, what's the point of comparing languages if the language doesn't really matter?