It doesn't exclude "idiomatic" versions, but the data is presented so as to make them useless. How do I compare "idiomatic" rust to "idiomatic" haskell? All you can get is "here's the fastest versions from each language". If you want to compare any other version you have to do it on an individual basis, one at a time. So there's absolutely no reason for people to bother adding "idiomatic" versions.
"Idiomatic" in whose opinion? "Idiomatic" is a slogan not a well-defined property.
The reason for people to bother adding "idiomatic" versions, is that other people really do work through looking at the source code "on an individual basis, one at a time."
This exact thread came up last year when we started talking about the shootout and haskell. And Its always the same people who bring up the shootout and defend the existing entries.
Please stop bringing up the shootout (esp when haskell is mentioned) and dragging everyone through this conversation again.
Benchmarks are great to debate but this one is causing more harm than good; let's talk about other benchmarks.
That's the point I'm making. People constantly dismiss the results they don't like as "unidiomatic" and embrace they ones they do as "idiomatic".
>is that other people really do work through looking at the source code "on an individual basis, one at a time."
Except that people don't bother adding them, so your reasoning for why isn't relevant. If you want to claim the shootout is useful for this, you need to allow for grouping benchmarks together to be presented against the "standard" ones. Let Bob and Sally submit "Bob and Sally version of Foo benchmarks" and let me compare them against Foo and against other languages.
The point of it is to show the performance differences, but that doesn't exclude a program like --
http://benchmarksgame.alioth.debian.org/u64q/program.php?tes...
-- or like --
http://benchmarksgame.alioth.debian.org/u64q/program.php?tes...
etc