> The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%
I think this is a misunderstanding of what that 83% probability means. You think you're asking Jev to assign a probability to a dice roll, and if so then obviously it should pick each answer 1/6 of the time.
However, what Jev is doing is assigning a probability to the answer it should choose. And it's choosing 1 with 83% probability, as the correct answer to give, when there's no additional knowledge available.
1 is a perfectly valid answer here. In the absence of other data there's no more (nor less) valid answer than that. and I assume that just like LLMs, the token prediction must assign higher probability to one outcome.
If you keep feeding the previous answers back in (if you can do that with Jev) I assume it would start giving other numbers because it's no longer operating from zero info. Maybe that would even reach ~1/6 after enough rounds, although maybe not.
TypeSafe defines the probabilities Jev returns as "calibrated probabilities". "Probability" here means the probability of the answer being correct. If the probability is 10%, the choice should be correct about one time in ten. So, when Jev returns 83% probability it should be correct about 83 times out of 100, but the choices were only correct about 19 times out of 100, and the true probability is 1/6.
"Higher probability should correspond to a greater chance that the answer is correct."
As someone who hasn’t used Jev, this seems reasonable in the sense that “probability that I’ve given the best answer” seems like a useful thing to provide.
Anyway, if it is “probability that I gave the best answer” why 83%? I mean, the question is a sort of a textbook-style probability question in the sense that everything is well defined. Shouldn’t it be basically 100%?
Really? You don't want to share those messages at the end of a long day where where you found yourself typing "STOP! WHAT THE HELL ARE YOU DOING THAT FOR? I CLEARLY TOLD YOU..."
There's also the Snapdragon X2 Elite Extreme, 3,438 points in single-core and 27,519 points in multi-core, which I think was the record when it was released a year ago.
I guess they'll release an X3 in six months. So it's interesting to see the race is now between Apple and Qualcomm, both on Arm, with Intel and AMD getting left behind, at least on this benchmark.
I moved from a cold climate to a tropical climate and let me assure you there's no human universal understanding of what's comfortably hot or cold, definitely not to the point of putting a percentage on it. I can walk around in a T-shirt and shorts at 17C (62f) while locals will be wrapping up in scarves and talking about how freezing it is. Meanwhile they'll be fairly comfortable at 35C (95f), at least if the humidity isn't high, and I'll be stuck in with the AC blasting.
So in other words Fahrenheit is a western centric measurement (or at least, temperature climate centric), while Celsius is universal.
The subjective experience doesn’t really matter because many places in the US will experience a 100 degree F range over an entire year. So it’s not the “comfortable range” but the actual range you’ll get in many parts of the US (whether you like it or not).
Right, but "room temperature" is pretty universal everywhere because we are going with the "average human" rather than "outliers exist and maybe feel differently"
No it's not. Room temperature in tropical countries is often 26-30C.
I work in the food industry and this whole "room temperature" thing is a major bugbear for me actually because when so much food dates markings are designed around a room temperature of ~22C, it's a big health and safety issue for people (and shops, warehouses, restaurants) who don't know that and happily store the goods up to the date marking at 30C. Deciding that room temperature ~=22C and basing worldwide food safety on that is absolutely Western centric.
Obviously people here are not stupid and don't store meat etc. past when it's gone bad. But I've often bought e.g. oils from suppliers and it's clear that it's on the edge of rancidity even though it's got a year+ to go on the date because it's been stored at "room temperature".
The US has the widest range of temperatures of any country. Locations can vary by >50°C in a 24 hour period. Extreme temperatures are familiar to Americans and their society is well-adapted to it.
Calling Fahrenheit "western centric" is nonsensical. Every other country has a narrower temperature range that fits within the US temperature range.
Instructing to use the `gh` CLI, not ideal for those of use using BBC at work
They also have instructions about how to format certain output, which conflicts with the instructions we have in repo. I only discovered yesterday because we were wondering why the agent kept picking certain tools.
Thomas Edison did not attend the electrocution of Topsy the elephant, but his film company did record it, and he did directly fund electrocution of dogs years earlier (1888 is the year mentioned here).
I think this is a misunderstanding of what that 83% probability means. You think you're asking Jev to assign a probability to a dice roll, and if so then obviously it should pick each answer 1/6 of the time.
However, what Jev is doing is assigning a probability to the answer it should choose. And it's choosing 1 with 83% probability, as the correct answer to give, when there's no additional knowledge available.
1 is a perfectly valid answer here. In the absence of other data there's no more (nor less) valid answer than that. and I assume that just like LLMs, the token prediction must assign higher probability to one outcome.
If you keep feeding the previous answers back in (if you can do that with Jev) I assume it would start giving other numbers because it's no longer operating from zero info. Maybe that would even reach ~1/6 after enough rounds, although maybe not.
reply