On the bar chart I included what I think are the most popular models right now (not objectively, based on my anecdotal observations, link to include/exclude what you want).
What do you think? Is it fair? There was, and to some digree still is, a bit of controversy about AA score indexes after they updated their benchmarks to V5.0. I am mainly interested in your thoughts about QFN in relation to other models.
Gemini 3.8 does not match its benchmarks.
It forgets system instructions, spends too much time thinking, and has this weird bug with reasoning token reprocessing that totally throws off the model.
Mistral Large should be higher up, it’s far more reliable than its benchmarks shows.
What is this bug you described? First time hearing about it… Mistral seems weirdly low, it’s a very big model, maybe they didn’t finetune it for benchmarks?
Gemini encrypts their thinking traces, so if you inject a thought in an early thinking trace it always gets added after user context on any later message in the conversation.
That means if the thought trace disagrees with a user message it goes with the original thought trace.
E.g imagine you start a conversation with the date, then come back the next day. The thinking trace will make the model always think it’s the same date, no matter how much you argue.
They were working on fixing this, so not sure if the new models today fixes that.
Interesting, maybe caused by design focused on overprotection against distillation of the thinking process, so “it’s a feature, not a bug” mentality. I don’t use gemini models so I can’t confirm by myself, thanks for pointing that out!
That is part of it.
The thinking trace is a base64 encoded proton if with an encrypted string inside