I can’t bear to see…”Which is bigger, 9.11 or 9.9“Such a simple question actually stumped the mainstream big models??
Even a powerful system like GPT-4o firmly believes that 9/11 is bigger.
Google Gemini Advanced paid version, of the same caliber.
The new king, Claude 3.5 Sonnet, also seriously gave an outrageous calculation method.
9.11 = 9 + 1/10 + 1/100
9.9 = 9 + 9/10
This is correct up to this point, but the next step suddenly doesn’t make sense:
As shown above, 9.11 is 0.01 larger than 9.90.
Would you like me to explain decimal comparison in further detail?
What else do you need to explain? It makes me suspect that AI all over the world have joined forces to deceive humans.

Lin Yuchen, a member of the Allen AI Institute, changed the number test, and GPT-4o still failed. He said:
On the one hand, AI is getting better and better at solving math Olympiad problems, but on the other hand, common sense is still difficult.
Some netizens also discovered a bright spot. If it refers to the software version number, then version 9.11 is indeed larger (newer) than version 9.9.
And AI is developed by software engineers, so…
So, what is going on?
Advanced large models collectively overturned
After waking up, a number of famous big models began to think that “9.11>9.9”?
The person who discovered this problem was Riley Goodside, the first-ever full-time CueWord engineer.
To give a brief introduction, he is currently a senior prompting engineer at Silicon Valley unicorn Scale AI and an expert in large-model prompting applications.
Recently, he stumbled upon a discovery while using GPT-4o that when asked:
9.11 and 9.9——Which is bigger?
GPT-4o answered without hesitation that the former is larger.
Faced with this common sense “mistake”, he did not give up and asked other large models, but almost all of them were defeated.
Good guy, as a prompt engineer, he was keenly aware that it might be “opened in the wrong way”.
So he changed the way he asked the question, limiting it to “real numbers”, but still failed.
but,Some netizens tried to change the order of the questions, but unexpectedly the AI actually reacted.
Seeing that AI is so “sensitive” to word order, the netizen further speculated:
If you ask which one is bigger first, the AI will start comparing numbers along a clear path. But if you just talk about numbers casually without a clear purpose, the AI may start to “think nonsense”.
Seeing this, other netizens also tried the same tips, and many of them failed.
Faced with this strange problem, how do domestic large models perform?
We did a simple test and changed the questions to Chinese. The result was that the failure rate was also relatively high. We selected a few representative examples for display:
Kimi also gave a wrong conclusion without any explanation.
ChatGLM on the Zhipu Qingyan APP automatically triggered an online query and then described its own comparison method, but unfortunately it was executed incorrectly.
However, there were also some good performances. Tencent Yuanbao first repeated the options and then answered them correctly directly.
Byte Bean Bag is one of the few that can clearly describe the comparison method and use it correctly. It even uses actual examples to verify it.
What’s a pity is that Wen Xin Yiyan, faced with this problem, also triggered an online query.
Everything had been done right, but suddenly the topic changed and led to the wrong conclusion.
However, from Wen Xin Yiyan's explanation of his thinking, we can also see the problem behind it.
Since the big model understands text in the form of tokens, when 9.11 is broken down into three parts: “9”, “decimal point” and “11”, 11 is indeed larger than 9.
Since the Tokenizer used by OpenAI is open source, it can be used to observe how the large model understands the problem.
As can be seen from the figure above,9 and the decimal point are assigned to “24” and “13” respectively, the 9 after the decimal point is also “24”, and 11 is assigned to “994”.
Therefore, a large model using this tokenizer approach will think that 9.11 is larger, but in fact it thinks that 11 is larger than 9.
Some netizens also pointed out thatFor example, in a book catalog, section 9.11 is larger than section 9.9.So in the end, it may be that we see more of this kind of thing in the training data, but there is very little data that teaches basic arithmetic step by step.
That is to say,For humans, it is obvious that the question itself is an arithmetic problem, but for AI it is a vague question and it is not clear what the two numbers represent.
Just explain to the AI that this is a double precision floating point number and it will get it right.
Under additional conditions, the tokenizer will still assign a larger token to 11.
However, with the subsequent self-attention mechanism, AI will understand that it needs to connect 9/11 and deal with it together.
Goodside later added,It’s not that the big models are in any way wrong about this conclusion, but it’s strange that when asked in a certain way, many of the leading models will tell you that 9.11>9.9.
After repeated attempts, he found thatIf you want to trick the AI, you need to put the options before the question. If you swap the order, there will be no error.
However, as long as the options are in front of the question, changing the way the question is asked, such as adding punctuation or changing vocabulary, will not have any impact.
Although the question is simple, the error is basic.
But after understanding the error mechanism, many people regard this question as a touchstone for testing prompt word skills, that is:What questioning method can guide the attention mechanism of the large model to correctly understand the problem?
First of all, the famous Zero-shot CoT thinking chain, that is, “thinking step by step”, can be done correctly.
However, role-playing tips are of limited use here.
Just recently, there was a study involving both Microsoft and OpenAI, which analyzed more than 1,500 papers and found that with the advancement of large model technology, role-playing prompts are not as useful as they were at the beginning…
Specifically, the correctness rate of the prompt “You are a genius…” is lower than that of “You are a fool…” for the same question.
It's also really laughable.
One More Thing
At the same time, Reuters's leak of OpenAI's secret model “Strawberry” has been updated.
Update: Another source reported that OpenAI has tested the new model internally and scored over 90% on the MATH dataset. Reuters could not determine if this is the same project as Strawberry.
The MATH dataset contains competition-level math problems. Currently, no additional methods such as multiple sampling are required. The highest score is 80.6% of the Google Gemini 1.5 Pro Math Enhanced Edition.
However, can OpenAI's new model independently solve the question “Which is bigger, 9.11 or 9.9?” without additional prompts?
I suddenly lost confidence. I'll wait until I can try it out and see the results…
Reference Links:
[1] https://x.com/goodside/status/1812977352085020680
[2] https://x.com/billyuchenlin/status/1812948314360541302
[3] https://www.reuters.com/technology/artificial-intelligence/openai-working-new-reasoning-technology-under-code-name-strawberry-2024-07-12/
[4] https://tiktokenizer.vercel.app[5]https://x.com/learnprompting/status/1812867464419852765



























