Which is bigger, 9/11 or 9/9? AI big models are collectively losing their minds and almost all overturned – Fast Technology – Technology changes the future

I can’t bear to see…”Which is bigger, 9.11 or 9.9“Such a simple question actually stumped the mainstream big models??

Even a powerful system like GPT-4o firmly believes that 9/11 is bigger.

Google Gemini Advanced paid version, of the same caliber.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

The new king, Claude 3.5 Sonnet, also seriously gave an outrageous calculation method.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

9.11 = 9 + 1/10 + 1/100

9.9 = 9 + 9/10

This is correct up to this point, but the next step suddenly doesn’t make sense:

As shown above, 9.11 is 0.01 larger than 9.90.

Would you like me to explain decimal comparison in further detail?

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

What else do you need to explain? It makes me suspect that AI all over the world have joined forces to deceive humans.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Lin Yuchen, a member of the Allen AI Institute, changed the number test, and GPT-4o still failed. He said:

On the one hand, AI is getting better and better at solving math Olympiad problems, but on the other hand, common sense is still difficult.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Some netizens also discovered a bright spot. If it refers to the software version number, then version 9.11 is indeed larger (newer) than version 9.9.

And AI is developed by software engineers, so…

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

So, what is going on?

Advanced large models collectively overturned

After waking up, a number of famous big models began to think that “9.11>9.9”?

The person who discovered this problem was Riley Goodside, the first-ever full-time CueWord engineer.

To give a brief introduction, he is currently a senior prompting engineer at Silicon Valley unicorn Scale AI and an expert in large-model prompting applications.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Recently, he stumbled upon a discovery while using GPT-4o that when asked:

9.11 and 9.9——Which is bigger?

GPT-4o answered without hesitation that the former is larger.

Faced with this common sense “mistake”, he did not give up and asked other large models, but almost all of them were defeated.

Good guy, as a prompt engineer, he was keenly aware that it might be “opened in the wrong way”.

So he changed the way he asked the question, limiting it to “real numbers”, but still failed.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

but,Some netizens tried to change the order of the questions, but unexpectedly the AI ​​actually reacted.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Seeing that AI is so “sensitive” to word order, the netizen further speculated:

If you ask which one is bigger first, the AI ​​will start comparing numbers along a clear path. But if you just talk about numbers casually without a clear purpose, the AI ​​may start to “think nonsense”.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Seeing this, other netizens also tried the same tips, and many of them failed.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Faced with this strange problem, how do domestic large models perform?

We did a simple test and changed the questions to Chinese. The result was that the failure rate was also relatively high. We selected a few representative examples for display:

Kimi also gave a wrong conclusion without any explanation.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

ChatGLM on the Zhipu Qingyan APP automatically triggered an online query and then described its own comparison method, but unfortunately it was executed incorrectly.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

However, there were also some good performances. Tencent Yuanbao first repeated the options and then answered them correctly directly.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Byte Bean Bag is one of the few that can clearly describe the comparison method and use it correctly. It even uses actual examples to verify it.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

What’s a pity is that Wen Xin Yiyan, faced with this problem, also triggered an online query.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Everything had been done right, but suddenly the topic changed and led to the wrong conclusion.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

However, from Wen Xin Yiyan's explanation of his thinking, we can also see the problem behind it.

Since the big model understands text in the form of tokens, when 9.11 is broken down into three parts: “9”, “decimal point” and “11”, 11 is indeed larger than 9.

Since the Tokenizer used by OpenAI is open source, it can be used to observe how the large model understands the problem.

Which is bigger, 9/11 or 9/9? AI big models collectively lost their minds and almost all failed

As can be seen from the figure above,9 and the decimal point are assigned to “24” and “13” respectively, the 9 after the decimal point is also “24”, and 11 is assigned to “994”.

Therefore, a large model using this tokenizer approach will think that 9.11 is larger, but in fact it thinks that 11 is larger than 9.

Some netizens also pointed out thatFor example, in a book catalog, section 9.11 is larger than section 9.9.So in the end, it may be that we see more of this kind of thing in the training data, but there is very little data that teaches basic arithmetic step by step.

That is to say,For humans, it is obvious that the question itself is an arithmetic problem, but for AI it is a vague question and it is not clear what the two numbers represent.

Just explain to the AI ​​that this is a double precision floating point number and it will get it right.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Under additional conditions, the tokenizer will still assign a larger token to 11.

However, with the subsequent self-attention mechanism, AI will understand that it needs to connect 9/11 and deal with it together.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Goodside later added,It’s not that the big models are in any way wrong about this conclusion, but it’s strange that when asked in a certain way, many of the leading models will tell you that 9.11>9.9.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

After repeated attempts, he found thatIf you want to trick the AI, you need to put the options before the question. If you swap the order, there will be no error.

However, as long as the options are in front of the question, changing the way the question is asked, such as adding punctuation or changing vocabulary, will not have any impact.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Although the question is simple, the error is basic.

But after understanding the error mechanism, many people regard this question as a touchstone for testing prompt word skills, that is:What questioning method can guide the attention mechanism of the large model to correctly understand the problem?

First of all, the famous Zero-shot CoT thinking chain, that is, “thinking step by step”, can be done correctly.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

However, role-playing tips are of limited use here.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Just recently, there was a study involving both Microsoft and OpenAI, which analyzed more than 1,500 papers and found that with the advancement of large model technology, role-playing prompts are not as useful as they were at the beginning…

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Specifically, the correctness rate of the prompt “You are a genius…” is lower than that of “You are a fool…” for the same question.

It's also really laughable.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

One More Thing

At the same time, Reuters's leak of OpenAI's secret model “Strawberry” has been updated.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

Update: Another source reported that OpenAI has tested the new model internally and scored over 90% on the MATH dataset. Reuters could not determine if this is the same project as Strawberry.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

The MATH dataset contains competition-level math problems. Currently, no additional methods such as multiple sampling are required. The highest score is 80.6% of the Google Gemini 1.5 Pro Math Enhanced Edition.

Which is bigger, 9/11 or 9/9? AI models are losing their minds and almost all fail

However, can OpenAI's new model independently solve the question “Which is bigger, 9.11 or 9.9?” without additional prompts?

I suddenly lost confidence. I'll wait until I can try it out and see the results…

Reference Links:

[1] https://x.com/goodside/status/1812977352085020680

[2] https://x.com/billyuchenlin/status/1812948314360541302

[3] https://www.reuters.com/technology/artificial-intelligence/openai-working-new-reasoning-technology-under-code-name-strawberry-2024-07-12/

[4] https://tiktokenizer.vercel.app[5]https://x.com/learnprompting/status/1812867464419852765