New GPT-4-Powered Humanoid Robot: No Programming Needed, Learns from Zero Samples, and Adjusts Behavior with Verbal Feedback

What can be done by letting GPT-4 control a humanoid robot without any prior programming or training?

the answer is:So curious!

No, the robot is required to perform “crazy popcorn in a dark cinema, but suddenly realizes that the food it eats is from a stranger next door”.

Under the control of GPT-4, the robot’s smile froze, and it shook its head and leaned back in “embarrassment”:

But even though he was embarrassed, he didn’t forget to grab another handful and stuff it into his mouth? ? ?

Or it’s like asking a robot to “play the guitar.”

GPT-4 thought for a while, then started to control the robot to move its fingers, and then shook its head wildly, which seemed to have a bit of a rock and roll feel.

But if you look closely, you can see that when you shake your head, your fingers don’t move at all…

If you want to say that this is the arrogant Mr. Street Stall, there seems to be nothing wrong (manual dog head).

To summarize a series of actions——

Compared with Boston Dynamics humanoid robots, every move is carefully controlled by human programs. This robot directly controlled by GPT-4 has a ferocious expression and weird movements, but all its behaviors meet the requirements of the prompt.

After this series of videos of GPT-4 controlling robots were posted online, many netizens called out “the uncanny valley effect”:

It even scared a robotics expert with 20 years of experience:

These movements gave me the creeps. Do you see for yourself how beautiful this is?

Some netizens joked: “It looks just like me on stage.”

However, some netizens believe that it is incredible that humanoid robots can be controlled through GPT-4.

It turns out that this isThe first humanoid robot powered by GPT-4Research from the University of Tokyo and Japanese Alternative Machine Company.

Relying on this research, users do not need to program the robot in advance, only language input, that is,Chat with GPT-4 for a whileallowing the robot to complete actions according to instructions.

Large model + new attempt at robots

Without programming or training, using GPT-4 as its brain, what other frowning but reasonable actions can this humanoid robot Alter3 make?

Why not give an instruction and let Alter3 pretend to be a !

It does know how to get into a scene in one second, opening its mouth wide and stretching its hands forward.

But I don’t know why, the slightly parted lips and empty eyes make it feel more like a zombie that even Lin Zhengying would call an expert:

If you ask it to take a selfie, Alter3 will be able to take a slapstick photo on the spot.

He just didn’t know that it was because he was so ugly in the original camera that he didn’t enjoy his expression. Instead, he closed his eyes in pain:

Let’s listen to rock music again and start walking.

You say that it nods frequently because it is following the beat, which is definitely correct; but if you say that it stands in front of you respectfully and says “Ah, yes, yes, yes, yes”, it seems to be very appropriate (manual dog head):

Among all the video demos released, the behavior of “drinking tea” is the least weird and weird, and it even seems to be playing on me:

When I’m at work and I’m desperate, drinking tea is so life-threatening. Let me tell you, you don’t open your mouth until it’s brought to your mouth. It doesn’t matter if you don’t drink the tea.

As a humanoid robot, Alter3’s human behavior is a bit funny, so… why not try something else?

For example, interpret a snake that is inspired by music and sways:

Did you see it? It’s not that flexible, but it is really twisting its trunk as much as it can. It can be said to be a crazy version of a snake.gif.

Looking at it this way, it is possible to directly integrate a humanoid robot with GPT-4, but the aesthetics are really not enough…

In fact, looking back, scientists and researchers have been busy this year to combine large models with robots.

However, it is common practice to domore trainingand then try to transfer the capabilities and knowledge of the image language large model to the robotics field.

Many projects, including Microsoft’s ChatGPT for Robotics, Google’s PaLm-E, RT-1, RT-2, VoxPoser, BoboCat, etc., all follow this route.

Among them, the sensational Google RT (Robot Transformer) series is extremely effective, but it took Google 17 months to train it and collect the data of 13 robots. 130,000 pieces of robot-specific data ——It is difficult for ordinary teams to have this kind of funds and energy.

In the middle of the year, Li Feifei’s team’s embodied intelligence achievements have made further progress. By combining LLM (Large Language Model) + VLM (Visual Language Model), the robot’s ability to interact with the environment has been further improved.

Under this idea, the robot can complete the task without additional data and training.

However, Li Feifei’s team provided the hardware in the demo, and the main body is just a robotic arm. In the research we introduce today, the experimental subjects are GPT-4, the strongest in the large model industry, and Alter3 as the “body”.

Whether it is GPT-4 developed by OpenAI or the humanoid robot Alter3 jointly developed by the University of Tokyo and Japan’s “Father of Robotics” Hiroshi Kuroishi, they are all existing research results.

The real purpose of this research is to explore how to use large models like GPT-4 to control humanoid robots to complete various actions without programming, in order to verify GPT-4’s ability to generate actions and reduce the complexity of human-computer interaction. Spend.

Relying on this series of results, Alter3 can complete the various complex actions seen above (let’s put aside the completion and viewing ability for now).

Another thing is that when the researchers integrated Alter3 and GPT-4, they found that even if the same command is given to Alter3, the action fed back by Alter3 will not be the same every time.

After some analysis, they believe that this is related to the characteristics of the large language model itself, that is, the same input may correspond to different outputs, but it does not mean that GPT-4 cannot control humanoid robots well.

For example, if you ask the robot to “eat”, it may make different actions of eating with chopsticks and eating with a knife and fork twice.

So, then again, how does GPT-4 know how to control Alter3 as soon as it receives sentence input?

The key lies in the 2-step prompt word

In fact, before being connected to GPT-4, Alter3 had a brain (AI neural network) and various sensors on its body.

Previously, Alter3’s behavior mainly relied on its built-in CPG (Central Pattern Generator) to analyze the data from the sensor, and then drive the 43 pneumatic devices in the body in a certain order to complete the corresponding actions.

The entire process often requires human intervention, patching, and improvements.

But! Everything is different now, and the research team says the integration of GPT-4 has given them “relief.”

It is now possible to command Alter3 with verbal instructions. The technologies behind them are mainly the following:

CoT (thought chain), and zero-shot (Zero-shot learning).

Relying on these two technologies, the control of Alter3 no longer relies entirely on the hardware itself, but can use GPT-4 to directly convert natural language into actions that the robot can understand and execute.

Most importantly, the entire processNo need to explicitly program any one body part.

Okay, now let’s talk about how to integrate GPT-4 and Alter3.

It can be roughly divided into two steps:

firstfirst use prompt to describe what behavior or action you want Alter3 to be able to complete, such as “Let’s take a selfie” and “Raise your arms higher when taking a selfie.”

GPT-4 receives input and generates a series of thought steps that detail what needs to be done in order to complete this action.

This process, which the research team calls part of CoT, breaks down a complex task into a sequence of simpler thought steps.

Thenthe researchers used another prompt to convert the decomposed detailed steps into action instructions that Alter3 can understand.

A simple understanding is to convert people’s instruction descriptions into Python codes, which can be directly used to control the specific motion parameters of each body part of Alter3.

With the converted code, Alter3 can wink if he wants to, and curl his lips if he wants to.

.

The research team regards this second step as part of CoT, because what it completes is “transforming an abstract description into a concrete operation.”

The team stated that CoT allows GPT-4 to effectively control Alter3 and command it to perform various complex actions without additional training or fine-tuning.

A few words, in addition to the above two prompts to control the robot, the research team also completed some other research.

For example, break it apart and look at Alter3’s behavior in the conversation, mainly focusing on itsConversation trackandSemantic time evolution.

For conversation trajectories, the research team used a method called UMAP (Uniform Manifold Approximation and Projection). The team embedded the conversation content into a two-dimensional space, making it easier to observe the development process of this simplified version of the conversation.

They found that when the order of dialogue is fixed, the conversation trajectory will show a circular pattern, that is, the same topics will be repeated over and over again.

And when the dialogue order is random, the dialogue content is more divergent or creative.

An interesting point is that research has found that GPT-4 tends to say “goodbye” repeatedly after chatting for a long time. Without some human intervention from you, it will become obsessed with trying to say goodbye to you.

In the process of doing semantic time evolution analysis, the team observed changes in chat content over time.

They found that some keywords in the early stages of the conversation, such as “art” or “learning”, would be forgotten by GPT-4 during the conversation, and replaced by “culture”, “humanity”, “inhibition” and so on. word.

This shows that the content of the conversation is gradually evolving.

Of course, if you start saying “goodbye” to GPT-4, it will almost exclusively want to say byebye to you~ (doge)

from the University of Tokyo

This popular research on the Internet comes from the University of Tokyo and the Japanese Alternative Machine Company.

One work Takahide Yoshidafrom the Department of General Systems Science, University of Tokyo.

Two other authorsAtsushi Shōmori(Atsushi Masumori) andTakashi Ikegami(Takashi Ikegami), both at the University of Tokyo and affiliated with Alternative Machine Company.

Finally, I have to mention the protagonist of this research Alter3the researchers behind it are also from the University of Tokyo, and were jointly created by Takashi Ikegami, an AI research scholar at the University of Tokyo, and Hiroshi Kuroishi, the “father of robots” in Japan.

Alter3 was born in 2020 and is the third generation of the same series of robots.

It is understood that both iterations of the Alter series were completed during opera singing. The first appearance of the 3rd generation was conducting the orchestra and participating in other live performances at the New National Theater in Tokyo.

At that time it featured enhanced sensors and improved singing expressiveness and vocal systems.

And the CPG in the body that can ultimately drive 43 pneumatic devices.

How sensitive is CPG’s analysis of data? That is, if the temperature in the room where Alter3 is staying drops sharply, Alter3 will shiver, indicating that it is cold.

This may also provide some foundation for it to be able to make expressions and complete actions vividly after being connected to GPT-4 as a brain.

One More Thing

Speaking of the latest news about humanoid robots, we must mention Lao MajiaTesla Optimus Prime The latest news:

Just now,MuskA video of Optimus was suddenly posted on Twitter, indicating that the Optimus second-generation robot (Gen 2) will be released this month.

A little “minor” improvement is that the walking speed of the second-generation Optimus has increased by 30%.

Balance and body control also improve.

Can’t wait to wait!

Reference links:

  • (1)https://tnoinkwms.github.io/ALTER-LLM/

  • (2)https://arxiv.org/abs/2312.06571

  • (3)https://twitter.com/elonmusk/status/1734763060244386074