Hongmeng native intelligence will not let anyone fall behind–Fast Technology–Technology changes the future

“Darkness will make him appreciate light more; silence will make him love sound more.”

For most people, “seeing, hearing, and speaking” are common things. But there are still a group of people who can't watch their phones, watch videos, make calls, or send voice messages like us.

Even in the highly convenient Internet age, people with visual and hearing impairments have to face many challenges in their daily lives, travel and social interactions. Light and sound may be the most precious gifts for them, just like Helen Keller's famous work, “If you are in darkness and long for three days of light, you can only “if”.

Gaining three days of sight requires an “if”, but with the help of “black technology”, people with hearing and visual impairments can “see, hear and speak” with just a mobile phone.

During HDC2024, Huawei released HarmonyOS NEXT, a full-scenario intelligent operating system based on OpenHarmony. As the biggest upgrade since the birth of HarmonyOS, it not only achieves ecological unification, easy interconnection of multiple devices, and improved experience such as fluency and energy efficiency, but also integrates AI capabilities into the system for the first time, supporting image intelligence, call intelligence, document intelligence and other capabilities.

Xiaoyi has also evolved into Xiaoyi Intelligent Agent, which can perform more complex operations and gain better scene perception, intent recognition and user understanding capabilities with the support of end-cloud big model collaboration and intent framework.

The upgraded Xiaoyi intelligent body is one of the key “black technologies” to help people with hearing and visual impairments achieve barrier-free mobile phone experience. It is also the most special experience I unlocked at HDC2024.

In the “Black Super Tour”, after my eyes were covered and I fell into complete darkness, I was led by the staff into a simulated supermarket shopping scene, and worked with my partners to complete the purchase of designated products without seeing anything. Moreover, one of my three partners played the role of a hearing-impaired user, and the communication between us was done with the help of Xiaoyi AI, and my “eyes” were also a mobile phone with Xiaoyi AI installed.

Hongmeng native intelligence leaves no one behind

Here, my friend and I play the roles of two visually impaired users, and I would like to focus on my experience with the “black technology” of Xiaoyi's view of the world.

When your vision is deprived, the first thing you feel is that it's difficult to move – you will subconsciously slow down and shorten your stride, and your hands will unconsciously grope forward (even though I confirmed that there were staff to ensure that there would be no danger, this would still happen, so much so that the staff told us: Just go ahead, no problem).

Then you come to a completely unfamiliar space, and you will imagine the surrounding environment, wanting strongly to know where you are and what is around you – but you can't see anything at all, and you will feel uneasy.

When the staff handed the phone to me and told me that my task was to call my “hearing-impaired” partner, I habitually unlocked the screen, only to find that I couldn't see anything. But I long-pressed the power button to wake up Xiaoyi, said to call XX, and completed the task easily.

Because my companion has hearing impairment, during this call, Xiaoyi will convert what I say into text and present it to the other party, and the text sent by the other party will also be converted into voice and played on my phone. Here I know what I want to buy.

Hongmeng native intelligence leaves no one behind

Next, I will use my own “eyes” to find the task product. After holding another mobile phone and opening Xiaoyi to see the world, Xiaoyi will continue to broadcast the content captured by the mobile phone through AI image recognition. For example, he will tell me that there is a person in the picture (the staff member who plays the role of the cashier), a pair of hands touching a black object (my hand is in the picture), XX drink 300ml, etc.

In this way, I fumbled on the shelves, and with the guidance of my “hearing-impaired” partner, I finally found the products I needed, and under the guidance of Xiaoyi, I went to the checkout counter and completed the payment. It's not over yet, we still need to leave the “supermarket” with the help of Xiaoyi's world view.

The staff put up signs on the exit door that read “This door is blocked”, “Door locked”, and “Correct door” as reminders. Xiaoyi reported to the corresponding door and we successfully went out and ended this “black ultrasound journey”.

Hongmeng native intelligence leaves no one behind

Undoubtedly, Xiaoyi can provide a lot of help to users with visual and hearing impairments. As for the visually impaired user I simulated, Xiaoyi can guide me to understand the surrounding situation more accurately, especially some information that we cannot obtain by touching with our hands. After all, Braille, voice prompts, etc. are not ubiquitous in life.

At the same time, in some specific scenarios, it allows me to achieve my goals faster and independently without being able to see anything, not just shopping, but also browsing exhibitions, traveling, socializing, and so on.

Of course, at this stage these functions still have a lot of room for improvement. For example, the user needs to adapt to or try the distance between the phone and the identification subject. If combined with some voice prompts (“hold the phone farther away, hold the phone closer”), the efficiency will be higher.

Since it relies on the mobile phone camera, it still has high requirements for ambient light. In dark or low-light environments, that is, if it is at night, the experience of this function may be worse.

There is another key point. Although my vision was completely blocked during the experience, I have rich experience with similar scenes and mobile phone operating modes in my mind. Therefore, I can make full use of the impressions in my mind when moving and looking for objects. Combined with Xiaoyi's guidance in viewing the world, the time to complete the task is further shortened.

But for real users with visual impairments, they may face much more challenges than we do.

Hongmeng native intelligence leaves no one behind

It is worth mentioning that when developing these features, Huawei listened to the real feedback from users with such needs. We also saw some visually and hearing-impaired users who were invited to experience these features and give feedback on their experience. Being able to seriously accept the feedback from these users is also a key part of the iteration of these features.

Although it is not perfect, it has many surprises and has the potential to be more worth looking forward to and supporting in the future.

First of all, although I did not ask whether this function is completely based on local AI computing power, the improvement of the AI ​​capabilities on the terminal side allows image recognition to be performed locally, which will effectively reduce latency and can also be used in an offline environment, greatly ensuring the experience.

Judging from the on-site experience, the entire recognition process was very smooth. Basically, while I was moving the phone, Xiaoyi had already started reporting the content of the screen.

Hongmeng native intelligence leaves no one behind

Secondly, the end-side AI has an inherent advantage in protecting user data privacy, reducing worries. At the same time, if combined with the “end-cloud integration” solution, such as leveraging the technical advantages of Huawei Pangu Big Model and OpenHarmony, the consumer-oriented Hongmeng native intelligence will have more powerful capabilities, not only natural language interaction, but also more personalized and prepared intention understanding, as well as the interconnected experience brought by the unified ecosystem.

Hongmeng native intelligence leaves no one behind

Third, the terminal's hardware and software capabilities have been improved, making the image recognition more accurate and detailed. For example, in the experience, Xiaoyi Kanshijie was able to accurately report some small words on the edge of the sign placed on the door by the staff, and even punctuation marks.

Hongmeng native intelligence leaves no one behind

Fourth, with the help of the most popular terminal devices such as mobile phones, if the capabilities are transferred to mainstream models or even entry-level models in the future, it will definitely be good news for users with specific needs. After all, the cost they need to pay to “see, hear, and describe” the world will be much lower, or as close to what they can afford.

Final Thoughts

The “Journey in Black”, or Huawei's barrier-free communication exhibition area, is only a small part of HDC2024, but from this small part we can see the power of technology. Perhaps this is not a function that most users need or will use, but for tens of millions of users with visual, hearing, and language disabilities who are also in the era of the Internet of Things, these functional points can be called “must-haves.”

Some senior industry insiders also shared their views while watching the HDC keynote speech, saying that this is also the important significance of AI universal access.

Hongmeng native intelligence leaves no one behind

As He Gang, CEO of Huawei Terminal BG, mentioned: Hongmeng's native intelligence will not leave anyone behind.

Since its official release on August 9, 2019, the HarmonyOS operating system has lasted 1,778 days and gone through 4 generations. The number of HarmonyOS ecosystem devices has exceeded 900 million. 2.54 million HarmonyOS developers have been involved in HarmonyOS development, 4.35 million people have studied in HarmonyOS Academy, and the number of developer service calls is 82.7 billion times per month.

Hongmeng native intelligence leaves no one behind

Currently, Hongmeng native applications have entered the full sprint stage. More than 5,000 commonly used applications have started development, and more than 1,500 of them have been put on the shelves.

Hongmeng native intelligence leaves no one behind

In terms of pushing updates, HarmonyOS NEXT based on OpenHarmony launched beta upgrades for developers and pioneer users on June 21. The Beta version is expected to be pushed in August, and the official version will be pushed in the fourth quarter of this year. Next year, it will support more models. Let us wait and see.

Hongmeng native intelligence leaves no one behind