"Setting up for the machine": building a two-way alignment system for the value of humans and robots

 　　"Our research results indicate that artificial intelligence systems have the ability to learn human value functions in communication and align current human value goals in real time. It is an important step for machines to realize the paradigm of 'small data, big tasks'. A step closer to true autonomous intelligence and general artificial intelligence," said Zheng Zilong, a researcher at the Beijing Institute of General Artificial Intelligence.

　　Recently, a team led by Professor Zhu Songchun from the Beijing Institute of General Artificial Intelligence and the Institute of Artificial Intelligence of Peking University has constructed a human-machine collaborative two-way value-aligned computing framework through a game of "human-machine collaborative exploration", which proves that in this framework Intelligent systems and humans can trust each other and work together to achieve goals like humans.

　　This achievement demonstrates a new model of human-machine collaboration, which will help to design better artificial intelligence systems and be applied to human-machine teamwork scenarios in the future.

　　Today, artificial intelligence is gradually beginning to penetrate people's lives. You may have noticed that in everyday life, your intelligent voice assistant often makes mistakes, and even after you correct it, the same mistakes still occur. There are also intelligent sweeping robots, which can only act according to pre-set logic, and will not change the path immediately after hearing your command.

　　The current artificial intelligence cannot align with human value in real time, which is a huge obstacle for artificial intelligence assistants to enter thousands of households.

　　This research work by Zhu Songchun's team demonstrates the potential to solve these problems, and is a step toward the realization of general artificial intelligence, which may help millions of people to better cooperate with artificial intelligence in the future.

　　The research paper was published under the title "Human-Machine Real-time Bidirectional Value Alignment". The co-first authors of this research work are Yuan Luyao, Gao Xiaofeng and Zheng Zilong.

　　In the past 10 years, artificial intelligence technology represented by deep learning has made great progress. However, this model based on big data training is a kind of passive intelligence, which can only complete specific tasks mechanically according to the codes programmed by humans in advance. It lacks the same values ​​as humans, let alone human-like reasoning and cognitive abilities.

　　In this context, it is a huge challenge to study how to make artificial intelligence systems truly understand human value needs and intentions and gain human trust. Research progress in recent years has shown that the success of human-machine collaboration depends not only on team members' consistent cognition of the status quo and goals, but also on whether the team holds the same value orientation. Only through the two-way communication between humans and machines can a consensus of value be efficiently established in the team, so that team members can take trusted behavioral decisions to achieve the ultimate goal.

　　In this study, Professor Zhu Songchun's team designed an ingenious "human-machine collaborative exploration" game to explore the process of robot and human value alignment and the use of two-way communication in this process.

　　The content of this game is: under the command of humans, 3 robots cooperate with humans to find the optimal path from the starting point to the end point on a specific chessboard. The game is played on a checkered board, as shown in the figure below. The lower right corner and upper left corner of the chessboard are the starting point and the ending point of the robot respectively, the black part is the obstacle, and there are gold bricks (materials) and bombs on the chessboard.

　　However, this chessboard environment is not for the human commander to have a panoramic view from the beginning, but is constantly explored by the robot and revealed to the human beings.

　　Scout bots have several additional goals when finding their way: get to their destination as quickly as possible, defuse bombs, explore unknown areas, and collect supplies. However, only the human commander knows the relative priorities of the four goals, and the robot does not. During the game, the robot needs to predict the relative value of these four goals according to the human feedback, and the weight of the relative value is the value function of the human user. For example, assuming that the main goal of human users is to collect materials (golden bricks), then the robot should set a larger weight of the value goal of collecting gold bricks, rather than the timeliness of reaching the destination.

　　This game more realistically simulates the real human-machine collaboration scene, that is, the artificial intelligence system autonomously explores and achieves specific goals in the environment under the supervision of humans (such as robot rescue scene, home service robot scene).

　　Experimental results show that by providing humans with appropriate explanations of their intentions, robots can help humans perceive their worthwhile goals. And as both a listener (inferring the user's intent from the feedback it receives) and an expressor (explaining its decision-making process to the user), bots can more quickly align with humans to realize value.

　　In other words, the whole game actually reveals the real-time value alignment in the mutual cooperation between humans and machines, which can be achieved through the interpretation and evaluation of the value goals by both parties.

　　The above experimental process and results profoundly reveal how the real-time value alignment between human-machine collaboration is achieved through two-way collaboration:

　　First, based on human feedback, the robot estimates the human commander's value goals, and adjusts its own behavior and strategies.

　　Second, the robot needs to explain to the human commander the actions it has taken and planned to take based on the current situation. In cooperation with robots round by round, humans constantly evaluate their intentions and capabilities, and timely constrain and adjust their behavior through instructions. Obviously this is a two-way process.

　　Finally, the value goal of the robot gradually converges, and the commander's feedback to the robot gradually becomes more peaceful, which forms the consistency and unity of the real value of human beings and the value of the robot, and the human and robot system have reached a high degree of mutual trust.

　　In this work, Professor Zhu Songchun's team creatively proposed a two-way collaborative system between humans and robots, and confirmed the availability of the real-time value alignment framework.

　　Several reviewers of this paper have highly affirmed the significance of this research. One reviewer found this study to be important and interesting, and a strong illustration of what it means to use bidirectional communication between humans and AI for value alignment. Another expert commented: This paper successfully proves that two-way cooperation between humans and agents is possible by allowing humans to participate in the game with several specific agents, bringing artificial intelligence research in the field of human-machine teamwork. A big step forward to advance to a more advanced state of the art, and other scholars will greatly learn and be inspired by this research.

　　Zhu Yixin, an assistant professor at the Institute of Artificial Intelligence of Peking University, mentioned some stories that impressed him when he recalled the entire research work.

　　He said persevering when the team was having a hard time and finding a way to fix it was critical to the progress of the project. At the beginning of the project, due to the impact of the new crown epidemic, the school's experimental platform was closed indefinitely. Fortunately, they found an alternative to online experiments in time, and the entire team took the time to quickly learn a new programming language to reduce the cost of online research and solve some technical problems.

　　He also mentioned that it is also important to stand by what he believes to be the right position. During several mid-term reviews, reviewers questioned the project design many times. They made revisions based on some worthy input, but also stuck to what they thought was the right approach, rather than fully adopting the recommendations of the reviewers. Despite the enormous pressure on the team, the process also contributed a lot to the end result.

　　Regarding the next step of this work, researcher Zheng Zilong said that value alignment is the first step towards universal human-machine collaboration. In the future, they will seek to apply the framework to more tasks and AI agents, and explore the alignment of human-machine value in multiple tasks, such as realizing the multi-tasking capability of a single robot, rather than just focusing on the environment of a single task. In addition, they believe that it is also a promising direction to study other factors of the mental model between humans and robots, such as beliefs, desires, intentions, etc., which are all the process of "setting your mind for the machine".

　　"We believe that in the future, humans can build an intelligent society in which humans and machines coexist harmoniously," said researcher Zheng Zilong.