Honda Research Institute (HRI) Japan and EEMCS collaborate on human-centred computing for intelligent communication and social interaction
Scientists from TU Delft faculty of EEMCS and Honda Research Institute (HRI) in Japan are collaborating on developing a comprehensive AI system for human and group understanding. This novel system is envisaged to realize empathetic interaction in a harmonious hybrid society where humans and AI agents live together and interact 24/7. The project – that will run from 2023 to 2027 – is focused on creating a generic AI system that uses nonverbal cues from human body poses, e.g., hand gestures, eye gaze, facial expressions and speech, for various social interaction tasks. This AI system could make a significant impact on many real-world applications like industrial work, personal daily assistance, and healthcare, where intelligent agents are getting more common in recent years.
Human-centred computing
Scientists from the faculty of Electrical Engineering, Mathematics and Computer Science and the Honda Research Institute (HRI) in Japan are collaborating on an advanced AI system designed to understand user behaviour and generate responses for AI agents. Jointly funded by the Honda Research Institute (HRI) Japan and the Dutch Top Consortium for Knowledge and Innovation (TKI), the project has funded two PhD students and will run from 2023 to 2027. The research focuses on creating a novel AI system that integrates input signals from human body poses, hand gestures, eye gaze, facial expressions, and speech, for various social interaction tasks. By employing a holistic approach and state-of-the-art generative AI technologies, these nonverbal cues are fused to represent social dynamics to drive AI agents such as robots and/or virtual characters in Extended Reality (XR) applications (virtual reality and other immersive technologies). This AI system has the potential to significantly impact assistant robots in real-world applications, including industrial work, personal daily assistance, and personal healthcare.
Satoshi Shigemi and Lucas van Vliet
Left to right: Jouh Yeong Chew, Satoshi Shigemi, Lucas van Vliet and Xucong Zhang
Scientist collaboration
Assistant Professor Xucong Zhang from the Pattern Recognition and Bioinformatics Section is leading the project at EEMCS. His primary research interest lies in human-centred computing, aiming to develop technologies that sense and serve human users. His research encompasses face, hand, and body estimations as part of holistic human behaviour modelling for intelligent interaction between humans and AI agents. The goal is to achieve intelligent human-robot interaction and avatar generation in XR environments.
Representatives from the Honda Research Institute (HRI) Japan - President Satoshi Shigemi and Scientist Jouh Yeong Chew, who is leading the project at HRI Japan - visited TU Delft from the 4th to 5th of July 2024 to discuss the progress of research collaboration and future research direction. They met EEMCS Dean Lucas van Vliet and the management of the Department of Intelligent Systems. They talked about the joined efforts in the field of human-centred computing for intelligent communication and social interaction to solidify the cooperation between both research institutes.
Satoshi Shigemi and Lucas van Vliet