ZiMT Journal Club March 2022: Prof. Dr. Seung Hee Yang / Intelligent Operation Room Speech Assistant

Symbolic picture for the article. The link opens the image in a large view.

Prof. Dr. Seung Hee Yang, Junior Professor of Biomedical Spoken Language Processing, Department of Artificial Intelligence in Biomedical Engineering (AIBE), FAU

There has been a growing interest to support surgical procedures with machine models that analyse and understand operations in different semantic levels. These procedures are necessary for developing context-aware intelligent systems, which can assist physicians during their operations by automatically detecting surgical phases, such as sterilsation, anesthesia, and incision, and by understanding commands uttered by the physicians.

Towards this objective, we are developing a new kind of intelligent assistance system for X-ray minimally invasive port catheter placement surgery. The system accesses X-ray images, parameters such as gantry and table position, and understands speech uttered during the surgery. While previous researches concentrate on instrumental signals and video clip-based activity detection, this new approach uses time-aligned multimodal dataset comprising of video, image, speech, and text. With this unique dataset, meaningful features will be extracted, and reliable patterns will be learnt using unsupervised active learning techniques. In its final state, the intelligent workflow assistance system will be able to control the C-arm according to the commands and the X-ray image input and help efficient management of operating rooms.