Subsequent projects | Special projects

Start of funding 01.07.2024

Silent Speech: Enabling quiet communication through EMG

Prof. Dr. Björn W. Schuller
Technische Universität München
Klinikum rechts der Isar - Chair of Health Informatics

Prof. Dr. Shrikanth S. Narayanan
University of Southern California, Los Angeles
Signal Analysis and Interpretation Laboratory (SAIL)


Silent Computational Paralinguistics (SCP) focuses on recognizing speaker states as well as traits during non-audible speech from sources such as facial ElectroMyoGraphy (EMG) signals. SCP can help to interact with next generaon socio-emoonally competent speech technology in a private manner or the mute. The cooperaon aims to significantly advance the field of SCP by collecting a larger EMG-speech corpus and developing improved machine learning models. The project will advance research in the following directions:
1) Collecting a larger, more diverse, more expressive EMG-Silent Speech dataset with sessions being recorded from a more diverse speaker set consisng of project participants from both partner institutions, with the participants themselves performing more varied communication expressions.
2) Establishing relevant baseline metrics for modeling the collected dataset, this is achieved by applying more traditional machine learning approaches to establish baseline dataset modeling parameters.
3) Investigating advanced deep learning approaches for SCP modeling, methods into transfer learning from the speech modality to the EMG modality, and representaon learning, with EMG-to-speech synthesis being of high priority for invesgaon.