Mandarin Chinese Cued Speech Dataset (MCCSD)

天气预报怎么说                                                                  你好, 很高兴认识你

我明天过来接你                                                            为我们的合作成功干杯

This MCCS dataset is the first large-scale Mandarin Chinese Cued Speech dataset. This dataset covers 23 major categories of scenarios (e.g, communication, transportation and shoping) and 72 subcategories of scenarios (e.g, meeting, dating and introduction). It is recorded by four skilled native Mandarn Chinese Cued Speech cuers with portable cameras on the mobile phones. The Cued Speech videos are recorded with 30fps and 1280x720 format. We provide the raw Cued Speech videos, text file (with 1000 sentences) and corresponding annotations which contains two kind of data annotation. One is continuious video annotation with ELAN, the other is discrete audio annotations with Praat.

We are excited to announce the release of MCCSD-2024, an expanded version of the Mandarin Chinese Cued Speech Dataset.

This new version includes 7 native chinese cuers and a total of 7,000 sentences (1000 for each cuer), significantly increasing the diversity and scale of the dataset.

The dataset is divided into two main groups:

• 6,000 sentences recorded by normal-hearing individuals.
• 1,000 sentences recorded by hearing-impaired individuals.

This new version aims to provide a more comprehensive resource for researchers studying Cued Speech, particularly in the context of cross-modal learning and accessibility technologies.

The inclusion of data from hearing-impaired individuals offers unique insights into the variations and adaptations in Cued Speech production.

The dataset maintains the same high-quality standards as the original MCCSD, with videos recorded at 30fps and 1280x720 resolution.

Annotations are provided in both ELAN and Praat formats, ensuring compatibility with a wide range of research tools.

If you use this MCCS dataset for your research, please consider citing the following papers:

1. Objective Hand Complexity Comparison between Two Mandarin Chinese Cued Speech Systems. Li Liu, Gang Feng, Xiaoxi Ren, Xianping Ma International Symposium on Chinese Spoken Language Processing (ISCSLP) (2022) LINK
2. Re-synchronization using the Hand Preceding Model for Multi-modal Fusion in Automatic Continuous Cued Speech Recognition. Li Liu, Gang Feng, Denis Beautemps, Xiao-Ping Zhang. IEEE Transactions on Multimedia (2020) LINK
3. A Pilot Study on Mandarin Chinese Cued Speech. Li Liu, Gang Feng. American Annals of the Deaf (2019) LINK
4. Cross-modal mutual learning for cued speech recognition. Liu, L., & Liu, L. (2023, June). 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2023) . LINK
5. A Novel Interpretable and Generalizable Re-synchronization Model for Cued Speech based on a Multi-Cuer Corpus. Lufei Gao, Shan Huang, & Li Liu (2023). arXiv preprint arXiv:2306.02596. Accepted by Interspeech 2023 LINK