Manchester Metropolitan University's Research Repository

    Self-supervised spontaneous latent-based facial expression sequence generation

    Yap, Chuin Hong ORCID logoORCID: https://orcid.org/0000-0003-2251-9308, Yap, Moi Hoon ORCID logoORCID: https://orcid.org/0000-0001-7681-4287, Davison, Adrian K and Cunningham, Ryan ORCID logoORCID: https://orcid.org/0000-0001-6883-6515 (2023) Self-supervised spontaneous latent-based facial expression sequence generation. IEEE Open Journal of Signal Processing, 4. pp. 304-312. ISSN 2644-1322

    Published Version
    Available under License Creative Commons Attribution Non-commercial No Derivatives.

    Download (17MB) | Preview


    In this paper, we investigate the spontaneity issue in facial expression sequence generation. Current leading methods in the field are commonly reliant on manually adjusted conditional variables to direct the model to generate a specific class of expression. We propose a neural network-based method which uses Gaussian noise to model spontaneity in the generation process, removing the need for manual control of conditional generation variables. Our model takes two sequential images as input, with additive noise, and produces the next image in the sequence. We trained two types of models: single-expression, and mixed-expression. With single-expression, unique facial movements of certain emotion class can be generated; with mixed expressions, fully spontaneous expression sequence generation can be achieved. We compared our method to current leading generation methods on a variety of publicly available datasets. Initial qualitative results show our method produces visually more realistic expressions and facial action unit (AU) trajectories; initial quantitative results using image quality metrics (SSIM and NIQE) show the quality of our generated images is higher. Our approach and results are novel in the field of facial expression generation, with potential wider applications to other sequence generation tasks.

    Impact and Reach


    Activity Overview
    6 month trend
    6 month trend

    Additional statistics for this dataset are available via IRStats2.


    Repository staff only

    Edit record Edit record