Toggle Main Menu Toggle Search

Open Access padlockePrints

Language as a latent sequence: Deep latent variable models for semi-supervised paraphrase generation

Lookup NU author(s): Dr Lei ShiORCiD



This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND).


This paper explores deep latent variable models for semi-supervised paraphrase generation, where the missing target pair for unlabelled data is modelled as a latent paraphrase sequence. We present a novel unsupervised model named variational sequence auto-encoding reconstruction (VSAR), which performs latent sequence inference given an observed text. To leverage information from text pairs, we additionally introduce a novel supervised model we call dual directional learning (DDL), which is designed to integrate with our proposed VSAR model. Combining VSAR with DDL (DDL+VSAR) enables us to conduct semi-supervised learning. Still, the combined model suffers from a cold-start problem. To further combat this issue, we propose an improved weight initialisation solution, leading to a novel two-stage training scheme we call knowledge-reinforced-learning (KRL). Our empirical evaluations suggest that the combined model yields competitive performance against the state-of-the-art supervised baselines on complete data. Furthermore, in scenarios where only a fraction of the labelled pairs are available, our combined model consistently outperforms the strong supervised model baseline (DDL) by a significant margin (p<.05; Wilcoxon test). Our code is publicly available at

Publication metadata

Author(s): Yu J, Cristea AI, Harit A, Sun Z, Aduragba OT, Shi L, Moubayed NA

Publication type: Article

Publication status: Published

Journal: AI Open

Year: 2023

Volume: 4

Pages: 19-32

Print publication date: 03/06/2023

Online publication date: 26/05/2023

Acceptance date: 18/05/2023

Date deposited: 24/08/2023

ISSN (electronic): 2666-6510

Publisher: Elsevier


DOI: 10.1016/j.aiopen.2023.05.001


Altmetrics provided by Altmetric