1.we propose a new solution for integrating Pre-trained LM into Diffusion Model, to perform discrete diffusion for text-to-text generation. It requires only low-cost fine-tuning and can performs better than vanilla fine-tuning;
2.we unify the inference process of PLMs and denoising process of discrete diffusion models, into the same masked token recovering task in the non-autoregressive manner;
We conduct extensive experiments on seven text-to-text generation datasets, where our approach can outperform competitive NAR text generation methods, even surpassing SOTA autoregressive PLMs;
Also, as we can use DDIM for fast inference, we can set the diffusion steps for balancing the inference latency and performance:
conda create -n diffusion python=3.10
conda activate diffusion
pip install -r requirements.txt
To obtain the data, first download from the Google-Drive:
tar -zxvf datasets.tar.gzcd scripts
bash run_train_CNNDM.shIf you find this repository helpful, please consider citing our paper:
@inproceedings{zhou2024diffusion,
title={Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation},
author={Zhou, Kun and Li, Yifan and Zhao, Wayne Xin and Wen, Ji-Rong},
booktitle={Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
pages={1438--1451},
year={2024}
}