Abstract
Large-scale pre-trained language models have demonstrated impressive results in producing human-like texts. However, controlling the text generation process remains a challenge for researchers. Controllable text generation consists of generating sentences that satisfy desired constraints (e.g., sentiment, topic, or keywords). Recent studies that control the decoding stage of a language model have proved the high efficiency of this approach for control of generated texts. This approach, in contrast to the fine-tuning of pre-trained language models, requires much less computing resources. In this work, we propose and investigate a method that controls the process of language generation using perplexity minimization. The method is designed to create stories from a sequence of guide phrases that form a storyline and is based on the search for sequences of tokens that reduce text perplexity when generation is directed towards the guide phrase. First, we generate several arbitrary small sequences of tokens from the language model vocabulary. Then we choose the most probable subsequence - the one, the probability of following the guide phrase after which is the biggest. The proposed method induces the model to shift the content of the generated text to the guide phrase. Experiments on the Russian-language corpus of fairy tales with storylines have shown the high efficiency of the proposed method for creating stories corresponding to the user-specified storyline.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
References
Amidei, J., Piwek, P., Willis, A.: Agreement is overrated: a plea for correlation to assess human evaluation reliability. In: Proceedings of the 12th International Conference on Natural Language Generation, pp. 344–354 (2019)
Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877–1901 (2020)
Campos, R., et al.: YAKE! Keyword extraction from single documents using multiple local features. Inf. Sci. J. 509, 257–289 (2020)
Cavazza, M., Pizzi, D.: Narratology for interactive storytelling: a critical introduction. In: Göbel, S., Malkewitz, R., Iurgel, I. (eds.) TIDSE 2006. LNCS, vol. 4326, pp. 72–83. Springer, Heidelberg (2006). https://doi.org/10.1007/11944577_7
Dathathri, S., et al.: Plug and play language models: a simple approach to controlled text generation. arXiv preprint arXiv:1912.02164 (2020)
Fan, A., Lewis, M., Dauphin, Y.: Hierarchical neural story generation. arXiv preprint arXiv:1805.04833 (2018)
Hejazi, H.D., Khamees, A.A., Alshurideh, M., Salloum, S.A.: Arabic text generation: deep learning for poetry synthesis. In: Hassanien, A.-E., Chang, K.-C., Mincong, T. (eds.) AMLTA 2021. AISC, vol. 1339, pp. 104–116. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-69717-4_11
Keskar, N.S., McCann, B., Varshney, L., Xiong, C., Socher, R.: CTRL - a conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858 (2019)
Korobov, M.: Morphological analyzer and generator for Russian and Ukrainian languages. In: Khachay, M.Y., Konstantinova, N., Panchenko, A., Ignatov, D.I., Labunets, V.G. (eds.) AIST 2015. CCIS, vol. 542, pp. 320–332. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-26123-2_31
Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045–3059 (2021)
Li, Y., Li, K., Ning, H., Xia, X., Guo, Y., Wei, C., Cui, J., Wang, B.: Towards an online empathetic chatbot with emotion causes. In: Proceedings 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2041–2045 (2021)
Li, X.L., Liang, P.: Prefix-tuning: optimizing continuous prompts for generation. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 4582–4597 (2021)
Meister, C., Vieira, T., Cotterell, R.: If beam search is the answer, what was the question? In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp. 2173–2185 (2020)
Mirowski, P., Mathewson, K.W., Pittman, J., Evans, R.: Co-writing screenplays and theatre scripts with language models: an evaluation by industry professionals. arXiv preprint arXiv:2209.14958 (2022)
Pascual, D., Egressy, B., Meister, C., Cotterell, R., Wattenhofer, R.: A plug-and-play method for controlled text generation. In: Findings of the Association for Computational Linguistics, pp. 3973–3997 (2021)
Prabhumoye, S., Black, A.W., Salakhutdinov, R.: Exploring controllable text generation techniques. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 1–14 (2020)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. In: OpenAI Blog, vol. 1, no. 8 (2019)
Rosenthal, J.A.: Qualitative descriptors of strength of association and effect size. J. Soc. Serv. Res. 21(4), 37–59 (1996)
Škrlj, B., Repar, A., Pollak, S.: RaKUn: rank-based keyword extraction via unsupervised learning and meta vertex aggregation. In: Martín-Vide, C., Purver, M., Pollak, S. (eds.) SLSP 2019. LNCS (LNAI), vol. 11816, pp. 311–323. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-31372-2_26
Touvron, H., et al.: LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS), vol. 30, pp. 6000–6010 (2017)
Vychegzhanin, S., Kotelnikov, E.: Collocation2Text: controllable text generation from guide phrases in Russian. In: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference “Dialogue-2022”, vol. 21, pp. 564–576 (2022)
Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., Weston, J.: Neural text generation with unlikelihood training. In: Proceedings of the 8th International Conference on Learning Representations, pp. 1–18 (2020)
Wiseman, S., Shieber, S., Rush, A.: Challenges in data-to-document generation. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2253–2263 (2017)
Yang, K., Tian, Y., Peng, N., Klein, D.: Re3: generating longer stories with recursive reprompting and revision. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), pp. 4393–4479 (2022)
Yao, L., Peng, N., Weischedel, R., Knight, K., Zhao, D., Yan, R.: Plan-and-write: towards better automatic storytelling. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, pp. 7378–7385 (2019)
Yuan, A., Coenen, A., Reif, E., Ippolito, D.: Wordcraft: story writing with large language models. In: 27th International Conference on Intelligent User Interfaces, pp. 841–852 (2022)
Zehtab-Salmasi, A., Feizi-Derakhshi, M.R., Balafar, M.A.: FRAKE: fusional real-time automatic keyword extraction. arXiv preprint arXiv:2104.04830 (2021)
Zhang, H., Song, H., Li, S., Zhou, M., Song, D.: A survey of controllable text generation using transformer-based pre-trained language models. arXiv preprint arXiv:2201.05337 (2022)
Zhang, Y., Wang, G., Li, C., Gan, Z., Brockett, C., Dolan, B.: POINTER: constrained progressive text generation via insertion-based generative pre-training. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8649–8670 (2020)
Zhu, Y., et al.: Texygen: a benchmarking platform for text generation models. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1097–1100 (2018)
Acknowledgments
This work was supported by Russian Science Foundation, project № 23-21-00330, https://rscf.ru/en/project/23-21-00330/.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
Appendix A
Distribution of the number of plot phrases.
Number of sentences in fairy tales.
Appendix B
The first column of Table 2 contains the number of phrases in the plot, the second - the number of fairy tales with such a number of phrases, the third - the share of the total number of fairy tales in the training corpus. The fifth and sixth columns contain statistics on the number of tokens received using the ruGPT-3 Large and LLaMA tokenizer in tales of training corpus, depending on the number of plot phrases.
Appendix C
Table 3 contains statistical characteristics of generated texts: avg length - the average length of texts (in words); vocab size - the number of different words; distinct-n - the ratio of distinct n-grams over the total number of n-grams.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Vychegzhanin, S., Kotelnikova, A., Sergeev, A., Kotelnikov, E. (2024). Controllable Story Generation Based on Perplexity Minimization. In: Ignatov, D.I., et al. Analysis of Images, Social Networks and Texts. AIST 2023. Lecture Notes in Computer Science, vol 14486. Springer, Cham. https://doi.org/10.1007/978-3-031-54534-4_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-54534-4_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-54533-7
Online ISBN: 978-3-031-54534-4
eBook Packages: Computer ScienceComputer Science (R0)