Abstract
Synthesizing complex images from textual descriptions presents a significant challenge. Relying on tremendous training data and model size, recent Diffusion, Autoregressive and GAN models have made significant progress in synthesizing photo-realistic images. However, the high computing budget and hardware requirements due to the large data and model size hinder the flexibility of employing these models. This paper introduces Multiscale Semantic Context-Aware Generative Adversarial Networks (MSCA-GAN), which achieves strong text alignment with limited data while balancing generation quality and efficiency. The proposed MSCA-GAN incorporates innovative modules for textual semantic injection, delivery, and validation. Specifically, the Semantic Adaptive Affine Fusion (SAAF) module dynamically adjusts expression weights of textual semantic information to align with the feature generation process, encompassing global to detailed aspects. Furthermore, the CrossBlock Context Aware Encoding (CCAE) module explicitly establishes semantic context across different synthesis blocks during the local feature delivery. Finally, MSCA-GAN introduces an additional CLIP guidance term to verify the semantic consistency of local features at various scales. MSCA-GAN is pre-trained on the CC3M and CC12M datasets, which only contain limited data. Extensive experiments confirm that the MSCA-GAN performs competitively in terms of image quality and generation efficiency, both quantitatively and qualitatively.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Balaji, Y., et al.: eDiffI: text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 (2022)
Changpinyo, S., Sharma, P., Ding, N., Soricut, R.: Conceptual 12 m: pushing webscale image-text pre-training to recognize long-tail visual concepts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558–3568 (2021)
Ding, M., et al.: Cogview: Mastering text-to-image generation via transformers. Adv. Neural. Inf. Process. Syst. 34, 19822–19835 (2021)
Ding, M., Zheng, W., Hong, W., Tang, J.: CogView2: faster and better text-toimage generation via hierarchical transformers. Adv. Neural. Inf. Process. Syst. 35, 16890–16902 (2022)
Gu, S., et al.: Vector quantized diffusion model for text-to-image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696–10706 (2022)
Härkönen, E., Aittala, M., Kynkäänniemi, T., Laine, S., Aila, T., Lehtinen, J.: Disentangling random and cyclic effects in time-lapse sequences (2022)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a nash equilibrium. CoRR abs/1706.08500 (2017). http://arxiv.org/abs/1706.08500
Jia, C., et al.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International Conference on Machine Learning, pp. 4904–4916. PMLR (2021)
Kang, M., et al.: Scaling up gans for text-to-image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124–10134 (2023)
Karras, T., et al.: Alias-free generative adversarial networks. Adv. Neural. Inf. Process. Syst. 34, 852–863 (2021)
Krishna, R., et al.: Visual Genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vision 123, 32–73 (2017)
Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning, pp. 12888–12900. PMLR (2022)
Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1), 5485–5551 (2020)
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2), 3 (2022)
Ramesh, A., et al.: Zero-shot text-to-image generation. In: International Conference on Machine Learning, pp. 8821–8831 (2021)
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695 (2022)
Saharia, C., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Adv. Neural. Inf. Process. Syst. 35, 36479–36494 (2022)
Sauer, A., Chitta, K., Müller, J., Geiger, A.: Projected gans converge faster. Adv. Neural. Inf. Process. Syst. 34, 17480–17492 (2021)
Sauer, A., Karras, T., Laine, S., Geiger, A., Aila, T.: StyleGAN-T: unlocking the power of gans for fast large-scale text-to-image synthesis. arXiv preprint arXiv:2301.09515 (2023)
Sauer, A., Schwarz, K., Geiger, A.: StyleGAN-XL: scaling Style-GAN to large diverse datasets. In: Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings (2022). https://doi.org/10.1145/3528233.3530738
Schuhmann, C., et al.: Laion-5b: an open large-scale dataset for training next generation image-text models. Adv. Neural. Inf. Process. Syst. 35, 25278–25294 (2022)
Tao, M., Bao, B.K., Tang, H., Xu, C.: Galip: generative adversarial clips for text-to-image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14214–14223 (2023)
Tao, M., et al.: DF-GAN: a simple and effective baseline for text-to-image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16515–16525 (2022)
Wu, C., et al.: NÜWA: visual synthesis pre-training for neural visual world creation. In: European Conference on Computer Vision, pp. 720–736. Springer (2022). https://doi.org/10.1007/978-3-031-19787-1_41
Yu, J., et al.: Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789
Zhou, Y., et al.: Towards language-free training for text-to-image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17907–17917 (2022)
Acknowledgment
This work is supported in part by the Key R&D Program of Shandong Province, China (Grant no. 2023CXGC010801).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Dong, P., Wu, L., Meng, L., Meng, X. (2024). Text-to-Image Generation with Multiscale Semantic Context-Aware Generative Adversarial Networks. In: Huang, DS., Pan, Y., Guo, J. (eds) Advanced Intelligent Computing Technology and Applications. ICIC 2024. Lecture Notes in Computer Science, vol 14873. Springer, Singapore. https://doi.org/10.1007/978-981-97-5615-5_16
Download citation
DOI: https://doi.org/10.1007/978-981-97-5615-5_16
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-5614-8
Online ISBN: 978-981-97-5615-5
eBook Packages: Computer ScienceComputer Science (R0)