Abstract
As the fields of deep learning and artificial intelligence rapidly advance, significant progress has been made in image understanding and natural language processing. However, the challenge of accurately and deeply understanding images in complex scenes, such as remote sensing imagery, remains a critical issue in current research. This paper introduces a novel approach that combines targeted object detection results with large language models to address the deep understanding and description of complex visual scenes. By incorporating multimodal understanding models (such as CLIP and GPT) and prompt engineering, along with BPO strategies, our method achieves a deep and nuanced understanding and description of complex scenes. We have developed a user interface and experimentally validated the effectiveness and accuracy of our proposed method in real-world application scenarios, demonstrating the frameworkâs superior performance in understanding complex scenes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this paper.
References
Brohan, A., et al.: RT-2: vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv: 2307.15818 (2023)
Cheng, J., et al.: Black-box prompt optimization: aligning large language models without model training. arXiv preprint arXiv: 2311.04155 (2023)
Chowdhary, K.R.: Natural language processing. In: Chowdhary, K.R. (ed.) Fundamentals of Artificial Intelligence, pp. 603â649. Springer, New Delhi (2020). https://doi.org/10.1007/978-81-322-3972-7_19
Hossain, M.Z., Sohel, F., Shiratuddin, M.F., Laga, H.: A comprehensive survey of deep learning for image captioning. ACM Comput. Surv. (CsUR) 51(6), 1â36 (2019)
Li, L., Zhang, Y., Chen, L.: Prompt distillation for efficient LLM-based recommendation. In: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 1348â1357 (2023)
Liu, X., et al.: Large language models are few-shot health learners. arXiv preprint arXiv: 2305.15525 (2023)
Liu, Z., He, X., Tian, Y., Chawla, N. V.: Can we soft prompt llms for graph learning tasks? arXiv preprint arXiv: 2402.10359 (2024)
Manmadhan, S., Kovoor, B.C.: Visual question answering: a state-of-the-art review. Artif. Intell. Rev. 53(8), 5705â5745 (2020)
Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., Stanovsky, G.: State of what art? A call for multi-prompt llm evaluation. arXiv preprint arXiv: 2401.00595 (2023)
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748â8763. PMLR (2021)
Wang, H., Zhang, Y., Yu, X.: An overview of image caption generation methods. Comput. Intell. Neurosci. 2020 (2020)
Wang, J., Wang, Z., Weng, Y., Li, Y.: DRPDDet: dynamic rotated proposals decoder for oriented object detection. In: Luo, B., Cheng, L., Wu, Z.G., Li, H., Li, C. (eds.) ICONIP 2023. LNCS, vol. 14452, pp. 103â117. Springer, Singapore (2024). https://doi.org/10.1007/978-981-99-8076-5_8
Wen, C., Hu, Y., Li, X., Yuan, Z., Zhu, X.X.: Vision-language models in remote sensing: current progress and future trends. arXiv preprint arXiv: 2305.05726 (2023)
Xie, X., Cheng, G., Wang, J., Yao, X., Han, J.: Oriented R-CNN for object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3520â3529 (2021)
Xu, H., Han, L., Yang, Q., Li, M., Srivastava, M.: Penetrative AI: making LLMS comprehend the physical world. In: Proceedings of the 25th International Workshop on Mobile Computing Systems and Applications, pp. 1â7 (2024)
Zamfirescu-Pereira, J., Wong, R. Y., Hartmann, B., Yang, Q.: Why johnny canât prompt: how non-AI experts try (and fail) to design LLM prompts. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1â21 (2023)
Zou, Z., Chen, K., Shi, Z., Guo, Y., Ye, J.: Object detection in 20 years: a survey. Proc. IEEE 111(3), 257â276 (2023)
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grants 52274160, 51874300, the National Natural Science Foundation of China and Shanxi Provincial Peopleâs Government Jointly Funded Project of China for Coal Base and Low Carbon under Grant U1510115, Fundamental Research Funds for the Central Universities under Grant 2023QN1079.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
Âİ 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wang, Z., Xu, Z., Yang, W., Chen, W., Yang, Y. (2024). A Large Model Assisted Remote Sensing Image Scene Understanding Algorithm Based on Object Detection. In: Huang, DS., Si, Z., Guo, J. (eds) Advanced Intelligent Computing Technology and Applications. ICIC 2024. Lecture Notes in Computer Science, vol 14867. Springer, Singapore. https://doi.org/10.1007/978-981-97-5597-4_5
Download citation
DOI: https://doi.org/10.1007/978-981-97-5597-4_5
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-5596-7
Online ISBN: 978-981-97-5597-4
eBook Packages: Computer ScienceComputer Science (R0)