LLM-Driven Multimodal Knowledge Graph Construction for Bridges

Wu, C; Fan, Y; Irvine, H; Robertson, A

doi:10.1109/IGARSS55030.2025.11243950

LLM-Driven Multimodal Knowledge Graph Construction for Bridges

Lookup NU author(s): Hope Irvine, Alex Robertson

Downloads

Full text for this publication is not currently held within this repository. Alternative links are provided below where available.

Abstract

© 2025 IEEE. Recently, there has been a surge of interest in Large Language Models (LLMs), this has been accompanied by notable advances in knowledge graphs for text information extraction and database interactions via natural language instructions. For critical transportation infrastructure such as bridges, knowledge graphs that are limited to textual information are profoundly constrained. Vector data which includes spatial and topological information, and raster data that provides real-time imagery are equally important. The construction of multimodal bridge knowledge graphs remains largely unexplored. The extracting and storing of non-textual modalities typically requires additional AI models and scripts, this extends beyond the capabilities of large language models and results in a significant increase in operational complexity. To address this challenge, we introduce Bridge-MMKG-Agent: an LLM-driven framework that integrates a LLM with a suite of AI-powered models and scripts to construct multimodal knowledge graphs for bridges. We trained a core driver model named Qwen-Bridge, an instruction fine-tuned version of Qwen2.5 using high-quality instruction tuning obtained from GPT-4o. This model is capable of task planning and tool invocation. We also developed a toolkit for constructing multimodal knowledge graphs for bridges, which encompasses modules for data collection, entity extraction, graph construction, and knowledge integration. Users can perform a series of complex operations by interacting with the Bridge-MMKG-Agent through natural language. Experiments and case studies on bridges in China demonstrate that our method significantly enhances the model’s ability to automate complex tasks, bridging the gap between diverse data modalities and improving overall system efficiency. The code and demo will be publicly available at https://github.com/EliuciM/Bridge-MMKG-Agent.