Exploring knowledge extraction techniques for system dynamics modelling: comparative analysis and considerations

Abstract

Knowledge graphs, semantic networks, and similar formal knowledge modelling structures have become popular solutions to express linguistic entities and relations between them. They are computer representations of human knowledge with all its semantic richness, and therefore powerful tools for structuring all kinds of data. Creating such models is, however, an extensive task which involves meticulous manual work by developers and experts. With the large amount of online textual information, models ideally can be automatically extracted, enriched, and maintained using Natural Language Processing (NLP) techniques. In this paper, we explore these techniques and present a comparative study of traditional knowledge graph extraction techniques (Keyword Extraction, Named Entity Recognition, co-occurrences) and a state-of-the-art large language model (LLM) in the context of systems dynamics modelling. System dynamics modelling is an approach to model the behaviour of complex systems, such as policy developments and politics. Often graphs and diagrams are used to support this process. The main contributions of this paper are twofold: first, a comparison and evaluation of traditional NLP methods compared to an LLM; second, an interface for including human feedback, enhancing future collaboration between system and domain expert. Our study informs practitioners about the suitability of these techniques to support problem analysis using system dynamics modelling and paves the way for future research in refining and integrating approaches.