AI-assisted causal mapping: a validation study

How well does an untrained AI assistant perform in identifying and labelling causal links
within a corpus of stories told by recipients of international development aid, compared
with human experts?

People doing causal systems mapping are often interested in harvesting claims about causal links from text sources, for example from interview transcripts with experts: a task which used to be time-consuming. In this paper, we show how to use generative AI as a low-level assistant to exhaustively and transparently identify and then summarise causal claims. We use techniques from causal mapping (Axelrod, 1976). We do not try to model the system or assess the strength of causal links but rather to assess the strength of the evidence for each causal link or pathway: an approach which is comparatively easy to automate. We ask: Is the ability of LLMs to identify causal claims within texts of sufficient quality to be useful, and what can we say about reliability or validity? The results are encouraging. We conclude by discussing risks and ethical issues, as well as suggesting some areas for further research.