AI-assisted causal mapping: a validation study

How well does an untrained AI assistant perform in identifying and labelling causal links
within a corpus of stories told by recipients of international development aid, compared
with human experts?

Here is a preprint of a forthcoming paper (not yet peer-reviewed).

People doing causal systems mapping are often interested in harvesting claims about causal links
from text sources, for example from interview transcripts with experts: a task which used to be
time-consuming. In this paper, we show how to use generative AI as a low-level assistant to
exhaustively and transparently identify and then summarise causal claims. We use techniques
from causal mapping (Axelrod, 1976). We do not try to model the system or assess the strength
of causal links but rather to assess the strength of the evidence for each causal link or pathway:
an approach which is comparatively easy to automate.
We ask: Is the ability of LLMs to identify causal claims within texts of sufficient quality to be
useful, and what can we say about reliability or validity? The results are encouraging. We
conclude by discussing risks and ethical issues, as well as suggesting some areas for further
research.