causalmap.app/events/pilots-or-passengers
EES Lille: pilots or passengers?
An informal chat at the European Evaluation Society conference in Lille, on Thursday 29 October 2026, from about 16:30 at the café tables near Room 0.2.
Are you impressed but concerned by the ability of frontier LLMs, given interviews or other documents, to answer arbitrary evaluation questions? Are you interested in exploring how we can force LLMs to make their workflows and evaluative reasoning more explicit, transparent and checkable, so that evaluators stay the pilots, not passengers?
Scope
- In: general apps and toolkits that let a user answer at least the desk-based, text-analysis part of most of the questions you would see in a typical evaluation’s terms of reference.
- Not our focus here: of course it matters who owns and participates in the evaluation data and processes, but that is not what this group is about.
Starting points
- Why now: a frontier model can already answer the desk-based part of most evaluation questions in minutes, about as accurately as a competent evaluator, and the models get better every month (more).
- Can the logic of evaluative reasoning be built from a few pieces? Such as: Choose documents, mark passages, group them into themes, count, judge against a rubric, write the narrative (more)?
- Should an LLM-powered evaluation analysis be required to be in some sense replicable (more)? What do we think about pre-registration of qualitative analysis designs (more)?
Join in
This is not a formal group with membership. We have in mind a burst of about six months, after which we will see whether it is worth carrying on.
- A webinar or meetup roughly once a month, online, with a show-and-tell from one or two people each time.
- If you are interested, write to Steve Powell at steve@pogol.net to be invited.
- The hands-on work could happen in the MERL Tech Slack and/or its github. Its Slack channels come with membership of the MERL Tech Community of Practice, which is free.
Reading and tools
- Pilots, not passengers: the argument in full. Who chooses the method, who decides what counts as good, and how can anybody else check what was done?
- An open-source plugin for Claude that we have been building. You agree the analysis with it before anything is read, then fixed rules count every number and check every quotation.
- An open format for recording an analysis so that somebody else can rerun it (latest version).
From the MERL Tech Initiative
- MERL Tech on GitHub: code from the MERL Tech Initiative (MTI).
- MTI’s playground: their experiments, including anything built by vibe coding (describing software to an AI and letting it write the code), as tools for responsible practice in monitoring, evaluation, research and learning.