causalmap.app/events/pilots-or-passengers

EES Lille: pilots or passengers?

An informal chat at the European Evaluation Society conference in Lille, on Thursday 29 October 2026, from about 16:30 at the café tables near Room 0.2.

It is for evaluators, technical or not, who want AI to follow an explicit evaluation method, in steps that somebody else could check or rerun, so that evaluators stay the pilots.

Starting points

• Why now: a frontier model can already answer the desk-based part of most evaluation questions in minutes, about as well as a competent evaluator, and the models get better every month (more).
• Agreeing the method with commissioners: reconstructibility as an option, or even pre-registration, with the workflow fixed in advance in full, half-way or not at all (more).
• A reconstructible zip file handed over with the report, holding the sources, the workflow, what every step produced and the rubrics, so that somebody else can rebuild the analysis (more).
• Can the logic of evaluative reasoning be built from a few pieces? Choose documents, mark passages, group them into themes, count, judge against a rubric, write (more).

Join in

This is not a formal group with membership. We have in mind a burst of about six months, after which we will see whether it is worth carrying on.

• If you are interested, write to Steve Powell at steve@pogol.net and you will get the occasional invitation to calls and show-and-tell sessions. You do not need Slack or any particular software.
• The hands-on work happens in the MERL Tech Slack. Its channels come with membership of the MERL Tech Community of Practice, which is free.

Reading and tools

• Pilots, not passengers: the argument in full. Who chooses the method, who decides what counts as good, and how can anybody else check what was done?
• An open-source plugin for Claude that we have been building. You agree the analysis with it before anything is read, then fixed rules count every number and check every quotation.
• An open format for recording an analysis so that somebody else can rerun it (latest version).

From the MERL Tech Initiative

• MERL Tech on GitHub: code from the MERL Tech Initiative (MTI).
• MTI’s playground: their experiments, including anything built by vibe coding (describing software to an AI and letting it write the code), as tools for responsible practice in monitoring, evaluation, research and learning.