The other time I noticed it, though, was when in a different project, I was working long hours with Claude Code going back and forth doing analysis. In my mind, each time I was requesting something new to be done, I thought it was simply editing an existing code. I would check that code and it would be correct, but then later on I took a step back and found not K programs, but more like 5K programs. It was like I was looking at a very old problem that used to haunt me when I was first starting out where I had copies of the same code with weird naming conventions, only this time Claude Code was doing it, putting even more distance between me and the work.

In several posts, I alluded to the fact that I have been building my own personalized dashboard that functions as a manifestation of a rigid checklist for panel data. I already have a rigid checklist I follow based partly on Don Rubin’s article “Design Trumps Analysis”, as well as a diff-in-diff checklist by Pedro Sant’Anna that I’ve talked about on here before, as well as tweaked (as well as included in my new book that comes out next month). Plus things by Guido Imbens that he always seems to include in his survey articles (see here for an example, and flip more towards the end). And so the spine of the dashboard is the checklist, and once I have it done, I’ll share it, with a description and a walk through.

I suspect that there is some kind of high level general conceptual framework of a harness-workflow-for-empirical-research-using-AI-agents, and that once we heard it articulated, it’ll be obvious that that is the way that work should be organized when using AI agents for empirical research. And then we will pivot our workflows towards it. But my hunch is that even if that simple model is articulated, we will still be iterating on that to make our own dashboards and workflows if only because comparative advantage probably will drive it towards that equilibrium.

This morning, though, I was thinking about my own. I’ve got a checklist with a series of rigid steps. It’s clocking in at around 12 to 14 steps that I follow when working on a causal inference project, and it more or less runs like this:

0. PKG: Write down all the software packages I will be using for the project

1. TARGET: What is the causal parameter, expressed using potential outcomes, populations and weights? And if there is more than one causal parameter, name them clearly.

2. BITE: Illustrate the treatment’s bite. This is where I simply document facts about the intervention. Usually the goal is to create around 2-3 exhibits (tables and figures) per stage, too. So I might show a time series of meth use over time. I might show policing differences using a heat map laid over a county shape file. Something, though, to really help me understand the treatment operating in the world.

3. COVARIATES. This is the stage where I select covariates needed to satisfy unconfoundedness, conditional parallel trends, or possibly the a factor model of the untreated potential outcome. I have a skill I created, also, call /covariates that I’ll explain another time which helps pull from my own personal beliefs candidate covariates based on a particular modeling philosophy I believe in. Again for another day. But in covariates, I produce tables measuring imbalance, plot propensity scores, and so on. Again — 2-3 exhibits.

4. ROLLOUT. This is where I document the rollout of the treatment, be it two groups of more than two groups. I make a table counting the number of units by cohort, named if possible, as well as sample shares (number of treated units divided by the total number of treated units). But I also then make a figure showing the rollout using Yiqing Xu’s “panelview” command.

5. OUTCOME. This is where I first present exhibits (tables and figures) of the outcome. If panel data, then it’s the outcome’s evolution over time. If not, then some other way. The goal is a presentation, though, that matches the analysis that will be done.

6. POWER. In this stage, I do power calculations. What is the minimum detectable effect for size 0.05 and power 0.8? And am I powered for it in my own sample? I create power curves for this typically, as well as try to quantify over-rejection problems I found in my Monte Carlos so that I can decide if I’m going to even choose a particular inference method or a given estimator in the first place.

7. ESTIMATOR. And then there is the selection of the estimator. And this is entirely flowing from the TARGET stage, the COVARIATES stage, and my interpretation of BITE. Because if I think I see signs of randomized treatment assignment in that BITE stage, I may lean in one direction, but if I do not, I may lean more towards something like parallel trends or a factor model of the untreated potential outcome. Whichever, though, this is where the estimator is chosen, and when it is estimated.

8. EVENT. Straightforward but I have for now separated out the production of event study graphics from the ESTIMATOR stage, though I am leaning towards combining them.

9. FALSIFY. Now I start running down possible falsifications, like outcomes that should not be affected by the treatment versus groups that should not be affected by the treatment.

10. MECHANISM. Here I begin to work through mechanisms. Are there certain mechanisms that I think I could be studying as possible explanations for the things I found in the ESTIMATOR and EVENT and FALSIFY stages?

11. PIPELINE. Then I rerun everything to ensure it replicates. I also run /referee2 here as well as /blindspot, though frankly those have also been being ran at each stage as well. I just am compulsively double checking.

12. DELIVER. If I am done, then this is the stage where I am going to take what I have produced and beginning storyboarding the findings. This is outlining, this is short little papers, this is whatever process I use but the purpose of the stage is to start producing deliverables based on everything that came before it.

13. SIGN. And then once I’m done, I sign off.

I had to develop an insanely rigid checklist because I kept experiencing that “drift” problem I noted, so this checklist is both design principles, but it’s also a reaction to AI Agent drift.

So, within each of those stages, what I’ve been doing is coming up with a series of rituals that have to be done each time to ensure that I have completed the stage. I use a variety of things to do this. For instance, the dashboard has a visual going left to right of circles that are color coded green (closed), red (open) and amber (“you are here”). Each circle goes with one of those 13 stages. And if it’s green, then it means I have gone through a series of steps that has given me a level of confidence that that work is completed and correct. But if it’s red, it means it is not yet closed, and as no project can ever have a red circle, that means the project is also not closed. And then amber, which pulses visually through various hues of yellow-amber, is where I know each day I am when I start the project back up.

Well, in the future, I’m going to walk you through a project using diff-in-diff so that you can see how I’m doing this, but I know that this is just going to be how I do it, as mature researchers all have their own style, their own preferences, their own needs. But I do think “AI Agent drift” is a real problem, and what I want to leave with is a metaphor.

The early church fathers had their own drift in the sense that there were a lot of letters and books floating around. These letters and books were like old files. They had different names, and they had similar sounding names, and they told similar stories, but not identical stories. And at some point, the early church got together and decided on the canon. What was the canon? The canon was the pipeline of code that would produce the exhibits (tables and figures) that would support the final manuscript that the author presented as their own.

They had some process of selection and curation. That process of selection and curation did two things. It chose what would be in the pipeline, and it chose what would be deprecated. They did not use the word “deprecated” though. They used more dangerous sounding words like “heresy”, or if that’s too harsh, then “apocryphal.” These were various gradations as to what would be tolerated.

Well, here’s the thing. AI Agents are going to produce, when used for empirical research, an abundance of files, figures, tables and code. There is something about the process of a lengthy empirical study whereby without a very closely controlled harness, you will end up with a multiplicity of those items. They will have similar sounding names like “The Gospel of Thomas” versus “The Gospel of Mark”. They will tell similar stories — but not the exact same story, and their differences may be benign such that they can both be included (e.g., The Gospel of Mark versus the Gospel of Matthew). But sometimes their differences will be too much (e.g., The Gospel of Mark versus The Gospel of Thomas) and a hard decision will have to be made.

You and I are going to need a criteria and a process and a rule that selects the canon and that process is likely to be an unfamiliar one. It is likely to be an unfamiliar process because prior to AI Agents used for empirical research, I think we were unconsciously curating our work without really knowing it.

I mean, I take that back. Not all the time. It’s a very very common thing to educate young people into coding etiquette and warn of various pitfalls that evolve naturally when certain things are ignored. So it’s very common to see entire advice devoted to what you’re going to name files, how you will organize your directories and version control (see Gentzkow and Shapiro’s classic manual on this for instance). We develop rules and best in practices in response to the technologies and the problems they create for us and the opportunities they create for us.

Well, soon someone is going to write that for the use of AI Agents for empirical research, and when it comes, it will bear some resemblance to those old ways but I think it will very quite bizarre at first to us. But one of the things that it will most definitely have due to the natural tendency of code drift from agents is that it will have to have an intentional process following a close set of steps that will be used to canonize code into a replicable pipeline. Mine is a checklist, using metaphors like locks and colors and maps and practices and hooks, to ensure that not just my future self, but also future Claudes can always pick up where left off and know precisely what is heresy, what is apocryphal and what is canon. That’s the goal. The goal is that the exhibits we end up with are exhibits we chose and that they mean what we think they mean, and the sample does not just drift and change over the course of the project unknowingly because of a forgetful Claude who improvises all the time without awareness.

Read more

On Substack

New on Substack: Subscriber Perks, saved Notes, scheduled Chats, MCP, redesigned podcast player, and more

Whether you’re spending your Substack summer tinkering with your existing newsletter or podcast or launching your first one, we’re shipping features that make it easier to publish, customize, and distribute your work, and engage with your audience. We look forward to hearing what’s most useful and what you want more of. Here’s the latest…

Read more

18 days ago · 1502 likes

If you want to connect Claude Code to your substack, here’s the instructions to do so. But let me lay out what I’ve learned, and how I’m going to learn from what I learned going forward. I had Fable 5 do the analysis for this con…

Read more

Greetings folks. I am back from Europe. I left on May 9th (not the 11th as I previously said) and got back on the 4th of July to Boston. That’s 56 days in Europe across I think I counted half a dozen countries and approximately 10 cities. Quite a romp. I made new friends, saw new places, visited old ones, including Amelia in Sa…

Read more

Read more

There’s a few things associated with it that have …

Read more

Read more

I went to the Mercado de San Miguel last night for dinner at Botin, the world’s oldest restaurant. That part of the city is so interesting and lively. I will explore it more after CodeChella c…

Read more

B…

Read more

Read more

Read more

Read more

Read more

Read more

Read more

Read more

• an introduction and formalization of various causal parameters related to a treatment “dosage” appropriate for the diffe…

Read more

Read more

Read more

Read more