Mixtape University: Diff-in-diff with a checklist. Why Can't I Use an Already Treated Group as a Control?

Four videos, Including a video of me using ChatGPT with only the microphone to generate a simulation of the problem of using already treated as control in Stata, R and python!

Each Sunday, I post two new videos as I continue migrating material from my workshop over to the Substack. Lately, I’ve decided to give the series a bit more structure—both for clarity and for teaching. So going forward, I’m organizing the material around something I’m calling “the checklist.”

The idea of the checklist started with a lecture slide Pedro Sant’Anna used during a talk at Amazon, but it’s also rooted in other sources—especially the paper I coauthored with Andrew Baker, Brantly Callaway, Andrew Goodman-Bacon, and Pedro: “Difference-in-Differences: A Practitioner’s Guide.” Since that paper is designed to be a real-world guide for applied researchers, I’ve decided to build this video series as a walk-through of the paper itself.

This week, I recorded four videos. I only meant to do two—but I talked too long in one of them and hit 28 minutes, so I broke it into two parts. The third and fourth videos were also a little long, but easy enough to skip through if you feel it’s to long. If you’re not a subscriber, here’s what you’re missing. And if you are, here’s what’s new.

The Baker et al. paper has its own pedagogical structure, built around the idea of “reverse vs. forward engineering.” That framing is at the heart of the first two videos. I explain what it means, using examples from the literature (like Imbens and Angrist’s 1994 LATE paper), and also an analogy: training for and running a marathon. In that example, planning a training schedule and then executing it is like “forward engineering.” In contrast, looking at what you did after the fact and trying to infer the logic behind it? That’s “reverse engineering.” The same distinction applies in causal inference—and especially in DiD.

I think it’s one of the most useful and novel contributions of the paper. And I’m hoping that by walking through it across several short videos, the repetition will help make the idea stick.

In the third video, I return to what I’ve been calling the “reverse engineering of 2x2 designs”— the basic four-averages-and-three-subtractions logic of DiD. Previously, I showed two cases:

• Using a never-treated group as a control, with no anticipation

• Using a never-treated group as a control, but with anticipation violated

This time, I walk through a third case:

• Using an always-treated group as a control, with no anticipation

In each of these, I write out the 2x2 setup, I then plug in potential outcomes, and then I walk us through the substitutions and rearrangements until the expression is written in terms of causal effects and bias terms. It’s not a big deal—but I think it’s valuable, and the better you are at it, the more intuitive the diff-in-diff literature and its myriad estimators become.

And the fourth video is weird, but I was curious if I could pull it off — I used the microphone and had Cosmos create a simulation in Stata and R illustrating the reverse engineering that I did in the 3rd video. The basic setup was a simple 2x2 with and without constant treatment effects, satisfying no anticipation and using an always treated group as a control.

Thanks again for following along and supporting this work. I’m grateful to those of you who keep showing up, month after month. It’s a real privilege to share this material with people—wherever you are in your journey—and I hope these new videos help make things clearer, more practical, and a little more grounded.