Mixtape University: Diff-in-Diff with a checklist. Simulating the Importance of Weighting

One of the perks I’m creating for paying subscribers is a library of short videos where I teach causal inference.These are 15-minute (more or less) videos that aim to build over time into a helpful resource. If you’ve heard me teach before and wanted a quick refresher on a concept, that’s what these are for.

Right now, I’m taking a checklist approach to teaching difference-in-differences. The jury’s still out on whether that’s the best pedagogical move, but I’m giving it a go.

In the last couple of videos, I focused on the role of weighting. But rather than just walking through formulas, I tried to show how the aggregation already present in a dataset can influence the parameter we estimate. The most relevant videos are here:

I’ve been thinking about this for a long time—ever since our paper, “Difference-in-Differences: A Practitioner’s Guide.”In an earlier version of that paper, we discussed how different weighting schemes could change the interpretation of the ATT—from being about the “average person” to being about the “average county.” That distinction stuck with me.

But another idea that kept resurfacing came from years of teaching potential outcomes in Excel spreadsheets. When you use spreadsheets, it becomes really clear that average treatment effects are just averages over units. If the unit is a person, then the average treatment effect is about the average person. But what if the unit is a county or a state—and those units are aggregated from an underlying person-level dataset?

Finally, I’ve always felt a bit uneasy about how people write about population weights in panel regressions—especially when those weights are added to make things “nationally representative.” That never quite sat right with me in this context.

So, I wrote up some new slides that start from step one: define the target parameter. From there, I built a simulation to explore when heterogeneous treatment effects and Roy-like sorting into regions could make the same dataset—aggregated in different ways—produce different estimates.

Today, I’m sharing video simulations as well as the code in Stata, R, and Python that walk through exactly when different weighting schemes lead to the same answer, and when they don’t. I think it’s pretty interesting, honestly. Remember, people wanting to learn more about this can also read this great 2015 JHR by Solon, Haider and Wooldridge, “What are we weighting for?”

Thanks again for being a subscriber. It means a lot.