More covariates and diff-in-diff -- this time with a lot of bold and italics!
Okay, so in previous posts I laid out this thing I keep running into. A person will more or less say this:
If I run diff-in-diff without covariates, and find a number, but then run it with covariates and find a different number, this is bad and diff-in-diff is invalid.
It’s a gut check, vibe thing for sure. Of course, though, there is such a thing as “conditional parallel trends” and I gave an example a month ago. My example was assume two groups — males and females — each of whom are on different earnings, Y, trends. But there are two trends in outcomes per group. And they are this:
Y(0) trends. These are the trends in earnings if untreated. Say “high school only trends in earnings”.
Y(1) trends. These are the trends in earnings if treated. So “college educated worker earnings trends”.
See for diff-in-diff, since the target parameter is the ATT, then you have to ask yourself — which of those two random variables are we concerned about? And the answer is whichever one is missing. And for the ATT, we are missing this one:
Missing counterfactual: E[Y(0)|D=1]
We are missing the Y(0) trend, and we are missing it for a population of people who were treated which in this case is the college educated workers.
But for diff-in-diff it’s even more narrow than that. The diff-in-diff recall is a 2x2, or as I like to say “four averages and three subtractions. Well guess what the parallel trends equation is? The parallel trends equation is itself a 2x2. It is interestingly itself “four averages and three subtractions”. In fact, in many cases best I can tell, the calculation to estimate a treatment effect has a selection bias term that is simply the same calculation measured on the missing counterfactual. So when you calculate diff-in-diff:
DiD = Delta Y_1 - Delta Y_0
where Delta Y_1 is “after minus before earnings for the college educated workers”, and Delta Y_0 is “after minus before earnings for the high school educated workers.” Guess what that equals:
DiD = ATT + PT equation
The left hand side is as I said Delta Y_1 - Delta Y_0. But you know what the parallel trends (PT) equation is?
PT equation = Delta Y(0)_1 - Delta Y(0)_0
A lot of people don’t this, but did you know that the parallel trends equation is itself a diff-in-diff? That’s right. The bias of diff-in-diff is another diff-in-diff. It’s just that it’s a diff-in-diff on Y(0), which recall is “trends in earnings if high school only for the college educated worker” minus “trends in earnings if high school only for the high school only worker”.
That’s what makes it a selection bias term. Because selection bias for estimates of the ATT are simply the original calculation, measured in realized outcome Y, but this time for Y(0), which is missing for the treatment group!
But isn’t it interesting still? It’s interesting to me that we say “parallel trends” but really what it actually is simply another diff-in-diff!
Well look at it again close. The bias is differences in average Y(0) trends for two groups. What does that mean? Well, for one, it means this:
Average Y(0) trend for treatment group does not equal average Y(0) for control group
I meant that’s what it means literally. It means the means of two group’s Y(0) trends (after minus before in the Y(0) outcome) is not the same. It does not mean the trends in Y are not the same, because differences. in the trends in Y is literally the diff-in-diff calculation and if there is any treatment effect, then under parallel trends that difference is the ATT. No, this is an imbalance in counterfactual Y(0) trends. Counterfactual because Y(0) — “high school only earnings” — is missing for the treatment group (college educated) the period after they got their college degree because the period after they got their college degree, their Y became Y(1) not Y(0).
So, this is exactly why parallel trends is entirely a question about covariates. Why? Because what are the covariates that are responsible for trends in Y(0) and are those covariates unequally distributed across the two groups.
Apologies for yelling with a bunch of bold and italics but I get excited!
Anyway, so here is my example. All high school educated males earn +10 dollars year to year. They get raises, in other words. But high school educated females earn +8 dollars a year. What did I just write down? Trends in Y(0) for two groups — males and females.
Okay so that’s “heterogeneous group trends in Y(0)”. That is not imbalance. Imbalance is about the college educated workers versus the high school educated workers in Y(0). And if the males are just as equal in the treatment group than in the control group, then that covariate is irrelevant to a parallel trends violation. Why? Because watch. Let’s say 40% of all college educated workers are male and 40% of all high school educated workers are male. Then check this out:
College educated trend is: 0.4 x 10 + 0.6 x 8 =8.8
High school educated trend is 0.4 x 10 + 0.6 x 8 =8.8
And thus 8.8 - 8.8 = 0. Why does that matter? Because the difference in the mean trend in Y(0) in D=1 and D=0 along those dimensions is balanced and thus parallel trends holds.
You know what that means? That means even with heterogenous trends in Y(0) by observable groups, you actually do not need to control for sex. Because sex is balanced and even though there are heterogenous trend by sex, they’re neutered, they’re canceled out because of the balance.
That’s exactly the same reason why in an RCT you don’t have to control for covariates unless the randomization is conditional on those covariates or if those covariates are so deeply correlated with the outcome that including them can reduce the standard errors. My old colleague, Rebecca Thornton, would say that a lot when she would explain this to her students and it just always stuck with me that she was saying it. It’s the same though with diff-in-diff. It’s the exact same thing. Just because you have covariates that are predictive of trends or even cause those trends is not enough of a justification to condition on it in a model.
And in fact, I would caution you from doing that, especially if you are using Callaway and Sant’Anna. Why? Because you have to pay for these covariates. They are not free. There is no such thing as a free lunch. In CS, you often are incorporating these covariates into the model with a propensity score and logistic regressions actually need a fair amount of treated units per covariate. Sometimes as many as 7 or 10 treated units per covariate. Well, in state level panel data, the US only has 50 states in the first place! So how quickly does that collapse you think into the curse of dimensionality and make it such that you really are getting nonsensical coefficients on the logit (which btw are suppressed anyway as output in most of the CS packages I know of so you don’t even see first hand those coefficients anyway).
But let’s say you use regression adjustment. Well, just know that if you’re using regression adjustment, you don’t really get out of jail with the curse there either. The curse is always there. It’s just that if you use covariates in OLS specifications, you’re going to be identifying off the functional form because if you have the curse of dimensionality happening, you’re going to be projecting — OLS recall is the best linear predictor — into a space where there is no data. So you better be pretty confident about that regression specification and its functional form because it’s going to be imputing the Y(0) trend for the treated group off the Y(0) fitted trend for the control group. Dropping even one polynomial or one interaction will be technically an incorrect specification and can land you anywhere — especially in 2026 when no one is willing to relax a belief in heterogeneity.
So, what then do we do? Here’s what we do.
We check if the covariates that we think are causing the Y(0) trends are balanced for the treatment and control group at baseline, and then
We estimate a propensity score, plot the histogram, check the distribution, see if there is both overlap and if the max value of the propensity score in the control group is almost 1. Because if it is almost 1, then the inverse probability weight will explode.
Why? Because the weight in a diff-in-diff with inverse probability weights, which the original Abadie 2005 Restud had (“Semi Parametric Diff-in-Diff”) and the Sant’Anna and Zhao 2020 and Callaway and Sant’Anna 2021 also have, is equal to p(x)/[ 1- p(x) ], and that weight only applies to the control group in diff-in-diff. Why only the control group? Because, we are estimating the ATT, and nothing is wrong with the treatment group’s outcome. We are only missing the counterfactual for the treated group, and we are using the control group to get it as the control group is the only group that has the Y(0).
Well, check out what happens to that weight if the propensity score for the control group is “almost 1”. Assume it is 0.999, which you can easily get with a lot of heterogeneity in those Xs, weak to no support for some of them, and a large sample of control units. 0.999/[0.001] = 999.
What’s 999? That’s the weight on a single unit’s outcome. A single unit’s first difference outcome in diff-in-diff using IPW will become that outcome times 999. That is an outlier, it has massive leverage over the estimate. And you need to check it! You need to know if it’s there because that single observation could literally flip the sign or drive it to who knows where. What if that one unit is Elon Musk!
We talk about this in section 4.2 of our Journal of Economic Literature. I encourage you to read it closely. I’m still not quite ready yet to go through my empirical example to show you conclusively in my opinion about how to convincingly show you, without a simulation mind you, that correcting for covariates matter, but also how you correct for covariates matters just as much, but I am going to. Just wanted to get that rant out of my system first!