Mixtape Mailbag: Why is My Simple ATT Significant, but Not My Event Study?

Mixtape Mailbag Returns!

Mixtape Mailbag Returns!

Welcome back to the old Mailbag series! That was a long hiatus, but I want to bring this back now. The Mixtape Mailbag is where readers write in with problems and questions, and I take a stab at answering them. The Mixtape Mailbag is one of the perks I provide for paying subscribers, and today, I’ve got a letter from Baffled in Bootstrapland (all names have been changed to protect the innocent, and I had Cosmos rewrite the question) concerning a paradox she feels like she saw in Callaway and Sant’Anna where her simple ATT was statistically significant, but her event study coefficients weren’t. I discuss it, review the method and what it does, and end with a simulation.

---

Dear Scott,

I’m working on a panel data project with firms that enter treatment at different times (differential timing). I’m using the Callaway and Sant’Anna estimator to estimate treatment effects, and I’ve run into a puzzling issue.

When I estimate the simple ATT, it’s statistically significant. However, when I look at the event-study estimates, none of the individual post-treatment coefficients are significant. Given that the simple ATT is aggregating the same underlying ATT(g,t) estimates, I would expect at least some of the event-study coefficients to be significant as well.

What could be causing this? Am I missing something about how power works in this setting?

Thanks for your help!

— Baffled in Bootstrapland

---

Below the paywall, my paying subscribers will get an answer! Or at least a stab at one! Thanks for your support, and I hope to bring this series back more regularly—so feel free to write me with your questions at causalinf@mixtape.consulting!