The Mixtape Mailbag #2

Guest starring Brantly Callaway!

Creating this picture took me around 20 attempts with Dall-e 3. I started off trying to make a split screen postcard of me and the person who wrote me, MM, from France, with me writing under a tree in Texas near my Volvo with a sleeping cow nearby and her at a cafe in Paris. I’m really trying to push this Volvo/cow/tree/Texas branding if you haven’t noticed so far. But usually I just would end up with the Volvo being under the Eiffel Tower and MM drinking coffee with a cow asleep by her in a chair. So I decided to strip out the nonessentials, take me out of it completely, and just go with someone writing a letter in France. True, it’s going to somewhere either in Colorado or California, but it got to me anyway, no worries. Welcome to week 2 of the Mixtape Mailbag!

This week, an economist from France wrote to me about a problem that she and her colleagues were having on a revision for a journal using diff-in-diff. The editor and referees requested something that seemed to alter their results. Below will be the letter from this economist (“MM”), followed by two responses, both behind the paywall. I’ll post my response, and then I’ll report a response by diff-in-diff expert Brantly Callaway from the University of Georgia of the famed Callaway and Sant’anna estimator. I wrote my response and posted it having never read his response (and still haven’t) so that I wouldn’t be influenced to change literally every single thing to be just like what Brant said. It’s the real blind taste test! Pepsi versus Coke! So get ready for some serious advicing!

---

Dear Scott,

I am sorry for contacting you out of the blue but my co-authors and I are currently stuck with an empirical problem/question that we do not know how to properly solve.

We are in the midst of revising a paper for a journal in which we run a DiD analysis. At the moment we face the following problem: If we run the analysis in the baseline setup we do not find any evidence for a violation of the parallel trends assumption (neither visually nor when running more tests). However, the reviewers have asked us to run a subsample analysis of our treatment group and we now face the following problem: If we split our treatment group into two subgroups, both subgroups violate the parallel trends assumption - but in exactly different directions. Hence, the effect seems to be cancelled out if we look at the two groups in aggregate (which is our main treatment group in the paper). Now my question is: Is this a problem?

Both, in our theory as well as the baseline analysis we refer to the aggregate treatment group. We are currently not sure how to report and comment on all of this.

Getting your opinion on this would be extremely insight- and helpful.

Thanks so much in advance,

best

MM