Designing difference in differences: lessons from concealed carry laws in crime

A personal journey into the data, methods, and findings of county versus state level analysis using the uniform crime reports

Introduction

Welcome to a new series about designing your difference-in-differences studies with an explicit “design stage” and accompany checklist. I’m going to in this post discuss the research question and datasets that will serve as our running example. And I will also review a few of the papers associated with it. The series will be focused on the literature regarding estimates of the causal effect of concealed carry and right-to-carry laws on crime.

This question has captivated me for years, not just because of the policy implications, but because I decided to become an economist, and decided to study at the University of Georgia from 2002 to 2007 because of reading an early paper on the topic. I have not published on guns, and it’s not part of my future research agenda, but I have a fondness for one of the coauthors in the original study as he was a Gary Becker student, who was a major influence of mine too, and he was also one of my advisors in grad school — Dr. David Mustard. So in some ways, my decision to look back at his early work is an homage to a man who invested a lot in me, and whose paper inspired me on the path I took in life.

But for the purposes of this series, I am interested in it for other reasons. This series is about the “design stage” in diff-in-diff. It is about an explicit stage of research after the collection of data and before estimation itself. It is separate from the literature review, and it has nothing to do with writing. It is unlikely that this material would or should ever be discussed in an econometrics class. My hope is that by using a study that has been the subject of debate, by using a dataset that has been separately too the subject of debate, it will jar the reader to pay attention to every detail.

But there is another reason I am going to focus on the right-to-carry question. I have a particular pedagogical belief based on my readings in college as an English major by the Russian literary theorist Viktor Shklovsky. Schklovsky noted that art’s purpose was to “defamiliarize the familiar”. When it comes to debates about guns, it is like public policy debates about abortion — it is very polarizing, people already have many priors that dominate all of the discourse, and any dissension can alienate people. The debate is “familiar”, in other words. My substack may not be art, but the hope is that the decisions I make can “defamiliarize the familiar” elements of the debate. I will be focused on the disputed dataset, the county-level UCR data, so as to jar the reader and myself into constantly getting in touch with the arguments I am making. I am also not presenting any of my analysis as correct with regards to policy conclusions; I am just going to argue that it is correct with regards to the series of steps I am proposing for the design stage. The points I will be making are largely inspired by and my interpretation of Don Rubin’s classic “Design Trumps Analysis” and so my focus is decidedly not on what is the correct finding. Rather, my focus is on what constitutes “design” and what constitutes “analysis” and I will be noting that there are a range of open decisions researcher teams must make — they just need to be aware of them and transparent about them.

This series will also serve as a behind-the-scenes preview of the upcoming second edition of Causal Inference: The Mixtape. With new material on causal panel models and difference-in-differences, this edition reflects years of teaching and learning alongside you, my readers. Through these posts, I’ll share the tools, lessons, and stories that shaped this work. For subscribers, you’ll get early access to code, templates, and case studies—including the concealed carry analysis we’re embarking on now.

Let’s start with something small but fun: the randomization method behind this paywall post.

The Coin-Flip Experiment and Paywall Policy

I recently decided to paywall some of my Substack posts—but with a twist. Instead of arbitrarily choosing which posts go behind the paywall, I’ve left it up to chance. Specifically, I used a simple coin-flip randomization to decide.

Here’s how it works: I flip a virtual coin nine times. If it lands on heads more often than tails, the post gets paywalled after the first few paragraphs. If tails wins, the whole post remains free. Why nine flips? Because it’s an odd number, ensuring there’s no tie.

Here’s the Python code I used to generate the results, along with the output from this very post:

import random
import matplotlib.pyplot as plt

# Flip a coin 9 times
results = [random.choice(["Heads", "Tails"]) for _ in range(9)]

# Count the occurrences of Heads and Tails
counts = {"Heads": results.count("Heads"), "Tails": results.count("Tails")}

# Plotting the results
plt.bar(counts.keys(), counts.values())

plt.title("Results of 9 Coin Flips")
plt.ylabel("Count")
plt.xlabel("Outcome")
plt.show()

# Display the results
results

For this post, the coin landed on heads 5 times and tails 4. So, paywall it is. This small experiment serves as a playful way to randomize access while also showing how simple tools can add transparency and fairness to our decisions—values that carry over into the research process itself.

If you’re ready to dive deeper into the weeds of observational research—or if you’re just curious about where this journey will take us—consider subscribing today. This is a chance to join a community of readers and researchers who care about evidence, rigor, and, yes, even a little randomness in the process. What follows will be a discussion of the set up of the county versus state level concealed carry data sort of the intricacies around that as well as a description of the design based checklist.

My Connection to the Data

I first encountered the debate over concealed carry laws and their effects on crime during an unexpected turning point in my life. I was working as a market research analyst in New Orleans, freshly out of college, and spending my nights devouring articles and papers to understand human behavior better. My job was purely qualitative—I conducted focus groups, observed patterns, and wrote reports—but I wanted more rigor, more structure to explain why people did what they did.

Late one night, I stumbled onto Gary Becker’s Nobel Prize lecture. Reading his work was like turning on a light in a dim room. He wasn’t talking about stock markets or GDP—he was using economics to explain family dynamics and crime. His approach was grounded in utility maximization, the very same framework I’d been instinctively using to understand the behaviors of the focus group participants in my day job. For the first time, I realized economics could explain human behavior in ways that felt both precise and profoundly human.

Not long after, I found John Lott and David Mustard’s 1997 paper on concealed carry and right-to-carry guns legislation and their effect on crime. Their study used county-level FBI Uniform Crime Reports data to argue that these laws reduced violent crime. The idea was controversial, the evidence provocative, and the debate intensely polarized. But what grabbed me wasn’t just the argument—it was the data itself. David Mustard, as a grad student at Chicago in the early to mid 1990s, had made the Herculean effort to assemble the dataset from microfilm and paper reports and put it into a panel form that could be used for program evaluation purposes. The data ran from 1977 to 1992, and given there are today around 3,000 counties, and there were 8 crimes recorded for each county-year, it was a massive recording of data. The dataset was well put together, and I think has helped us better understand some of the issues with the UCR program’s own data collection methods at the county level.