Which research mistakes matter for human capital?

And does it matter?

I had a coding error once in a paper that I only discovered during the R&R stage. The second I saw it, my stomach sank. I immediately recognized the kind of error — a classic Stata one — and pretty much could see how bad it was going to be for me. And sure enough, fixing it changed the results from significant and interesting to not significant and not interesting.

I never did resubmit that article. I never even attempted the paper again. For a long time, it was actually still sitting at the journal, and when I’d go to submit a manuscript there, I’d see it with a clock ticking saying they were waiting on my resubmission after 10+ years.

I learned a lot from that experience. I learned everything from the narrow to the metaphysical. Interestingly, as I already knew the correct syntax, I didn’t really learn the correct syntax from it. It just was stray and stale code from a cleaning file that I had stopped running, but which I’d written in what might as well have been day one of the project, which was also my very first empirical project. But sometimes that is what you do in fact learn. You learn something specific like “this is the correct equation for calculating standard errors, not this other one.”

But that’s not really what I learned. I learned things like having well organized folders. I learned about having folders for the raw data separated from the cleaned data. I learned about having code that ran in a pipeline that would automate tables and figures. I learned about the importance of version control. I learned about the temptations around “liking my results”, which I grew to consider a very dangerous mindset. I learned about scientific integrity, a norm that had not occurred to me before the coding error, and how much I wanted to figure out the structure that would protect me from losing my integrity.

There’s two parts of me when I think about Claude Code and that error. The first part of me says that I don’t think it is good for a project that we have to go through such ordeals. It’s a huge waste of time, it leads to false beliefs, and puts errors into the scientific record because for every one of these that gets caught and never submitted there are twenty that didn’t get caught, were resubmitted and were ultimately published. So if AI Agents can find those errors, such that we avoid such a counterfactual outcome, wouldn’t we want that? And in the aggregate, it would seem so, and I put that on the pro side of the Claude-Code-for-science ledger.

But what about the personal growth part of it? What role do mistakes play in human capital accumulation for the empirical researcher? And is that the only way to get to that point? I learned valuable human capital from that experience. Stuff that cannot get taught in classrooms, stuff that cannot easily be apprenticed through mentorship and coauthoring. I learned about scientific integrity, honesty, temptations, dangers, coding etiquette, the importance of variable names, directory names, file names, the importance of version of control, when not to use one’s fingers to write down numbers and when to use one’s fingers to write down numbers.

See, the first one — avoiding mistakes — really has do with the output of the task. We make the output as clean as we can because the output is valuable intrinsically.

But the second one is about human capital and personal growth, and if scientific progress depends on accuracy at the moment and human capital accumulation in the future, then it’s not clear to me — do we want Claude Code to remove the ability to make mistakes in research? It’s like when I would play poker for actual stakes with real money, versus playing poker with no stakes and pretend money. I would play differently depending on the stakes. Sometimes with pretend money I had the courage to go all in.

But once I felt like I was pretty sure I had the better hand playing live against a guy was sitting behind a massive stack of chips. It felt like I was looking at the giant wall of ice from Game of Thrones, only made of clay chips, making my $500 in chips appear pretty inconsequential. But I’d grinded all night to get from $100 to $500 and I thought I had the better hand, partly because I’d seen him move all in several times and win by luck or force folds. And now he was doing it to me. On the turn, he went all in. And I just couldn’t call. So I folded, and realized that if I wasn’t willing to call with the right hand because of my fears of risk and variance, then I shouldn’t be in that seat.

Stakes really change one’s behavior. And you learn from bad beats. And you learn from good beats too. You learn because of the stakes.

And I guess I just wonder sometimes am I “better” for having found that coding error? For making it? I mean, I have many such events, and coding errors rank pretty low in that ladder of them, where mistakes and bad events in my life have taught me so much that I almost feel grateful for them. I mean, how can you be grateful for traumatic events in one’s life? And yet I do often feel that way, almost gratitude that I got to go through that and that I’m alive and have more compassion and curiosity about myself and others than I had going in.

Of course, that assumes a positive causal effect as opposed to natural aging, but put that aside. My point is simpler — when do we want machines to stop us from making mistakes? And is it good to literally never make them? And I don’t know. I mean that — I literally cannot seem to figure out the proper framing of what I feel as I know it’s important we don’t make mistakes. It’s why we wear seat belts. But I also know that you can have all threat of pain removed and as such become so lazy and haphazard that you just never gain the insights that can only be gained through that type of failure-making-knowledge event.