How to improve understanding with Claude Code

Paul Goldsmith-Pinkham posted on his substack feed the other day a link to this interesting blog post by Geoffrey Litt called “Understanding is the new bottleneck”. I encourage you to read it. It’s short but sweet as the old kids used to say.

I thought it was interesting because it edged me a bit towards a bumpersticker-style thing I’ve been saying the last six or so months, which is that “verification is the new bottleneck”.

“Verification is the new bottleneck” is a sentence and type of rhetoric you see everywhere in which a writer posits that the demand for research output is downward sloping at the individual level. Before AI agents, the marginal cost of producing research was some amount that determined research quantity demanded. I’ve drawn below two pictures representing how I think of this shift in research output, which is self-explanatory and intuitive to economists, but may be helpful to walk through for non-economists.

The first graph is the old equilibrium determined by the marginal cost of producing research which I represent as a horizontal “constant marginal cost” for simplicity. At this old marginal cost of production, MC_b for “before AI”, we produce an amount of research up to the point where the marginal benefits are equal to the marginal cost, most likely fixed by time and skill. To do more would require more human capital and/or more time, and given the scarcity of both, the researcher would not do it.

But now the second graph. Now with AI agents, a few things happen. First, I and others argue that the production of research increases under AI agents because the marginal cost of producing research fell from MC*_b (before marginal cost) to the much lower MC^hat_AI which is at some arbitrarily small amount above 0 equal to epsilon.

I also put surpluses in this graphic though I am vague about just what “research surplus” means precisely, mostly because I’m not really clear myself. The old surplus was B, and it constituted I think the subjective net benefits of the research. But under the lower marginal cost, we gain not only more research (the elasticity of which depends on the dimensions of the demand curve in this case), but also additional surplus too. Specifically, the old work we would’ve done is done with fewer time inputs, and thus the old work now has net benefits of B+AI_1, which is the area under the demand curve up to Q*b. But as research output rose from Q*b to Q*_AI, there is additional surplus generated from the new work equal to AI*2.

The thing that I am unclear about is this, though. In principle, all of us have additional time from this shift that we now get to reallocate to other tasks in our lives. What that is or will be is shaped by heterogenous preferences.

But one thing that existed, at least in my life, was that the subjective quality of the old work has changed. By subjective quality I mean that the old work, Q*b, had two qualities inside that triangle B. The first quality, was that it was produced at all, which is revealed by Q*b. That it existed at all under the human mode of production means that it was created — a statement that admittedly does seem tautological, but I think it’s worth saying anyway. Q*b was created by humans because humans created it, which leads directly to the second point which is that the net benefits of B was that it was the (net) value of that output to the researcher consisting both of its existence (production) but also something a bit slippery which I think is just the innate understanding of the work itself that happens almost by accident. I think of it as a kind of Hobbs-like squatting by workers wherein the sheer act of working to turn the natural earth into something else created ownership. There was, in other words, almost like ownership to the output. Not so much ownership as in no one else could have it so much as a type of mental ownership. I made it, and therefore I knew it. Somehow the production of the thing and the understanding of the thing existed simultaneously, as you could not make cognitive output without understanding it too as the understanding of it was usually a prerequisite to doing it, and also the byproduct of making it at all.

But now, with AI agents producing, what even is B? Is it the same B as before or has even the new B changed to something else — like a B’? This is where the geometry of this starts to break down. I don’t think it is the same B as before personally because there is not there the same kind of Hobbs-like squatter rights over it since we were not the ones who made it. By which I mean the epistemological function of work itself is not present. The latent understanding of the work has gone away because understanding may not be produced indirectly as a byproduct when we are not doing it.

Now it can be, 100%. Lab managers understand by proxy the work when interrogating and working closely withe the coauthor and RA who made it, but often that is not the case. Primary authors will be held accountable, for instance, when their RAs make mistakes and they did not do the due diligence to be on top of it. That has happened before. We can all think of famous cases over the last 10+ years or so in which people were more or less deceived by duplicitous coauthors who had fabricated data outright. I think B was in other words highly contingent on trust with a coauthor, historically anyway, even with managers. They would more or less free ride on the trust of the coauthor or RA’s own subjective understanding of the work so that B could exist even in the minds of the non-producer simply because they trusted the producer.

But put that aside, because now my point is that we have two new surpluses and that is AI1 and AI2. And I think in some way, these two new surpluses represent the additional gains subjectively that happens as a result of taking their hands off the wheel to some degree. It is gained subjective net benefits and it is gained subjective net benefits coming from manager-like trust that the work done was done well even if they did not do it, and even if they do not understand it.

I think for some, it is going to be the principal-agency problem, though. It could be the blind leading the blind. I trust the agent did the work well, but as the agent is not real, we do not actually know it was done well, and if our skills atrophy, we may not even possess the same comprehension of what it means to be done well.

Well, the question I have in my mind is that couldn’t we get there? I mean we have so much more time, don’t we? Couldn’t we just reallocate the time savings back to the verification task of that new and old material created and it become real surplus, something we both experience as positive net benefits and which is actual positive net benefits? That is, we experience it as valuable because we think it is correct, and it is actually socially valuable because it is correct. I think those might not be the same thing. I think they may be different.

See, the thing I keep thinking is that it is an assumption that the time savings we now have in our possession are even capable of being used to fill in those areas at all. We don’t know that it is. We don’t know, I don’t think, if the time savings we get from AI agents can be allocated to fill in that entire B+AI1+AI2 with subjective understanding. What if it’s not? What if the old way of understanding something was primarily an accidental consequence of the production process. Pollution is like that, for instance. Pollution is the byproduct of production. It’s an externality. What if understanding was always an externality — something which was created almost by accident when one set out to make some cognitive output using time inputs?

So I think I agree with this idea that it is not really the most accurate way to describe the situation we are now in to say that “verification is the new bottleneck”. I think it is more accurate to say “understanding is the new bottleneck” because it’s entirely possible that AI agents will be able to verify as well as produce, but that does not therefore mean we will understand it.

So I have been thinking a lot about this essay by Geoffrey Litt, as well as this idea Paul had put forth which is the unit of verification is the git diff operation. I now use the git diff operations like Paul suggested, but I am noticing that I am skimming and accepting them creating what Paul and others call cognitive debt — only it’s supposed to be that the cognitive debt is cleared by the sheer act of checking off those diffs. But I think I’m actually just silently accepting them without full understanding. That does not mean everyone else is so much as it means I am, and if you sense sometimes that you are doing the same thing over and over habitually, it is time to consider if perhaps you should try something different. Maybe not instead of the diffs, but in addition, where the goal is now two things, not one.

1. Zero Errors Remains the Constraint. I remain convinced that we must say to ourselves and one another that “zero errors is the constraint”, and not the objective function. Our goal should not be to minimize errors, because if you say that, then you will quietly come to accept the idea of “optimal errors”. And that cannot be what we move towards given the aggregate supply elasticities of research production with respect to prices is probably greater than 1. I suspect in the long run it is anyway, even if in the shortrun, due to fixed inputs, it is small.

2. Understanding is the constraint. Is understanding now the constraint or is it also the goal? I am thinking a threshold of understanding the work must become the constraint too. Paul usually quotes IBM to this point, at least indirectly: since machines cannot be held accountable for mistakes, they cannot be trusted to make decisions either. They either work to help us understand the work they’ve done, or we do not use them at all. I think it’s possible those are the two corner solutions.

3. Research output is the objective function. So then what is the objective function? It’s whatever we think is optimal in equilibrium but not perfected, and I think that could be the idea that we are trying to reach optimal research output, and optimal research output is the point where the social marginal benefits equal the social marginal cost, which requires verification and understanding.

So, what am I doing differently than git diffs. I am experimenting with a new skill called /quiz. It’s an extension of things I was already doing which were interviews used to extract my beliefs about which covariates to use and what the target parameter should be in situations where I could not really be sure, but I suspected I did know deep down enough to say something. But now what I’m working on is a little stranger of an interview concept.

My /quiz skill has two components. First, I am working on Claude Code creating a set of slides when invoked that teaches me something. And then I have Claude Code ask me multiple choice questions, one at a time, based on it. You know where I got the idea? The training videos that HR makes you do all the time at work. That’s how they do it — they give you something to read, oftentimes which includes a video, and then they give you a quiz. That’s where I’m going. I am going towards more or less “training video quizzes” to constantly create in me continuous understanding. On top of verification. On top of my detailed staged checklists.

We’ll see how this goes. I don’t know if I am over-engineering the production process using AI agents, or if it is optimal, but I am definitely moving towards something that is taping my hands to the wheels. I am tying myself to the mast to drift through the strait so that I do not go insane by the screaming of the harpies in the water, so to speak. There is simply too many things that can go wrong, and I for one do not get excited by the idea that a sniper will come along after me and find my analysis is just riddled with problems. The current environment in which problems in one another’s work is treated as scalps to be taken and pounds of flesh to be removed is far too antagonistic for my tastes. I am not interested in becoming helping someone get their 15 minutes of fame on a podcast as they discuss the problems with my work to be frank. But I also just want to be my best self. I want to continuously learn. Learning is why I got into this. I am, deep down, not really a competitive person so much as a romantic about learning, and if agents can help me learn in my creative work, I want them to sit in the cockpit with me. But if they are going to be auto piloting while I sleep, then my truest self would prefer I not do that as until the lights go out, I remain in love with creativity and creative tasks. I enjoy it, it motivates me, it fills my heart with lightness and joy, and my mental health desperately wants it.

So that’s all. I just wanted to share these thoughts. Now back to work trying to wrangle these agents into doing what I say even if they, in a very nice style, stumbling and mumbling as they do it.