I think it’s weirdly MORE important with Claude code not less. I wouldn’t have said that a few months ago. But now I just feel like I’m in constant “code drift”.
Not necessary weird if you consider that Claude Code itself is a harness built on top of LLMs, and harnesses are software engineering implementation of the general checklist manifesto idea.
Nice. Checklist manifesto is my general motivational inspiration as well (I've been recommending it to MBAs in my AI, Business, and Society class for a while as a key reading on how to work with AI). A parallel (to economics) domain implementation of these principles in software engineering is spec driven development (which is basically about how you build the harnesses you wrote about recently) -- also a key inspiration for me.
The missing thing in those is how to pick objects whose states you track with checklists. Domain specific objects don't translate between domains. They aren't always immediately obvious to people in the domain either, because a lot of this is "things that are left unsaid because everyone knows them" and it just feels not right to make them explicit. But the underlying principles for choosing the objects are same between domains. This is where I get inspirations from Category Theory (category theory for sciences and category theory for programmers are great intros, and the latter also helps see the connection between functional programming, checklist writing, and how to harness AI in econ).
I know. All of this is WAY EASIER said than done. I feel like Luke skywalker training with the helmet and the dark visor down. I still can’t seem to fully articulate my own problems, and I honestly think my problems are mostly unique to my workflow which I’m unwilling to change on in its fundamentals. But I’ll get there soon I’m sure.
I know, right??!!! 😭😭😭 Even with checklist manifesto, which is a phenomenally well written book, I feet quite dense when I can't write a good checklist on the spot, or even articulate when a thing can be a checklist box and when it can't. But while our problems feel unique for us, I do have a hunch that category theory does imply with mathematical certainty that they are not (as in, I do think there are isomorphisms from "our problems" to well defined structure representing general problems, though don't push me on what either the isomorphisms or the general structure are exactly!).
I feel like I have to somehow figure out the core concept of research that is exogenous, to all technologies, so that I can figure out what works best for me here. I think it has to have a checklist. Once I get this deck worked up on this I’ll post it so you can see it but I just know it’s there.
Yeah, I also feel like that's the right direction. Also, note that "core concept of research that is exogenous, to all technologies" sounds like something that should be described using category theory (which was kinda invented as a way to be a language of math that describes other fields of maths -- making it possible to do mathematical examination whose objects are fields of math!). Category theory for scientists book makes some progress on describing category theory as unifying language for sciences. And checklist is a very category theory friendly notion, not in a sense that it tells you what checklists to write, but in a sense that it can help define precisely what can and can't go into a checklist, or what is a valid checklist and what isn't (before even looking at specific things you put into one!).
I sent you my recent slides in an email, I try to do something broadly similar there, but instead of checklist I have "spec" as a fundamental object. But key elements of a spec are "what to verify" and "how to verify" -- boom, that's just "what becomes a line in a checklist" and "when can you mark a checklist line as complete".
Thanks, Scott. My take on all of this is, why not have the computer do this in real time and pause as it completes each step so you can view what was happening? Sort of like watching a student RA walk you through everything they did.
Then the verification happens in real-time and you don’t have to worry about a kooky bug blowing everything up.
I’ll bet if you had it walk you through the csdid install, it would’ve shown you the github fetch and then you could’ve know that it grabbed it from GH instead of CRAN, which then would’ve saved you debug time.
I might be wrong on all of this, but I plan to implement this type of approach as I continue working with AI models in research. Don’t write all the code without me, walk through it with me at a speed where I can grasp what’s going on so that I can be confident that what you’re doing is what I would’ve done. It’s still faster than doing it myself, but it saves loads of time at the verification phase.
Reading this late at night is quite exhilarating and I definitely needed it. Thanks for the book req too!
I think it’s weirdly MORE important with Claude code not less. I wouldn’t have said that a few months ago. But now I just feel like I’m in constant “code drift”.
Not necessary weird if you consider that Claude Code itself is a harness built on top of LLMs, and harnesses are software engineering implementation of the general checklist manifesto idea.
profound. the relevant scope is of course much much wider than econometrics.
Nice. Checklist manifesto is my general motivational inspiration as well (I've been recommending it to MBAs in my AI, Business, and Society class for a while as a key reading on how to work with AI). A parallel (to economics) domain implementation of these principles in software engineering is spec driven development (which is basically about how you build the harnesses you wrote about recently) -- also a key inspiration for me.
The missing thing in those is how to pick objects whose states you track with checklists. Domain specific objects don't translate between domains. They aren't always immediately obvious to people in the domain either, because a lot of this is "things that are left unsaid because everyone knows them" and it just feels not right to make them explicit. But the underlying principles for choosing the objects are same between domains. This is where I get inspirations from Category Theory (category theory for sciences and category theory for programmers are great intros, and the latter also helps see the connection between functional programming, checklist writing, and how to harness AI in econ).
I know. All of this is WAY EASIER said than done. I feel like Luke skywalker training with the helmet and the dark visor down. I still can’t seem to fully articulate my own problems, and I honestly think my problems are mostly unique to my workflow which I’m unwilling to change on in its fundamentals. But I’ll get there soon I’m sure.
I know, right??!!! 😭😭😭 Even with checklist manifesto, which is a phenomenally well written book, I feet quite dense when I can't write a good checklist on the spot, or even articulate when a thing can be a checklist box and when it can't. But while our problems feel unique for us, I do have a hunch that category theory does imply with mathematical certainty that they are not (as in, I do think there are isomorphisms from "our problems" to well defined structure representing general problems, though don't push me on what either the isomorphisms or the general structure are exactly!).
I feel like I have to somehow figure out the core concept of research that is exogenous, to all technologies, so that I can figure out what works best for me here. I think it has to have a checklist. Once I get this deck worked up on this I’ll post it so you can see it but I just know it’s there.
Yeah, I also feel like that's the right direction. Also, note that "core concept of research that is exogenous, to all technologies" sounds like something that should be described using category theory (which was kinda invented as a way to be a language of math that describes other fields of maths -- making it possible to do mathematical examination whose objects are fields of math!). Category theory for scientists book makes some progress on describing category theory as unifying language for sciences. And checklist is a very category theory friendly notion, not in a sense that it tells you what checklists to write, but in a sense that it can help define precisely what can and can't go into a checklist, or what is a valid checklist and what isn't (before even looking at specific things you put into one!).
I sent you my recent slides in an email, I try to do something broadly similar there, but instead of checklist I have "spec" as a fundamental object. But key elements of a spec are "what to verify" and "how to verify" -- boom, that's just "what becomes a line in a checklist" and "when can you mark a checklist line as complete".
I get that you have a unique writing style but I couldn't actually understand this post properly it was that strangely written.
Thanks, Scott. My take on all of this is, why not have the computer do this in real time and pause as it completes each step so you can view what was happening? Sort of like watching a student RA walk you through everything they did.
Then the verification happens in real-time and you don’t have to worry about a kooky bug blowing everything up.
I’ll bet if you had it walk you through the csdid install, it would’ve shown you the github fetch and then you could’ve know that it grabbed it from GH instead of CRAN, which then would’ve saved you debug time.
I might be wrong on all of this, but I plan to implement this type of approach as I continue working with AI models in research. Don’t write all the code without me, walk through it with me at a speed where I can grasp what’s going on so that I can be confident that what you’re doing is what I would’ve done. It’s still faster than doing it myself, but it saves loads of time at the verification phase.