How and why I am using hooks (part 1)

Don’t edit raw data

I am, as readers know, enthusiastically using agents for empirical research. And probably you’ve noticed over the last several months low key frustration at the same time. A feeling that something is not quite right, but that I cannot quite put my finger on what that is, creating some anxiety in me. But it’s not just anxiety — because I cannot settle, I end up wasting a lot of time doing a massive amount of repetitive checking of things. Almost like while I’m looking for my missing car keys, I might double and triple check the same spots over and over because I cannot 100% convince myself that they aren’t there.

I became inspired to shift how I was approaching these problems, including even identifying what those problems are, around the time I wrote this post.

It was after going through Paul Goldsmith-Pinkham’s NBER lecture slides that I began to piece together that I was over-relying on prose for directing Claude Code, and that I had to shift my mentality towards enforceable hooks, at minimum, for some of these things. I wanted to share with you some of the ones I’ve created. Keeping with my promise to provide for free the “Claude Code” series, this one will also be free. Thank you again everyone for your support. I’m going to write this in multiple posts as it’s kind of long.

---

3x3 Matrix

There are three ways to keep Claude from misbehavior, and they are not the same thing. I had Claude make a 3x3 matrix to help you see what I mean.

The columns describe ownership and possibility. “Who Runs It?” means is this the responsibility of me, the researcher, Claude the agent, or something else? Can it be skipped speaks to whether the action is truly something exogenous and constraining or merely suggestions. And then when does it actually get chosen?

The three tools meant to guide and constrain agent behavior are the rows. They are the use of markdown documents (e.g., Claude.MD, progress logs) and even the prompts themselves. Skills (e.g., /referee2, /beautiful_deck). And hooks, which I’ll explain below.

All 9 cells in this 3x3 matrix describe the way we can think about markdowns, skills and hooks. The problem at large is this discovery, from repeated experience that led to sadness and feeling a bit of demoralized, was my realization that my prose would not restrict the behavior of an agent, in either directions. Prose did not cause agents to do good things, and it did not stop them from doing bad things. There were so many false positives and false negatives accumulating, in spite of my prose (both in markdowns and in my own prompting) that I just couldn’t ignore them.

---

The Problem

When I first started working with Claude Code last year, I immediately could see a lot of the low hanging fruit and began addressing them. I could see how each new session was like starting over, which meant I needed to use markdowns to help keep myself and the agent on track. It was based on this idea that there were as many Claudes as there were independent sessions, and that I could not assume the work carried over. I had to leave breadcrumbs so that each new Claude could sort of download the consciousness of the other one, or if not that, at least remember the work we’d done. And so I did that constantly and everywhere.

One of the things I, like many of you, chose to do was create a Claude.MD file that I considered to be a Ten Commandments type of thing mixed with wisdom. I would put down all the things I did not want Claude to do — or even use very strong language of “don’t do these things”. And I would describe what I wanted it to do. And in my understanding at the time, that was sufficient. Those were like programmed instructions, or so I thought, and that they were therefore fully constraining such that I could henceforth take for granted that the markdown instructions had created an autonomous agent that would do whatever I asked.

In any reasonably deep and long empirical project using agents, though, you will if you know to look for it start to experience what I call drift, though. I call any time the agent begins to do its own thing, in spite of those markdown instructions, drift. It’s not really immoral behavior on the agent’s part so much as it’s lapsed judgment, and sometimes even cost minimizing short cuts taken due to the available discretion that it has.

I don’t have a full taxonomy of all the ways that agents have drifted for me, but I have some. They were things like making figures outside of a program that makes figures, for instance. This was one of the ones I first noticed. I had explicit markdown rules forbidding Claude ever produce a figure or table except in a file that was in what I called a “pipeline” of replicable code. The pipeline of replicable code meant code in a file that was stored in a code subdirectory, numbered as such, remembered as something you would ultimately rerun in order later. And that all exhibits — tables and figures — and numbers would always and only be traced back to pipelined code.

And Claude understood that, as he helped me write that constitution in readme.md and Claude.md to the point that I trusted him. But then one day, I was looking at a figure from a synthetic control model showing the “spaghetti plot” from permutation inference — something that had taken a very long time to run because of the size of the dataset (county level data). It had taken many hours to do all 3,000 of those permutations. And when it was done, the figure’s title wasn’t very good. I just didn’t really notice it until the next day or so, because I had to let it run all night.

So, I went to Claude and asked it to edit the title only to be told that it couldn’t because it didn’t have the results anymore. What it had apparently done is it had “vibe coded” that entire synthetic control permutation inference from terminal, on a side it controlled, and did not write any script whatsoever for it. I had caught it, I gave it a stern talking to, and had to rerun it. But any time I catch one of those kinds of things, it plants a seed in my mind: how many other things am I not catching? And how are they different from these kinds of problems?

Another way it would do this is the constant creating of new files, as opposed to editing the same file, such that I could end up with replicable code, but at the same time, stale code too. Code that was spread out over many files, or old code that existed in the pipeline that I was no longer using. All of which maybe made figures and tables, but which could not be easily replicated.

The problem is simply the sheer volume of work I am doing, including the production of figures and tables, is exceeding what I can reasonably verify. And what I’m wanting is a curated “final” version that I can focus on. And in my mind, the problem of trust. I was trusting Claude. It wasn’t so much that trust was creating problems, so much as trust would creating a set of practices that led to such problems. And the source of trust was markdowns. I thought that having a markdown, which is nothing more than words on a page, was sufficient to creating numbers, tables and figures that were reliable such that any time I looked at those numbers, tables and figures, I could say to myself “these numbers, tables and figures are what I think they are”.

---

---

How exactly are hooks doing what they do?

Above is an ASCII map I had Claude draw explaining how a particular hooking works. Notice it is a decision tree. Claude as an agent makes a decision to act and the question is will a particular action be allowed or will it be denied? In this one, I will describe how a particular hook I explain later called protect-raw-data will or will not allow something to occur.

First, the obvious: Claude does not have hands. Everything it does to my machine will necessarily pass through a short list of named tools. For instance, it reads a file with Read, it changes a file with Edit, it creates files with Write, it runs a shell command with Bash. Every one of those calls is announced, in advance, as a small structured packet: “here is the tool I am about to use, here are the arguments I am about to use it with.” A hook is a script that stands at those announcements, reads the packet, and answers yes or no before anything happens.

My hook protect-raw-data registers itself against the mutating or editing tools only. These are the Write, Edit, and Bash commands. If those commands are invoked, then Claude looks at one field in the packet which is the directory path being written to. If the directory’s path being written to lives under my raw data directory, which is where I keep all the raw data, then the hook denies the call and hands Claude a sentence explaining why, along with instructions to write to the derived directory instead. If it doesn’t, the hook says nothing at all and the work proceeds.

Notice that I had to already be pre-committed to a particular organization of the folders, though, for this hook to work. I already have been storying all raw data in a /data/raw folder. And all edited data in /data/clean folder. This layout was what made it such that the hook would literally kill and deny the action. It had to look at where the directory path was, and block it or not based on whether that directory was in the approved list or not.

But also notice what is not on that list. Read is untouched and that is on purpose. I am not sealing off the folder/directory. I still allow Claude to go into the folders/directories. Claude can read every byte of my raw data any time it likes, which is the entire point of having it. I do not forbid looking. What’s forbidden is a specific verb applied to a specific place. The promise I need is not nobody looks at the original data. Rather it is the original is the same file today that it was in June. So the hook is written against exactly the verbs that could break that promise, and against nothing else.

And that is why it works where the markdown rule didn’t. The rule required Claude to hold a fact in mind — that this directory is a special directory — at the moment it was busy doing something else. This hook does not require that I constantly remind Claude not to do something, nor does it even require I tell Claude in Claude.MD or readme.MD not to do something. Nothing has to be held in mind by anyone for this to work. Either the path starts with raw/ or it doesn’t, and it is that question and only that question that determines whether the action is allowed or banned. And it gets asked on every single write, in the same way, forever because it is binding the calling of tools itself, which Claude must do.

So, the fix is subtle. I’m not handcuffing all use of read/write/edit. It’s much more narrow than that. The hook is sitting on a tool call, it then matches the editing/mutating verbs only, and then it reads one and only one field, which is the directory pathname. Read is left open on purpose.

---

More Hook Mechanics

You can see in the 3x3 matrix how to think through what markdowns can and cannot do, but also what skills can and cannot do. Markdowns assume the agent will read them and do them. Skills assume that I will remember to do them, hoping then that agent will do what I think it is doing. But neither markdowns nor skills are the kind of enforceable mechanism you need to live in a truly civilized society. You need laws, but you need mechanisms to enforce them. You need an executive branch, not just a judicial and legislative branch, if you are going to use agents for research.

Hooks are the executive branch, not skills and not markdowns. Hooks literally halt behavior when done correctly — key idea there being “when done correctly”. There is undoubtedly human skill involved in knowing when and how to make a hook. Making good and effective hooks feels almost like solving a Soduku or Wordle puzzle. If you are good at puzzles, then I bet you will be good at making hooks, because hooks have to be created in anticipation of particular patterns of failures such that when the hooks are created, none of those types of failures can occur. You use hooks to patch up all the kinds of expected failures that agents will at scale perform.

There are two small mechanics to learn about hooks. First, some hooks run before an action is taken, and then cancel that action so that it is not taken. Think of it metaphorically as a locked door. When a hook is run before an action is taken, and then stops that action from happening, it is as if faced with a locked door. You cannot go through a locked door regardless of whether there is a sign (markdown, prompt, skill) explaining things. Hooks do not depend on prose, in other words. They forbid actions and some of the forbid it beforehand.

But other hooks run after the thing already happens. And when that happens, the hook shouts at me or at Claude so that in that moment, the thing gets fixed in the same sitting, rather than several days later (if ever).

The second kind of mechanic to know about my hooks is this idea of “fails open”. If the hook’s own code breaks, it lets me through rather than freezing me out. I did this as a safety guard. I have been concerned I’m over engineering these automated processes, after experiencing something with an older version of a skill I made that caused Claude to circle endlessly in a never ending task of checking graphics. I worried I was going to do this too with hooks so I made a safety guard wherein the thing will never stop me from working.

---

Five Hooks

• protect-raw-data

• no-fabricated-exhibit

• no-off-book-exhibit

• deck-from-pipeline

• no-stale-canon

So these are my five hooks, by name. I will focus on the first one in this post, protect-raw-data which I alluded to above in describing the concept of a hook. Feel free to borrow them by having Claude read this entire post and seeing if you need them too. It’s straightforward to make them, but I haven’t yet put these in my shareable repo, Mixtapetools. But I will soon.

protect-raw-data. “The raw data is untouchable”

My entire workflow rests on a single premise which is that the original data cannot be touched, edited or changed in any way. Every number I produce is credible insofar as that basic axiom is respected. If nothing guards that, then nothing I do downstream is reliable. And if all I ever say is “don’t edit the raw data”, I am trusting Claude to remember it, and then do it, which is like trusting someone with a loaded gun. Wouldn’t it be better to lock the gun with a safety, as well as in a locked gun box, in a locked closet, so that practically speaking the gun cannot be fired accidentally?

So, this hook matches every attempt to write or edit a file. If a file lives in a raw data folder, it stops cold before any edit and says “read it, write my changes to a derived folder.” It’s always find to read the original data. It justice not allowed to change it, so therefore changing it is the thing that is therefore blocked.

I made a hook because I wanted to start at the top and work down. The cost of quietly editing a source file once is that every single downstream number made from it is now suspect, and since there is a massive mountain of numbers being created when using agents for research, you really cannot know which ones it will be. It’s more like a haunting uncertainty than anything else. Remember the rule: anything that can happen, will happen with enough trials. We make hooks like these because they are fatal, and that fatality will happen if it is possible for it to happen, even if the probability is vanishingly small. You need the probability to be zero. Because remember, our goal isn’t to minimize errors in a world with AI Agents. Our goal is some other objective function subject to a zero error constraint. Zero errors is the constraint, not the objective, and so we hook around the “don’t edit raw data” because that alone will cause all data decisions to be wrong.

Note that this is not a remind me solution. This is a “make it impossible to do this” solution. And that’s the first step. Make it impossible to do it by hooking the read/write/edit part of the tools the agent uses. If you can hook read/write/edit, then it will not even allow the command in the first place.

---

Conclusion

So that’s my first hook. I will keep explaining my hooks as I go, and as I build them into the mixtape tools repo, I’ll share that too. But for now, I thought I’d share the parent hook guiding all of my work which is a hook that blocks altering of raw data.