An amazing journey with Claude 3.5 and ChatGPT-4o who helped me backwards engineer an econometrics theory paper and taught me a lot more in the process
An amazing journey with Claude 3.5 and ChatGPT-4o who helped me backwards engineer an econometrics theory paper and taught me a lot more in the process
Yesterday I spent about ten hours of my life trying to figure out something. I was revising the section in my book on “bias corrected nearest neighbor matching” which is a paper by Abadie and Imbens (2011) that works out a way to reduce the bias associated with matching when you don’t have common support. The steps as I knew them were this:
Interesting journey. I guess if you were using something besides linear regression for this imputation, then it would make sense to keep that format, rather than just applying the coefficient to mean differences.
When I added the posonly option to allsynth's bias correction implementation, I did it because just like you I found myself unsure whether Abadie and Imbens were talking about using all the control units or only the matched control (positively-weighted synthetic control donor) units. I recall asking a few people what they thought and no one seemed to have any idea. If you both are willing to share it, I'd love to read Abadie's full response about why they used j(i) in their application even though the theory was to use all j.
Also: you obviously learned how nnmatch implements the bias correction, but did you become convinced that using only the matched control units is the *correct* way to do it? Or is it just the way that nnmatch does it?
I was really surprised he figured out the alternative way to do the bias adjustment when I have literally never seen that written down anywhere before. I suspect it's obvious maybe for people who have a deeper intuition of regression mechanics that those would be the same. And I think maybe I just need to sit down with a pencil and paper and do it both ways and then I'll how they connect. You know how it uses python to sort of keep itself on track? I think it was doing that with me -- it would throw ideas at me, either code or hypotheses, and I'd run them down and we'd discuss it or I'd copy the output and we'd discuss it, and it was just instantaneous progress. I mean, in some ways it was funny because probably me and Claude are in the end the only ones who ever cared that there is an equivalence like that between the two approaches, and secondly, I bet it's a special case of using regression for the outcome model (maybe a machine learning model for instance could do the imputation but then this other thing might not work). But still, it was quite the joy ride.
That is very impressive. In particular because, until now, I have found LLMs substantially worse at Stata than at more widely used language like Python. And, if anything, last time I tried, ChatGPT was a tad better than Claude.
I’ve had a pretty good experience with them even for state, but my use cases for state I have been lately just getting simulations done. And what I think they’re both seem to know well is something about the structure of widespread you know, language agnostic things. Like looping or storing things or checking throughout the code of things are working. That’s one of the things yesterday that both of them kept doing that I was surprised by. They had many checks within the code to see if it was working right. But I think if you just sort of take them as not a aaa always that’s job is to do what I tell it to do but as a partner in the creative process as long as you’re willing to commit that time, then it can be really productive. I do wonder if I could’ve done that myself yesterday but the thing is I often sometimes just throw up my hands frustrated and then go do something else
Hi Scott, thanks for an interesting post! Two quick reactions: (1) minor point -- you wrote at the top of your post that Alberto's bias adjustment is a tool for reducing bias associated with matching when you don't have common support. I think I understand why you expressed it that way, but I don't think of the bias adjustment or common support that way. I think of common support as an assumption equivalent to "overlap" as defined in the classic imbens and abadie/imbens and athey papers, and I think of that as untestable assumption (as we've discussed previously), basically an expression of faith. I think of the bias adjustment as an adjustment for imperfect matching on the Xs.
I think that a reason that these nuances may matter is that you can assume perfect common support/overlap (i.e., you can absolutely assume every probability of assignment is greater than 0 and smaller than 1), and you can absolutely assume unconfoundedness, and you can still benefit from Alberto's bias adjustment when nearest neighbor matching on your Xs if your data set requires you to suffer matching discrepancies when using a function like nnmatch(). Whereas, if the bias adjustment were nothing more than a cure for the failure of the common-support/overlap assumption, then this would not be the case. Thoughts?
(2) After reading your post, I wanted to confirm that the Matching package in R replicates Alberto's bias adjustment and delivers the STATA answer you reported above, of 4.17 -- it does, as expected. Alberto's formulas, described in his STATA paper, were the original inspiration for much of this library (produced and maintained by Jas Sekhon). Code below (showing results with and without bias adjustment):
## Units 1-4 were treated units and units 5-10 were control units.
mout$est # with bias adjustment (we see that bias adjustment makes a big difference,
# a warning sign for sure)
# [1,] 4.166667
mout$index.treated
# 1 1 2 3 4 4
mout$index.control
# 7 8 5 10 7 8
mout$weights
# 0.5 0.5 1.0 1.0 0.5 0.5
The 'index.treated', 'index.control', and 'weights' outputs show you how the matched data set got put together. Unit 1 was matched in a 2-way tie with units 7 and 8, requiring each of those matched pairs to get 50% weight. Unit 2 was matched to unit 5, and unit 3 was matched to unit 10. Unit 4 was matched in a 2-way tie to units 7 and 8 (with 50% weights) -- so units 7 and 8 were involved in two different sets of ties (matching with replacement was allowed).
Interesting journey. I guess if you were using something besides linear regression for this imputation, then it would make sense to keep that format, rather than just applying the coefficient to mean differences.
That’s a good point. I didn’t think about that. If I was using any other predictive model, I might not be able to have this kind of equivalence
Wow - this was quite a journey Scott. Thanks so much for sharing it!!!
This was a great read!
When I added the posonly option to allsynth's bias correction implementation, I did it because just like you I found myself unsure whether Abadie and Imbens were talking about using all the control units or only the matched control (positively-weighted synthetic control donor) units. I recall asking a few people what they thought and no one seemed to have any idea. If you both are willing to share it, I'd love to read Abadie's full response about why they used j(i) in their application even though the theory was to use all j.
Also: you obviously learned how nnmatch implements the bias correction, but did you become convinced that using only the matched control units is the *correct* way to do it? Or is it just the way that nnmatch does it?
Fascinating use of Claude!
I was really surprised he figured out the alternative way to do the bias adjustment when I have literally never seen that written down anywhere before. I suspect it's obvious maybe for people who have a deeper intuition of regression mechanics that those would be the same. And I think maybe I just need to sit down with a pencil and paper and do it both ways and then I'll how they connect. You know how it uses python to sort of keep itself on track? I think it was doing that with me -- it would throw ideas at me, either code or hypotheses, and I'd run them down and we'd discuss it or I'd copy the output and we'd discuss it, and it was just instantaneous progress. I mean, in some ways it was funny because probably me and Claude are in the end the only ones who ever cared that there is an equivalence like that between the two approaches, and secondly, I bet it's a special case of using regression for the outcome model (maybe a machine learning model for instance could do the imputation but then this other thing might not work). But still, it was quite the joy ride.
That is very impressive. In particular because, until now, I have found LLMs substantially worse at Stata than at more widely used language like Python. And, if anything, last time I tried, ChatGPT was a tad better than Claude.
I’ve had a pretty good experience with them even for state, but my use cases for state I have been lately just getting simulations done. And what I think they’re both seem to know well is something about the structure of widespread you know, language agnostic things. Like looping or storing things or checking throughout the code of things are working. That’s one of the things yesterday that both of them kept doing that I was surprised by. They had many checks within the code to see if it was working right. But I think if you just sort of take them as not a aaa always that’s job is to do what I tell it to do but as a partner in the creative process as long as you’re willing to commit that time, then it can be really productive. I do wonder if I could’ve done that myself yesterday but the thing is I often sometimes just throw up my hands frustrated and then go do something else
Agreed. Awesome work, I look forward to seeing other applications you may have with them.
Hi Scott, thanks for an interesting post! Two quick reactions: (1) minor point -- you wrote at the top of your post that Alberto's bias adjustment is a tool for reducing bias associated with matching when you don't have common support. I think I understand why you expressed it that way, but I don't think of the bias adjustment or common support that way. I think of common support as an assumption equivalent to "overlap" as defined in the classic imbens and abadie/imbens and athey papers, and I think of that as untestable assumption (as we've discussed previously), basically an expression of faith. I think of the bias adjustment as an adjustment for imperfect matching on the Xs.
I think that a reason that these nuances may matter is that you can assume perfect common support/overlap (i.e., you can absolutely assume every probability of assignment is greater than 0 and smaller than 1), and you can absolutely assume unconfoundedness, and you can still benefit from Alberto's bias adjustment when nearest neighbor matching on your Xs if your data set requires you to suffer matching discrepancies when using a function like nnmatch(). Whereas, if the bias adjustment were nothing more than a cure for the failure of the common-support/overlap assumption, then this would not be the case. Thoughts?
(2) After reading your post, I wanted to confirm that the Matching package in R replicates Alberto's bias adjustment and delivers the STATA answer you reported above, of 4.17 -- it does, as expected. Alberto's formulas, described in his STATA paper, were the original inspiration for much of this library (produced and maintained by Jas Sekhon). Code below (showing results with and without bias adjustment):
## Units 1-4 were treated units and units 5-10 were control units.
scott_data <-
c(
"Andy", 200, 1, 23,
"Betty", 250, 1, 27,
"Chad", 150, 1, 29,
"Doug", 300, 1, 22,
"Edith", 225, 0, 27,
"Fred", 500, 0, 32,
"Gina", 200, 0, 26,
"Hank", 190, 0, 26,
"Inez", 180, 0, 28,
"Janet", 140, 0, 29)
scott_data <- data.frame(matrix(scott_data, nrow = 10, byrow = TRUE))
names(scott_data) <- c("name", "outcome", "treat", "covar")
scott_data[,2] <- as.numeric(scott_data[,2])
scott_data[,3] <- as.numeric(scott_data[,3])
scott_data[,4] <- as.numeric(scott_data[,4])
#scott_data
#str(scott_data)
mout <- Match(Y=scott_data$outcome,
Tr=scott_data$treat,
X=scott_data$covar,
M=1, BiasAdjust = TRUE)
mout$est.noadj # no bias adjustment
# [1] 36.25
mout$est # with bias adjustment (we see that bias adjustment makes a big difference,
# a warning sign for sure)
# [1,] 4.166667
mout$index.treated
# 1 1 2 3 4 4
mout$index.control
# 7 8 5 10 7 8
mout$weights
# 0.5 0.5 1.0 1.0 0.5 0.5
The 'index.treated', 'index.control', and 'weights' outputs show you how the matched data set got put together. Unit 1 was matched in a 2-way tie with units 7 and 8, requiring each of those matched pairs to get 50% weight. Unit 2 was matched to unit 5, and unit 3 was matched to unit 10. Unit 4 was matched in a 2-way tie to units 7 and 8 (with 50% weights) -- so units 7 and 8 were involved in two different sets of ties (matching with replacement was allowed).