@DLeonhardt Here David says “the relationship between test scores and college grades… is strong.” And golly what a steep line! The problem is that the strength of the relationship isn’t measured by the steepness of the line, but how closely data points hew to it.

@DLeonhardt Using statistical jargon, the correct measure of the strength-and predictive value-of the relationship is r-squared (degree to which data points hew to the line), not the regression coefficient (slope of the line).

@DLeonhardt You might look at this graph and think “those data points are awfully close.” But they aren’t the real data points. They're binned means. Given information in the @OppInsights technical appendix, the actual relationship looks more like the second image here (r=0.44).

@DLeonhardt @OppInsights According to the @OppInsights analysis standardized test scores explain less than 1/5 of the variation in college GPA. So over 80% of the variation in college grades exists among students with similar test scores. Does that sound like strong prediction to you?

@DLeonhardt @OppInsights Problem #2 : a strawman. Evidence is presented to answer the question “if you used only SAT or only HS GPA to predict college grades, which would be better?” And by either steepness of line or r-squared, SAT does better. However…

@DLeonhardt @OppInsights That question is completely irrelevant to our purpose. The real question is, “compared to a prediction that ignores SAT scores, how much better is a prediction that uses them?” And neither @DLeonhardt nor @OppInsights present the analysis required to address that question.

@DLeonhardt @OppInsights We can, however, make inferences based on the @OppInsights technical appendix. Lesson one: predicting with BOTH SAT and HS GPA gives you 4x more predictive power that using HS GPA alone. So far so good!

@DLeonhardt @OppInsights But admissions offices have access to way more than these two measures. When we build a model that includes some of them--gender, whether a student is a legacy, athlete, first gen, under-represented minority, early decision, etc.--a funny thing happens to SAT predictive power.

@DLeonhardt @OppInsights It gets cut. A lot. This gets at the question “who the heck are those students with sub-1200 SATs at elite colleges anyway?” Well, it appears a lot of them are legacies, athletes, etc. We don’t need a test score to know that some of these students are lesser academic performers.

@DLeonhardt @OppInsights With these controls, the predictive power of the SAT is cut more than a third. The predictive power of HS GPA barely budges. Our explanatory power, measured by R-squared, is higher than it was. But it goes up a LOT more if we control for one more factor…

@DLeonhardt @OppInsights While @OppInsights never shows us the regression we need – the one that has everything BUT the SAT scores – we can infer that, on the margin, test scores are definitely improving predictive power. But the clear majority of potential predictive power comes from other factors.

@DLeonhardt @OppInsights So where @DLeonhardt ’s thread might lead you to believe elite colleges are forced to take shots in the dark when deprived of test scores, admitting students whom they have no idea about, in reality they retain the clear majority of their predictive power.

@DLeonhardt @OppInsights And sure, the SAT can occasionally promote equity. But if it’s an equity agenda you want to pursue, standardized tests can’t hold a candle to: 1) Ending legacy admissions 2) Ending donor child admissions 3) Ending most athlete recruitment

@DLeonhardt @OppInsights There's a deeper question of whether elite colleges should lavish their ample resources on "sure things," the students who will do fine in college and life with or without those resources, or the students on the margin, the ones for whom resources could be transformational.

@DLeonhardt @OppInsights Good predictions are key to admissions strategy in either case, but if you think that colleges have just gotta have SAT scores to make decent predictions... think twice. /end.

@JakeVigdor @DLeonhardt @OppInsights Doesn't this kind of make the pro-SAT contigent's point? IE that the SAT is key for students who didn't get to go to elite high schools?

@karlbykarlsmith @DLeonhardt @OppInsights Depends on how you run admissions. If you go by straight expected GPA maximization those students need a boost. But suppose you instead set a floor (e.g., P(GPA puts you on academic probation)<0.02) and then target lower incomes. At Harvard probation kicks in around GPA=1.7.

@JakeVigdor @DLeonhardt @OppInsights Good thread, but if we think SAT is measuring something like intelligence, how many of these controls are post-treatment ("bad controls")? For example, HS FEs if there is sorting into schools based on test scores...

@JakeVigdor @DLeonhardt @OppInsights Yes elite highschools are awesome and churn out people who become incredibly successful. How is that relevant to the debate over the SAT? Are you saying that more poor people should just go to elite highschools?

@JakeVigdor @DLeonhardt @OppInsights What’s an elite high school?

@JakeVigdor @DLeonhardt @OppInsights Silly poor people! Why don’t they just go elite highschools?

@JakeVigdor @DLeonhardt @OppInsights I don't think this is meaninful? Fixed effects can absorb a lot more variation than a continuous variable.

@JakeVigdor @DLeonhardt @OppInsights You get into elite high schools by paying for elite real estate....

@JakeVigdor @DLeonhardt @OppInsights Wait, it’s income and high school? Income in this case just serves as a proxy for IQ.

@JakeVigdor @DLeonhardt @OppInsights What does the “yes” for admission controls, etc.? Are they also controlled?

@JakeVigdor @DLeonhardt @OppInsights What’s the implication here though? That we should replace SAT with high school fixed effects because school quality predicts college GPA better than SAT?

@JakeVigdor @DLeonhardt @OppInsights This seems to drastically miss the point that was made in the original article: the SAT is a way to pick out students with potential who did not go to a good high school, come from a high income family etc. If you eliminate the SAT, either colleges start purposely favouring

@JakeVigdor @DLeonhardt @OppInsights Why would you contrast “elite” high schools with “less resourced” ones? Elite high schools are often operated at lower cost per student than many of the worst high schools.

@JakeVigdor @DLeonhardt @OppInsights What do you mean by "less resourced ones"? Some of the worst high schools in America have huge spending per capita. So what other resources are you talking about and how do we measure them?

@JakeVigdor @DLeonhardt @OppInsights Most students don't go to elite schools. Almost none do percentage wise. Not a useful metric.