The anti-SAT push is misguided

The anti-SAT push is misguided

If you want to help less privileged students, help the schools they actually attend

Matthew Yglesias

 May 21 

Comment

Share

(Photo by Shawn Patrick Ouellette/Portland Portland Press Herald via Getty Images)

The University of California is moving to stop requiring SAT/ACT scores as part of college applications, and the trend seems likely to continue as part of a broader backlash against standardized testing, some of which originates from a completely different set of concerns.

But in terms of college admissions, the worry is basically that not enough Black and Latino kids earn high scores on the SATs, so relying on SAT scores as an important part of admissions makes it hard to assemble college classes that meet someone’s idea of adequate diversity. At the same time, it’s quite striking that Asian kids earn higher scores than white ones, and I think there is some reason to raise an eyebrow or two when antiracism initiatives that mostly seem to target Asians as the overrepresented group are concocted.

The whole anti-test push is misguided in some pretty fundamental ways.

And by relying on sloppy, factually inaccurate critiques of widely used tests, it’s ultimately going to end up empowering and reifying some very conservative, deeply inegalitarian views about how society and economics should work. Like anything you could come up with, the SAT is not a perfect test. But it does, in fact, work pretty well at its intended purpose of providing a meritocratic sorting tool for academic skill. If you want to build a better world for the kind of people who do poorly on that kind of test, you need to actually go do the work of building that better world.

Most critiques of SATs are wrong

People offer a lot of casual criticisms of the SAT that are false or misleading, such as noting that kids with richer parents have higher SAT scores and thus inferring that the test is easily gamed by high-income parents.

It’s true that there is a modest positive correlation between parental income and SAT scores, but you see a similar positive correlation with pretty much anything related to school or child development. Parceling out exactly why it is that the children of doctors and lawyers and business executives do better at school-related stuff than the children of waitresses and cashiers and cab drivers is probably really hard. But broadly speaking, people who do well in school and have high standardized test scores end up earning more money than those who don’t. They then have children who are genetically related to them, and they raise those children in households where the adults are able to constantly model the behaviors of a good-at-school person. There would be something profoundly weird about a world in which the children of good-at-school people were not, on average, better at school than the children of bad-at-school people.

What’s not the case is that rich parents are bestowing huge gains to their kids via the mechanism of extensive test prep. Slate’s Daniel Engber did a good roundup of this in 2019 — the benefits of test prep are modest, maybe between 10 and 30 points out of 1600. There’s a Wall Street Journal article making the same point.

Note that this is not the same as saying that practicing for the tests isn’t helpful! You will absolutely do better on a test if you are familiar with the kind of questions you are going to be asked than if you show up to it cold. But what that means is that taking a little time to prepare is going to help you, not that vast sums of money are going to dramatically boost your score. From the WSJ article:

Laurence Bunin, a College Board senior vice president, says the board's own research shows limited benefit from test-prep courses. He says familiarity with the SAT tends to provide the biggest short-term gains for students. He recommends free and low-cost College Board materials, including a $20 study guide.

This kind of practice can make a huge difference!

PSAT/NMSQT to SAT Score Gain and Time Spent on Official SAT Practice, by PSAT/NMSQT Score

Last but not least, there is an idea in widespread circulation that SAT scores are poor predictors of college success or of anything else useful in life. I used to think that there was a sense in which this was true, but that it was a statistical illusion induced by range restriction. It turns out that it’s actually not true at all, and that when you correct for range restriction, it becomes even less true. But Freddie deBoer wrote a good article about this that showed me even beyond the illusion, it simply isn’t true.

Schmitt et. al. find “the primary predictors of cumulative college grade point average (GPA) were Scholastic Assessment Test/American College Testing Assessment (SAT/ACT) scores and high school GPA (HSGPA) though biographical data and situational judgment measures added incrementally to this prediction.”

Kuncel and Hezlitt find that standardized tests for grad school admissions are good predictors, too.

Allensworth and Clark claim to have found that ACT scores are poor predictors of college performance, but this involved the application of an incredible amount of statistical controls in a way that I think invalidates the result.

That last study garnered plenty of popular press coverage that reveals how big a market there is for takes that suggest standardized test scores don’t convey useful information. But they do! And why shouldn’t they? There may not be a simple multiple-choice test that can convey the depth of your soul or the grace of your character, but the mission of the standardized test maker is to come up with a multiple-choice test that correlates strongly with being good at school. And if you think a test couldn’t possibly capture being good at school, I don’t know what kind of schools you were going to, because my schools had a lot of tests.

Here, again from deBoer, a link to a study showing pretty clearly that the people with high SAT math scores are way better, on average, at long-term achievement in the field of “doing stuff that requires you to be good at math.”

Again, we are talking about broad population averages here. I have met a lot of people with high SAT scores in my life, and a non-zero share of them are fuckups who, for one reason or another, never managed to do a good job of anything. But that is a minority. Usually if someone has very high standardized test scores, that’s because they are good at reading and math, and there are many areas of life where being good at reading and math is helpful.

What are we doing here?

At this point, I think the conversation refracts and starts flying off in a million different directions.

Down one road are the people who sincerely believe in the transcendent importance of sorting teenagers based on academic aptitude. Some of those people are test skeptics who sincerely believe that the flaws in the existing testing paradigm (which are certainly real) are a huge problem and that if we move to some other system, we will get much better at our teen-sorting.

I think those people should read Susan Dynarski, who persuasively argues that this is basically backward. The biggest thing we could do to improve our teen-sorting system would be to make sure all teenagers take the ACT or SAT. Rich parents don’t neglect to have their kids tested, but low-income ones sometimes do. They end up falling through the cracks. Universal screening based on standardized tests is our best available tool for preventing the intergenerational transmission of privilege. Indeed, that’s exactly what they were originally invented for — some colleges got concerned that they were overly reliant on a handful of feeder schools and were missing out on lots of talent around the country. So they needed a standard way to find that talent.

After all, imagine the alternative. Suppose your ability to get into an elite college dependent entirely on your high school GPA. Can you imagine the pressure that affluent, well-connected parents would bring to bear on teachers and administrators to give their little darlings a leg up? The scope for potential fraud and cheating?

But for some test-skeptics, I think this is exactly the point. On some level, they agree with Dynarski and just want to switch to a system that gives their kids a leg up versus Asian immigrants’ kids. The original backlash to standardized testing was that it was sending too many Jewish “greasy grinds” to the Ivy League, so they wanted to incorporate more subjective measures to ensure “well-rounded” gentlemen would get their shot. But other people, I think, have a really sincere but somewhat confused view about racism.

Testing and the Kendi view

A very influential factor in these debates is the view of Ibram Kendi and his intellectual allies that anything that shows a significant racial gap just is racist by definition.

And the SAT definitely shows big gaps:

In other words, there are almost as many Asian students as white ones with SAT scores above 1400 (53,628 versus 63,699) and very few Black (2,613) or Latino (11,387) students with scores that high. So the test is racist.

Twitter avatar for @JamaalBowmanNY

Jamaal Bowman 

@JamaalBowmanNY

Standardized testing is a pillar of systemic racism.

March 2nd 2021

2,263 Retweets14,806 Likes

It’s important to understand that’s not an analysis of the tests. Nobody is saying that the College Board deliberately cooked up a test designed to make Asians look good. Racism is not a hypothesis that might or might not empirically explain the gap. It’s a description of the gap’s existence. In Kendi’s hands, this turns into not just a critique of one standardized test or of all standardized tests, but of the idea of academic achievement:

But what if, all along, our well-meaning efforts at closing the achievement gap has been opening the door to racist ideas? What if different environments actually cause different kinds of achievement rather than different levels of achievement? What if the intellect of a poor, low testing Black child in a poor Black school is different—and not inferior—to the intellect of a rich, high-testing White child in a rich White school? What if the way we measure intelligence shows not only our racism but our elitism?

Gathering knowledge of abstract items, from words to equations, that have no relation to our everyday lives has long been the amusement of the leisured elite. Relegating the non-elite to the basement of intellect because they do not know as many abstractions has been the conceit of the elite.

What if we measured literacy by how knowledgeable individuals are about their own environment: how much individuals knew all those complex equations and verbal and nonverbal vocabularies of their everyday life?

There is a level on which I think this is insightful. There was a guy in my house the other today whose job was to figure out how a mouse had been intruding into our kitchen. He successfully identified several gaps I hadn’t found, located some mice droppings I hadn’t previous seen, and then did the work of plugging the gaps quickly and efficiently. He is a lot better at that than I am, and my strong guess is that I am a lot better at reading and math than he is. To dub one of those things “intelligence” and elevate it to higher plane than something practical like averting rodent intrusion is a kind of elitist construct.

At the same time, if you want to stop mice from getting into your house, you want to hire a good mouse guy. If the guy you’re hiring is in fact terrible at rodent control, it’s no good to say “well he’s an amazing guitar player.” Conversely, you do not want me playing guitar in your band. To acknowledge the diversity of human competencies is not to deny the relevance of ranking within one domain. And Kendi is really just talking in circles around the fact that “knowledge of abstract items, from words to equations” is exactly the thing that admissions-sorters are trying to sort for. The more profound question is: Should we actually be doing that?

Different ways of sorting

If you roll back the clock 150 years, it’s pretty clear why you’d want a big element of sorting in your higher education system. The vast majority of people weren’t going to college at all, and society didn’t really have the resources to do higher education at mass scale. In the face of a profound scarcity of schooling opportunities, you want to try to give those opportunities to an idiosyncratic minority of people who seem to have unusually strong skills.

That is not really the situation that we are in today, but we still have an education system dominated by ideas about sorting.

This sorting currently happens in two different ways. One is the pure test-based sorting of something like the New York specialized high schools. Here the idea is basically that there are some few, precious slots at Stuyvesant and they must be allocated to those who do best on an admissions test. The other is the more artful and delicate process of “crafting a class” that they do at selective colleges. When Natalie Portman was in my class in college, the admissions officers weren’t saying she was smarter than every single person who didn’t get in. They were saying she was smart enough to do the work (which she was!) and also that it would be kind of cool to have Natalie Portman in a class — which it was!

But you could think about that Portman Principle in another way. A school could set some minimum bar where they say “if you met this bar and were also a movie star, we’d be inclined to say yeah, you should definitely come here.” And then you could admit at random everyone who is over the bar. And you could aspire to grow the size of your institution to eventually incorporate more and more people who clear the bar. To me, that’s a kind of appealing model.

An idea we have more practical experience with is Texas, which instituted a policy where the top 10% of kids from each high school would automatically be admitted to UT Austin. A great study by Sandra Black, Jeffrey Denning, and Jesse Rothstein compares the kids who were “pushed out” of Austin by that policy with those who were “pulled in.” They find that the kids who were pulled in “see increases in college enrollment and graduation with some evidence of positive earnings gains 7-9 years after college.” The pushed-out kids end up in less-selective colleges, “but do not see declines in overall college enrollment, graduation, or earnings.”

Unfortunately, due to Texas not expanding the size of the Austin institution, this is now more like a top 7% policy than a top 10% policy. But I think it’s a pretty attractive model, and also seems to have egalitarian consequences for the real estate market because it essentially gives people a bonus for moving to neighborhoods with “bad” schools. Especially given the severity of California’s housing issues, I would strongly recommend that they copy this idea.

I would note, though, that a Top X% rule doesn’t have to have anything to do with anti-testing zeal. You could have every student in the state take the SAT, and then automatically admit the Top X% of SAT-scorers to Berkeley or UCLA or wherever and capture the same benefits. If anything, totally detaching grades from college admissions might make more sense to me, since that way grades could perform the function of helping students understand the quality of their work without having any high-stakes consequences for the future that create pressures toward grade inflation.

No major problem is going to be solved here

If you’re interested in these topics, I really recommend de Boer’s book “The Cult of Smart.”

He’s more of an education nihilist than I am, so I don’t fully endorse all his conclusions, but I do endorse what I think is his main conclusion: if you come to the world with egalitarian values, you’re just not going to find vindication of those values by pulling levers and twisting nobs in the education system. Half of students are going to be below average. Sixteen percent of the population will be a standard deviation or more below average. A small minority of the population is going to possess the most elite educational credentials, and they are going to be heavily networked with each other and disproportionately represented in the commanding heights of politics and the economy.

Those are features of the world that are not amenable to change, and an egalitarian worldview has to mean caring about the people who fare poorly in a ranking system, whatever that system is, not quibbling over exactly who gets to be at the top of the ranking system.

To put it another way, changing who is admitted to flagship state university campuses is not really going to impact child poverty or climate change or global economic development. It’s not going to accelerate productivity growth or generate full employment or extend life expectancy. Even if SAT abolition were a good idea, there are no big problems that it could possibly solve. And when you recognize that the test is, in fact, a pretty good (albeit imperfect) tool for sorting teenagers, it just starts to look even more perverse to be focusing time and energy on trying to uncover hidden flaws in the system.

There are some foundational questions to ask

The Delta Cost Project’s latest data on college and university funding is unfortunately a bit long in the tooth these days, but as of 2013, they showed that state and local governments are appropriating more money per full-time equivalent student for their research universities than for their community colleges.

If you want to try to do egalitarian policy inside the walls of the higher education system, this is probably the place to look. America’s community colleges provide vital services to a very large number of people from modest circumstances.

But the completion rates at these schools are very bad.

Anything you can do to invest more money in raising the completion rates at the institutions that serve large numbers of students from less-privileged backgrounds would be pretty impactful. Joe Biden’s American Families Plan includes significant community college commitments that could make a real difference here. He’s offering essentially a federal match plan for states that want to do zero-tuition community college. It’s designed to work out to Uncle Sam covering 75% of the cost on average, but the exact match ratio would vary by state so as to not unduly reward the stingiest states. Even if it passed, lots of states probably wouldn’t take the money, but plenty of them would. And Biden also wants to increase Pell Grants and make them usable for living expenses, so low-income students could focus more on their schoolwork and finish a two-year program in two years rather than trying to squeeze it in around a full-time job.

I have not done a deep dive on community college reform proposals, so I don’t know if there’s some other much better idea out there. But Biden’s idea, unlike the anti-testing push, is grounded in a plausible theory of action — targeting more help at the people who need the most help, rather than peddling what are at best half-truths about the flaws in our main standardized tests.

Comment

Share

This post is only for paying subscribers of Slow Boring .

Like & Comment

© 2021 Matthew Yglesias Unsubscribe

548 Market Street PMB 72296, San Francisco, CA 94104

Publish on Substack