Official data is flawed, epistemic nihilism is worse. By Matthew Yglesias

www.slowboring.com

Official data is flawed, epistemic nihilism is worse

Matthew Yglesias

11 - 14 minutes

Over the weekend, a handful of people from the tech sector, namely David Sacks and Balaji Srinivasan, seemed to accuse the Bureau of Labor Statistics of cooking the books on jobs numbers. This drew substantial (and substantive) criticism from Econ Twitter before the conversation devolved into name-calling.

I think the tech sector folks are pretty clearly wrong here, primarily because if they weren’t, we would already know. “The media” functions much better than its critics claim, and practically all of the outlets that are fairly characterized as biased toward the left would nevertheless love to publish a book-cooking scoop from a whistleblower at the Department of Labor or Commerce.

Many on the right are psychologically and politically invested in anti-media narratives, but I promise that if you log off for 10 minutes, take a few deep breaths, and think this through logically, you’ll realize that in whatever sense you think The New York Times and CNN and the Washington Post and ABC News and Reuters are in the tank for Joe Biden, it’s just not true that their reporters would deliberately bury a huge career-advancing scoop. It may also occur to you, in this calm and contemplative headspace, that if for some reason those reporters did try to bury the story, the whistleblower could take it to Fox or the Wall Street Journal or Joe Rogan or any one of the many, many other high-profile, extremely successful media outlets that you arbitrarily exclude from your account of “the media” when you characterize it as in the tank for Biden. All that stuff you know about that the media doesn’t want you to know about? You found out about it from other media outlets, the very same outlets that would blow up any effort to cover up an economic data conspiracy.

But while the bailey version of this argument is dumb, the motte version is underrated. It drives me crazy that the federal government reports homicide statistics with what is optimistically an 18-month lag, leaving us with no better option than to squint at Jeff Asher’s big city data and guess what’s happening (murder seems to be down 10% from last year, for the record). With the economy, the government does a much better job of releasing timely information, which I think is good. But measuring macroeconomic aggregates is objectively more difficult than tallying dead bodies, and it’s important to be realistic about how solid this information really is.

One should not run around accusing civil servants of participating in implausible conspiracies. I also think it’s fundamentally wrong to frame this as a question of the government somehow being inept or incapable of doing things. The lack of timely and comprehensive murder data really is a huge failure of state capacity and something that policymakers should address. But America’s macroeconomic data is really good by global standards, it is routinely used by private sector actors even as they supplement it with their own, and it’s revised over time as more information becomes available. Questions like “what is the total value of goods and services produced in the United States?” are simply difficult to answer, and questions like “what share of the quarter-to-quarter change in that value are due to changes in prices rather than changes in the quantity of goods?” pose both theoretical and empirical challenges.

It is always important to keep in mind that the official data might be off in either direction, and if you believe some non-government source of information gives you insight into which direction that is, that’s a totally reasonable way to go about generating a hypothesis. That said, as an intellectual discipline, you should also keep in mind that there are a lot of people at quant funds and investment banks scouring the world for an edge, and while these traders may not be perfect, they are putting real money on the line.

It’s also important to maintain some internal consistency in your hypothesis. The economy might be running hotter than the BLS says or it might be running cooler, but it’s quite unlikely that things are worse than they appear in all directions simultaneously.

If you want to convince the world that American economic data suffers from accuracy problems, the best way to do it is probably to share a chart like this one.

It shows the trajectory of inflation-adjusted Gross Domestic Product (GDP) and of inflation-adjusted Gross Domestic Income (GDI). You can see that these numbers are broadly similar, but they are different. They also move in different directions.

Real GDI peaked in Q3 2022 and has been shrinking ever since — shrinking to a level that’s actually lower than where we were in Q4 2021. By contrast, GDP indicates we had a small hiccup back in the first half of 2022 (when the economic impact of the Ukraine War was at its maximum) but growth has resumed.

The big problem here is that if you pop open your economics textbook, you’ll see that GDP and GDI are identical by definition. GDI is the total value of all income earned by everyone in the economy, and GDP is the total value of everything produced throughout the economy. The value of what you produce is, by definition, your income. So the sum of production and the sum of income should be the same. The reason they diverge is that real-world counting processes are imperfect — we have a regulative ideal of what we are trying to measure, but it’s not actually possible to measure it perfectly.

It’s no big secret that this is imperfect; the Bureau of Economic Analysis publishes the two figures quite openly.

And the correct takeaway is that this is an agency with a high degree of integrity. Publishing both GDP and GDI gives readers the chance to average the two in hopes of getting closer to the truth. The downside is it makes BEA look bad. If all they cared about was maximizing their own reputation, they might just publish GDP and insist it’s the capital-T Truth. Instead, they try their best with multiple methods, put everything out there, and we can all make use of it as we try to better understand the world.

And I think that’s the right way to think about the situation: a lot of earnest, hard-working people are using the best methods available (for the most part) to try and do something that’s simply very difficult.

Things like the statistical discrepancy between GDP and GDI aren’t always very interesting. But our post-pandemic situation is interesting, from a data perspective. There are very few unemployed people to re-employ and population growth is also very slow, so there isn’t big quarterly growth in the labor force. We also, unfortunately, do not appear to be living through a moment of booming productivity. That means small measurement errors can be the difference between growth that’s positive and growth that’s negative. On some level, we should probably train ourselves to not be so hung up on small disagreements that happen to have zero in their range. Everyone can see that 4.7% growth and 4.2 percent% are pretty similar, but 0.3% growth is growth whereas -0.2% growth is a recession, which seems like a big deal. The core point here is that it’s hard to estimate things with incredible precision.

This all started, more or less, with people observing that the government’s two different employment series (employment level from the BLS household survey and all employees from the BLS establishment survey) moved in opposite directions in May. What’s even more striking, though, is that the levels are consistently different by millions of jobs.

In part that’s because “how many people have jobs?” and “how many jobs are there?” are different questions — some people work more than one job. But that actually makes the problem worse because it implies that the employment level should be lower than the payroll count, not higher.

So what’s the deal? It turns out that when surveyed, a lot of people say that they are self-employed, but a survey of business establishments does not include the self-employed. Adjusting the household survey to subtract out self-employed people gives the “correct” result that the number of employed people is smaller than the number of jobs.

For a good, really thorough discussion of all the relevant technical issues — including the important reality that these numbers are always revised after their initial release — I would recommend Noah Smith’s post.

The point here isn’t to double down and insist that the official economic data is beyond question. It’s actually pretty flawed. But it’s flawed because the thing that statistical agencies are asked to do is difficult. The revised data is much better, and if the agencies wanted to spare themselves embarrassment, they would just put everything out with multi-month lags. But the current thinking is that the rushed data is good enough to be useful, and we shouldn’t hold it against the agencies that they try to do timely work. And in particular, we shouldn’t muddy the waters between the idea that the work is imperfect and the idea that the work is somehow fraudulent or manipulated.

Something that often frustrates me about media is that communicating uncertainty tends to make for bad copy. When media outlets err, it’s often by expressing an undue level of confidence about an initial conclusion. That, at root, is what I think happened with the Covid lab leak theory, but a lesser version plays out every month when the jobs numbers are published. The people who write the jobs day articles know the data is revised. They know that the accurate thing in the monthly report is the second revision of the monthly number from two months ago, not the hot-off-the-presses new number. They’re not ignorant, and they’re not lying to the audience, either. But at the end of the day, the emphasis on newness performs better, so that’s what you get — just like negativity bias seeps into the framing of so many stories.

The issue here, though, is primarily with the audience, which likes what it likes.

Meanwhile, the loudest “skeptic” voices on Twitter and in alternative media are very rarely displaying less overconfidence or more willingness to question established narratives than the mainstream press. They are just less confident about the specific things the mainstream press is confident in and less invested in the specific narratives the mainstream press is invested in.

It’s true that society’s epistemic institutions do not function perfectly. And I am a big advocate of getting them to function better. That includes encouraging people to make specific predictions, encouraging donors to care more about rigorous analysis, and encouraging more boldness in the face of “cancel culture.” It also includes supporting specific policy reforms to improve the government’s data. I believe that we currently place too much weight on privacy concerns in a way that prevents the Census from being as good as it can be, and that it inhibits the use of administrative data to complement surveys and improve precision.

But there’s a big chorus out there not encouraging specific measures to improve data and not funding rigorous research on relevant social questions. They’re just tearing things down, often in a highly biased and selective way, trying to create a world in which nobody has confidence in anything. This is good for people who have a lot of money or fame or power that they can use to broadcast their ideas, but bad for society. Understanding the world accurately is challenging, and we should be trying harder, not making things worse.