What’s Wrong with FiveThirtyEight.Com, Exemplified

By: Tom Freeland, HottyToddy.com, Blogger
Nate Silver of FiveThirtyEight.Com came up through sabermetrics. Sabermetrics began in baseball writing in the 80s, and used intense number analysis to draw conclusions about sports. It has the advantage, there, about looking at something that is measured, a lot. So, when Silver commissions a piece doing numerical analysis of baseball, we’d expect it to be a strength of his new site, right? After all, it is on ESPN, isn’t it?
Today’s link for the piece on the top page screams: “The Relentless Mediocrity of Most MLB Managers.” The headline on the story itself backs down a little in a trend that continues throughout the piece until its point vanishes.  The headline: “Most managers are headed to the hall of mediocrity.”
The story begins by declaring that it is going to demonstrate that baseball managers are a pretty mediocre bunch who, with a handful of exceptions, just don’t matter to their teams at all.
I’ve read sabermetric approaches to baseball analysis going back to nearly the start of the career of Bill James, who to an extent invented it. I was prepared to be interested and persuaded, both.
Using an analysis that calculates how many wins a manager contributes to a team (the analysis is supposed to answer the question, how many wins did this coach win above an average coach?), the article starts by saying that it concludes:

if we look at effects on player performance, it’s evident that hardly any manager can distinguish himself from his counterparts. Based on my analysis, 95 percent of all managers are worth somewhere between -2 and +2 wins per 162 games.

Ok.  Got your point.  Tell me more.  The article kicks around current managers a bit, and then hints at a problem:  Maybe managers are just not hanging around long enough for many individuals to show statistically differences from others:

This isn’t to say that all managers are average, only that we can’t prove whether they are until they manage more than 1,000 games.

Turns out that most don’t ever get to that threshold. Out of those 490 managers over the last 113 years, only 119 (24.3 percent) made it to 1,000 games. The ones who didn’t make it were probably fired too soon, before they actually had a chance to prove whether they were any good. Barring Bobby Valentine-level catastrophes, no team should fire a manager before he works those 1,000 games, because there’s no way to prove he’s any worse — or better — than his replacement.

So I’ve read 3/4 of this article to discover the point really isn’t that managers are mediocre (I’m sure they are.  I mean, isn’t the headline, “Most Managers are Average?” I was expecting a demonstration) but that the data for almost all is too incomplete to tell whether most are mediocre or not.   Shouldn’t the article’s headline be either, “We can’t tell from the data I chose whether any but a few baseball managers matter,” or “Owners tend to fire baseball managers before they could really know whether they are any good?”

If this were an academic paper, the title would be:  ”Quality of Baseball Managers:  My number crunching arrived at unpublishable results.”

From “relentless” to “we don’t know.”

I’ve been reading about the flack about Silver’s site.The controversy over the climate piece (which was headlined to be that greater storm activity wasn’t likely because, adjusted for GDP there wasn’t really greater damage, but a careful reading lead me to believe it concluded, “With the data I’ve got, I don’t yet see a statistically significant change.”).  The article itself did not present climate denier views, except it was hard to understand why the author was dressing up the point “we can’t tell yet!” as if it equated to “It isn’t happening” in a way that a careful read was required to see it was the former. Concerns about the article have lead Silver to commission an independent study about it.

Silver has got to get better at responding to critics.  He isn’t answering Krugman (who called the new site “something between a disappointment and a disaster” and makes some good points about how data is used) by saying Krugman took shots at him were because Krugman is still at the Times and Silver fulminates against op-ed writers.  I had much of the same reaction to the site that Krugman did, before I’d seen anyone’s reaction, and I was very disposed to like it.

He’s also got to do something on the site that would make me think that, in addition to being an interesting writer, he’s also a capable editor– that is, that he can get other folks working for him to produce interesting work and see that it is fully developed.

Read more blog posts by Tom Freeland at his NMissCommentor blog page.