Miscellaneous projection thread (now with link to '07 HF projections)

[Roberbola]

by this website:

Replacement Level Yankees Weblog

Now, I'm not very SABR-sophisticated but I just don't see how the Rockies score less runs than everybody but KC and Cincinnati, and give up more than everybody but Tampa.

How do they go from 813 runs scored last year to 740 this year? Obviously this CHONE projection system that he uses isn't very friendly to the Rox. I browsed through these projections for the Rockies and while they looked rather bad, I don't have a good context. I noticed that he hasn't updated the roster for the JJ trade.

Anyway, the projections also show the Giants winning the division going away, which I thought was humorous given how much heat they take here.

Interested in others' thoughts who are more conversant on this type of analysis.

[indianadrew]

66 wins good god. this will be just the type of thing to get this club fired up. 82-80, 2 games over is what i say. 2 mos is gonna fly by and it will be ST before we know it.

heltonfan are you going to spring training?

GO ROCKIES

[Heltonfan]

Those are some wacky projections. Massive regressions from Cook, Francis, and Fuentes... Ramirez looks worthless, because his past numbers as a starter are dragging down the projection (a perfectly understandable mistake, but a mistake nonetheless)... Hawpe projected as a league-average hitter... Carroll projected much closer to his '05 numbers than to '04 or '06... yeesh. There isn't a single player on the team who this system likes significantly more than I do, and my projections have historically been among the more pessimistic (compared to ZiPS, PECOTA, etc).

And of course, any team projections are going to be heavily influenced by the allocation of playing time. I have absolutely no clue how SG chose to do that for the Rockies, but it wouldn't surprise me if guys like Morillo and Stewart are dragging down the team projection a little.

I can't point to a clear methodological flaw here, because I don't know what the methodology is, but these projections look way off to me.

FWIW, here's my current projected NL West:

SD 88-74
LA 85-77
SF 81-81
ARI 81-81 (with Big Unit - that's looking like a done deal)
COL 77-85

[Heltonfan]

One quick-and-dirty thing I like to do to evaluate projection systems is to look for individual results that are completely inexplicable. All projection systems will struggle with the following categories of players:
- Guys coming off injuries
- Guys without much playing time in past seasons (ye olde sample size problem)
- Pitchers whose projected role (starter or reliever) is different from the one they've had in past years

If a system appears to be way off on a player in one of those categories, I don't judge too harshly. If a system produces crazy results for "ordinary" players, it suggests a logical flaw in the projections. For example... I can't remember if it was last winter or the year before, but ZiPS projected Zach Parker to have a lower ERA than Jennings. That was all the evidence I needed to deem ZiPS a poor system. PECOTA is a special case, because I actually know what their methodological flaw is; their MLEs are way too generous.

CHONE is a bit of a tougher nut to crack - it takes a little more searching to find something completely unreasonable. But I think I've found one - CHONE thinks that Justin Hampson is equivalent to Josh Fogg. I'm stumped. They were roughly equal last year, but in '04 and '05, while Fogg was getting hammered in the big leagues, Hampson was getting hit just as hard a level or two below that. I don't see any way to defend that one.

[rockieprogress]

HF, do you really think the loss of JJ is going to hurt the Rockies as much as Tulo replacing Barmes will help them?

[BigRapidsJackass]

HF, just out of curiosity, what do you think of Tango's "Marcel?"

As I've said, I'm purely a consumer of this stuff. But I like Tango's "less is more" approach.

Oh, and it's nice to see order restored to the world, with the Rockies projected to be comfortably in last place.

I take it you haven't factored in that big Karim Garcia signing?

[BigRapidsJackass]

To those who think we're a tough crowd on this forum, check out this comment on Randy Johnson's return to Arizona (from BTF):

At least Arizona is closer to Washington, so Big Gangly can be nearer to the illegitimate child he ignores. Family is important.

[Heltonfan]

HF, do you really think the loss of JJ is going to hurt the Rockies as much as Tulo replacing Barmes will help them?

I don't think Tulo replacing Barmes is an upgrade at all. Barmes is a better fielder than Tulo, and he's a much better hitter than he showed last year. And it's not like Tulo is a good major league hitter yet. Far from it. His lousy hitting after his callup was just about exactly what I expected (.224 ABR in AA, .226 in the majors).

Regarding Marcel: I'm a big fan of the concept. My projections are built off of Marcel, with five adjustments:
1) Minor league performance is included in the projection.
2) Park factors.
3) The younger the player, the more heavily I weigh his most recent performance (capturing the development curve of prospects; a breakout season from a 21-year-old is much more likely to be "real" than is a breakout season from a 28-year-old).
4) I don't regress to the mean, because I'm projecting a much wider range of players. Regressing an A-ball pitcher's performance to the major league mean would be ridiculous. The regression makes perfect sense if one is looking at big league stats only, as Marcel does. But for what I do, it would be a huge mistake.
5) I can update my projections mid-season, with different weightings for past seasons depending on where we are in the current year (i.e. the impact of previous years on the projections diminishes as the season goes along, so that by the end of the '07 regular season, the players' '04 stats will have no impact on their projections at all).

I love Marcel. It should be the foundation of any more sophisticated projection system. But it's important to note that Marcel's results should be used as a sort of sanity check for other projection systems, not as a viable alternative to them. That is, when there's a substantial difference between a player's Marcel projection and his projection in another system, the reason is either that Marcel isn't taking enough information into account or that the other system is doing something stupid. And the key, of course, is figuring out which of those two possibilities applies. I've been careful not to add any elements of dubious theoretical value (such as PECOTA's similarity scores) so that I know that the disagreements between Marcel and me are due to easily explained (and easily defended) aspects of my system.

[Heltonfan]

Time for another post about projection theory, as the '07 PECOTAs are out. They're subscriber-only, of course, so I can't share everything, but I can share enough to make any reasonable person dismiss them as absolute garbage.

Here it is: This guy is projected with a .266 EqA. That's right. A 21-year-old with no experience above short-season ball, and whose only positive attribute in that league was a high BABIP (note the lack of power and the virtually non-existent walk rate), projects as an above-average major league hitter. How is this possible? Two things:
1) PECOTA's MLEs are horribly, horribly wrong. I've gone over this before. Basically, they think that the difference between MLB and any given minor league is about 75% of what it really is.
2) Howie Kendrick appears on Pino's list of comparable players. I can see why this happens: Kendrick's '02 and '03 seasons are fairly similar to Pino's '05 and '06. That is, they're both second basemen, the same age, virtually the same height and weight, and their overall production in those two years was about the same. But this doesn't mean that it makes any sense at all to let Kendrick influence Pino's projection. Why? First, they're not really comparable, because Kendrick displayed a real knowledge of the strike zone at that age. Second, and more importantly, Kendrick's development was so abnormal that it shouldn't be used as a template for anyone else, ever. Kendrick had a .337 Pioneer League ABR in '03... in '04, the season corresponding to Pino's projected '07, he had a .360 in the Midwest League. That is, he moved up two levels, and his production relative to league increased substantially. There's no class of players, of any age, height, weight, statistical profile, Zodiac sign, whatever, for which that kind of improvement happens with any regularity. It's inconceivable.

Now, the obvious follow-up question: in light of all this nonsense, how is it that PECOTA does so well (slightly better than the other systems out there) in all the studies of projection accuracy? Simple answer: because the Wilmer Pinos of the world aren't included in those studies. The studies look only at players who amass a certain number of major league at-bats in the year in question (usually 500). And Wilmer Pino is a mortal lock not to sniff the big leagues in 2007. He'll go about his business this year, hitting .290/.320/.380 or so in Low-A, and, because no one has ever heard of him, that absurd projection will remain unknown to everyone except the readers of this board. And PECOTA will go on trumpeting itself as "deadly accurate."

And here's the really diabolical thing about all this: the only way for a player like Pino to amass the 500 big-league AB required to be counted in the accuracy studies is for him to hit about as well as PECOTA projects, because if he doesn't hit that well, he'll never be allowed to play that much. In other words, because of the way the accuracy studies are done, it's in a projection system's best interest to vastly overrate minor leaguers on the off chance that they happen to have a breakout year and crack the 500-AB mark. This turns PECOTA's greatest methodological flaw into a tremendous advantage.

In summation:
1) Every single study of projection accuracy that we have ever seen is completely meaningless.
2) Projecting a player based on the performance of other similar players is questionable at best and catastrophic at worst. The more really similar players a player has - that is, the more "generic" he is - the more likely it is that he'll simply follow the standard aging pattern. And for players who are less normal, and less likely to follow the standard aging pattern, the similarity scores method won't fare any better, because, by definition, abnormal players have fewer truly comparable players upon which to base a projection. Simply using a standard aging curve and scrapping the comparable players stuff entirely (along with using reasonable MLEs, of course) is a great way to ensure that every single projection you generate passes the smell test.
3) Contrary to what PECOTA would have us believe, Wilmer Pino is not a better hitter than Marcus Giles or Josh Barfield.

End rant.

[wolf213]

Helton fan, can you tell me where you find your RAR and ARB numbers?

[Heltonfan]

I calculate them myself.

I've been meaning to post my projections online someplace... I'll try to get around to that soon.

[wolf213]

Sounds great, I would love to be able to look at them.

[Heltonfan]

ANNOUNCEMENT: projections are up! I think you have to register an account with Google Docs and Spreadsheets in order to view the file, but that's easy enough.

And of course, if there's anyone who I left off the spreadsheet who you'd like to see a projection for, let me know.

[Roxpert]

Thanks, HF. Unfortunately, access to your spreadsheet is being denied even though I have a Google ID and logged in. So something is blocking me.

[Heltonfan]

Okay, it should be accessible now.