Does SIERA underrate Mo like FIP does?

Prospect Profile: Adam Warren
Link Dump: Cashman, Chan Ho, Milo

Nine years ago, Voros McCracken presented his theory of Defense Independent Pitching, or DIPS. In a well-researched bit for Baseball Prospectus, he showed that while a pitcher’s strikeout rate, walk rate, and, to a lesser extent, home run rate remain fairly consistent from year to year, the number of hits they allow on balls in play does not. Using this data, he concluded that, “There is little if any difference among major-league pitchers in their ability to prevent hits on balls hit in the field of play.” This ran counter to conventional thinking, but he brought enough data to the table that his findings were tough to refute.

Fielding Independent Pitching, or FIP, was borne of DIPS. Consistent with McCracken’s theory, FIP considers only a pitcher’s home run, walk, and strikeout rate. This isn’t to say that all balls in play are created equal. Instead, it is to say that balls in play are subject not only to the pitcher, but to the hitter at the plate and the defense behind the pitcher. With those three factors at play it becomes difficult to assign blame. FIP simple views a pitcher through the lens of what he himself controls.

Yet we know that a pitcher does have a degree of control over batted ball type. Some pitchers keep the ball on the ground, like Brandon Webb, while others induce more fly balls, like Jered Weaver. A pitcher’s batted ball tendencies add another dimension to analysis. Recently, Matt Swartz and Eric Seidman of Baseball Prospectus introduced their solution: SIERA, which stands for Skill-Interactive Earned Run Average. It attempts to combine batted ball types, in addition to strikeout, walk, and home run rates, in order to get a better gauge of a pitcher’s true skill.

While the five-part explanation resides behind Baseball Prospectus’s pay wall, they have been fairly up front with the formula. It might seem complicated, but it essentially takes the various factor and runs them through adjustments in order to not only put a value on a pitcher, but scale it to ERA. Peter Hjort of Capitol Avenue Club was kind enough to create a SIERA calculator, complete with pitchers from 2009. But that’s not what I’m interested. What gets me thinking more is how SIERA views a pitcher who has been consistently underrated by FIP.

In the comments of Rebecca’s post about Mariano Rivera’s playoff WAR, commenter CB noted that FIP does not treat Rivera fairly. It consistently estimates his ERA above his true level, and that has become more and more true in recent years. Sure, Mo’s FIP is consistently in the low- to mid-2s, because he doesn’t allow a lot of home runs, doesn’t walk batters, and usually strikes out more than a batter per inning. My question is, does SIERA provide a better look at Mo?

We’ll look at the years 2002 through 2009, since that covers the FanGraphs era.

Until 2007, FIP came much closer to Mo’s ERA than SIERA. His ERA skyrocketed that year, mostly because of a horrendous April (10.57 ERA), so SIERA, which for Mo was consistently higher than FIP from 2002 through 2007, came closer. In 2008 both formulas came to the same conclusion, and finally in 2009 SIERA predicted a lower ERA than FIP. Problem is, neither came all that close. Once again, Mo performed better than any fielding independent metric could have predicted.

(For those wondering, that counts tERA, though it came damn close in 2008.)

Despite the discrepancy, I do not consider this a bug in SIERA. In fact, I consider it a feature. I mean, if a projection system can accurately project god, well, then where do we have go go from there?

Credit: AP Photo/Rob Carr

Prospect Profile: Adam Warren
Link Dump: Cashman, Chan Ho, Milo
  • Andy in Sunny Daytona

    “Vorus McCracken” sounds like an evil pirate.

    • Mattingly’s Love Child

      A space pirate even. Should be a villain on Star Trek.

    • Andy in Sunny Daytona

      I mean, if a projection system can accurately project god, well, then where do we have go go from there?

      You can’t predict God, he’s got a great sense of humor.

      /Duck-billed Platypus’d

    • tommiesmithjohncarlos a/k/a Archimedes Torquemada

      Voros McCracken sounds like a horrible made up alias that John Edwards would use when checking into a hotel to have a forbidden tryst with his baby mama chick on the side.

  • Mattingly’s Love Child

    Antimatter? The god particle?
    /Angels and Demons nonsense

    • radnom

      Huh? Are you thinking of the Higgs boson?

      It has been a while since I read Angels and Demons, but I don’t remember mention of a “god particle”. IIRC their treatment of antimater was fairly accurate, with of course taking some liberties in assuming technology to efficiently create/use it as energy (which is quite common in science fiction).

      The Higgs boson is a theoretical particle which in most “standard models” of physics is believed to be the particle which results in all of the mass in the universe. It has never been observed, but it is hoped that when the Large Hadron Collider finally gets to full power (at CERN, same place as in A&D) that we will finally be able to observe this particle. This will obviously have major ramifications in the field of physics.
      This is actually one of the main goals of the LHC, and when it was first build the media ran with the phrase “the god particle” to describe the Higgs boson.

      I’m not a Dan Brown fan either, but I think you’re mixing up your physics here.

      • Church of the Perpetually Outraged

        His reference per the Dan Brown story is correct, not per science as you noted.

        [Warning Spoiler Alert]

        In the book, the Camerlengo references the God Particle part when he’s confessing to everyone why he committed the crimes. In the movie, Commander Richter(sp)’s character makes the mocking “God Particle” reference right before his self-righteous speech about his “Church”.

        [Warning Spoiler Alert]

        Just one of dozens of changes/mistakes they made in the movie compared to the novel.

  • tommiesmithjohncarlos a/k/a Archimedes Torquemada

    Hang on a sec.


    Just as I suspected. I looked up “statistical outlier” in the dictionary, and yup, there was a picture of Mariano there.

    • Mattingly’s Love Child

      How are we ever supposed to be able to play the games on spreadsheets if there are “statistical outliers” like Mo actually playing the game?!

      Maybe after Mo retires he could give us the mathematical formula for his greatness.

  • bexarama

    Doesn’t FIP generally underrate relief pitchers and closers in general?

    • tommiesmithjohncarlos a/k/a Archimedes Torquemada

      Doesn’t really look like it. Good relievers are guys who fit FIP’s metrics perfectly (strike guys out, don’t allow walks or homers).

      The pitchers most underrated by FIP wouldn’t be situational relievers (who generally attempt to pitch to as little contact as possible), it would probably be sinkerballers who pitch to induce weak contact; guys like Halladay, Wang, Webb, etc.

      Mariano is the exception to the rule as he’s basically a quasi-sinkerball reliever; the rare reliever who doesn’t mind the batter getting the bathead on the ball because you’re not going to do anything other than break your bat with a slow roller back to the mound.

      • bexarama

        Ah, thank you.

        Wow, Bailey was really nasty last year.

  • Matt Imbrogno

    So SIERA is kind of comparable to tRA?

    • Matt Imbrogno
      • Steve H

        Nothing is comparable to Teammate Rallying Ability

        /alex gonzalez’s

        • Matt Imbrogno

          Haha, classic.

      • miketotheg

        killer crickets man. i had my whole office looking around.

  • pete

    i think batted ball data will improve fielding-independent pitching metrics. I do think that there is certainly a correlation between batted ball speed (and LD% vs. GB% vs. FB%) and BABIP. And i do think that pitchers have a degree of control over how well balls are hit. balls are not hit well off of mo, hence his consistently low ERA despite high ball-in-play rates. Pitchers like this are rare, but I do think it’s misleading to call them “statistical outliers” rather than what they are, which is pitchers whose performance trends are not alike to typical trends when it comes to the relationship between results and balls in play.

    • Chris

      One problem is that the batted ball data (as it’s generally presented) doesn’t distinguish between a sharp groundball in the 3.5 hole and Mo’s typical broken bat groundball that’s basically just pitcher’s fielding practice.

      The other problem is understanding how much control a pitcher has on batted ball type. I agree that it should be included, but it needs to be weighted appropriately – which SIERA tries to do (even though it fails for Mo).

      • Don W

        +100 Chris

        All groundballs are not created the same, Mo has so deemed it.

  • Fabio

    Well, next to Mo picture in the dictionary, there is also the explanation:
    “whose performance trends are not alike to typical trends”

    • tommiesmithjohncarlos a/k/a Archimedes Torquemada
      • Fabio

        Yes, I was following up on your joke. Making the point that you shouldn´t complain that something is called an statistical outlier and then present a synonim for it.

      • Fabio

        Hence the also in my first post.

        • tommiesmithjohncarlos a/k/a Archimedes Torquemada

          Ah, gotcha.

          Reply button fail… happens to the best of us.

          • Fabio

            And to me and you too, my friend.

            • JMK the Overshare’s Mystique and Aura


              • Tom Zig

                No then he would have said he invented the blackberry.

                • JMK the Overshare’s Mystique and Aura

                  Al Gore’d?

                  No, that’s the internet.

  • Januz

    I have read a lot of numbers and comparisons of Mariano with current pitchers and Hall of Famers (None of which does this man justice). He is simply the best who ever played the game at his position. His like may never be seen again. Thank God, for the opportunity to have seen him in action.

    • Salty Buggah

      That’s true…unless we put Jobber in TEH EIGHTH!111!!!1 so he can take over for Mariano once he’s done.

      Also, I think you’re forgetting someone named Melvin Croussett

      • tommiesmithjohncarlos a/k/a Archimedes Torquemada

        (eagerly awaits Januz’s deadpan serious staccato parentheticaled response to your silly jovial bundled meme-jokes)

        • JMK the Overshare’s Mystique and Aura

          Attack of the run-on paragraph, of which density in composition is only matched by density in thought!


  • Steph

    The articles on SIERA are actually not behind the BP pay wall. You can check each part of the series out for free.

  • Am I the only Kevin?

    Everyone talks about how Mo induces grounders. I agree, but I think that only describes one unusual quality he has. Another thing that I expect will skew SIERA upwards on Mo is disproportionate number of caught flares, week popups and humpback liners that he induces (ie, BABIP would be abnormally low on flyballs and liners). I would wager that his converted out percentage on fly balls and liners is much higher than average.

    • larryf

      Damn-I forgot about that weak humpback in Arizona that Jeter should have caught. We could have won that series!!

      • Rob in CT

        They would have won, had Mariano not thrown a surefire DP ball into CF. It wasn’t Jeter’s fault (and I’m 90% sure he pulled his quad trying to catch the ball Mo threw away).

        • larryf

          once again…Jeter>>>>>>Mo

      • Warren

        The weak hit was from the highly juiced Luis Gonzalez.
        If there were no juice, it would’ve been a comebacker to Mo instead.

  • Josh

    this articles describes SIERA as the same thing as tRA. They’re not the same thing.

    • JMK the Overshare’s Mystique and Aura

      It does?

      • Josh

        This article describes SIERA as FIP with batted ball data added in. That’s what tERA is. SIERA includes things for declining marginal utility, situational pitching, etc.

        That said, I still prefer xFIP out of everything.

        • JMK the Overshare’s Mystique and Aura

          Forgive me for being obtuse (math isn’t exactly my bag). You’re saying that Joe described SIERA as the same thing as tRA but SIERA contains other important data points (like declining marginal utility, situational pitching, etc.) but those points weren’t mentioned by Joe?

          Is it possible that he simply only discussed some of what he felt were the most critical points of SIERA and those happen to confer with tERA?

          The only mention I saw of tERA in the article was “(For those wondering, that counts tERA, though it came damn close in 2008.)”

          That paragraph seems to note a difference (though it doesn’t expand on what that difference may be) or at least uses tERA in SIERA to use as a point of comparison with FIP.

          • Josh

            That is possible. I just feel as if he did not explain SIERA as clearly as he could have, in terms of how it’s different from other advanced pitching metrics.

  • MikeD

    If I remember correctly, when DIPS first took the stats community by storm, there was some evidence that some pitchers, although not many, were harder to hit, with Pedro Martinez and Mariano Rivera being two that were called out. Seems to me all attempts to come up with new systems have shown the same thing when it comes to Mariano. Even when batters hit the ball, they have less success against him than other pitchers. The conclusion was his cuttter prevented the hitters from hitting the ball with authority. Made sense then, makes sense now.

  • Jared

    Why these systems underestimate Mo is because he is the rare pitcher who CAN “prevent hits on balls hit in the field of play.” His cutter, the godliest of all pitches, induces weak groundballs and pop-ups. His career BABIP against is .276 with 2007 (the year his ERA skyrocketted) being an outlier year at .335. This is why I do not believe in FIP; it fails to recognize that pitchers do have some control (albeit not 100% and variable on the skill of the hitter) to the type of contact he allows on balls in play. SIERA is nice as it tries to account for this, but until I see the final formula (I only saw v.1 which was modified) I cannot comment on how good it does this.

    • Josh

      There’s no reason to discount FIP as a whole because of Mariano Rivera. Mo is a statical anomaly, and thus it makes no sense for your to judge the validity of a stat based on him. FIP is pretty good at what it tries to do, and the average value of a ball in play varies so little for 99% of pitchers that it’s ok for FIP to ignore batted ball types in all situations except for guys like Mo, who are extremely rare.

  • Pingback: What the Good Book says: Yankee pitchers | River Avenue Blues