jay

Curious to hear some thoughts on this... Would it help if defensive measures were designed to present the data in a way that is more similar to more traditional stats? For example, instead of trying to put a potentially convoluted, complicated run value on a player's defense, what if the stat presented it more like a slugging percentage and we called it defensive efficiency percentage?

Thanks for the sunshine!

The same run values are used. There isn't more correlation, it's just arrived at sooner for the following reasons (some of which you've addressed)... On defense, you're dealing with the combination of both smaller sample sizes AND the question of how that expected run value was arrived at. That's where using the word "accuracy" can be a confusing term -- those two issues are distinct and separate. The smaller sample sizes are what is largely being addressed in those statements from the "creators". In order to represent "true talent level" (which is always changing, btw), you need the x years of data. Those statements are not saying that a player's defensive measures need 3 years of data to tell you the expected run value of the plays he made or didn't. The values are what they are, just like a HR binge in June. That's the context in which I say they are "what happened" (and specifically not the same as win probability). That context of those statements is commonly removed and used to discredit defense measure accuracy. Here's the exact quote from FG: "How many UZR opportunities do you need for UZR to be reliable? There isn’t any magic number. If I asked you how many AB you need before a player’s BA becomes reliable, you would likely answer, “I don’t know. The more the merrier I guess.” That is true with UZR and with all metrics." However, the need for a sufficient sample is true across all stats. It takes longer on defense to collect enough data points. That longer period of time doesn't mean the individual events didn't "happen". Offense metrics arrive at a higher confidence level in a shorter period of time due to a higher frequency, in addition to better expected run value accuracy for individual events, which we'll address next. The other aspect is how the expected run values were arrived at. At the base level, both offense and defense metrics start with a binary question -- safe or out. From there, they both assign an expected run value to the event. Offense has the advantage of very distinct outcomes that can be assigned their own values. Defense is inherently at a disadvantage here in that interpretation has to be done to categorize the event in order to assign a run value. A simple safe or out or even 6 to 7 possible outcomes wouldn't tell you much. This is why there are more potential outcomes and values compared to offense that are used to distinguish a catch from a catch (type, location, speed of ball, etc) -- to improve granularity. It's fair to question the accuracy of the categoization methods. The challenges here are what require more data points (which, again, takes more time). Additional granularity in assigning the defensive value (ie Field F/X) will increase the accuracy. This will allow for smaller and smaller data samples to assess reliability and a player's true talent level. It won't change the need for a sufficient sample size.

There's a difference between expected run value and win probability. I think you're ignoring that. Expected run value is the concept used in UZR as well as the offensive metrics. It's the same. Win probability factors the specific scenario of that event in that game. Some offensive stats are designed to use that (WPA), but most don't and nor do the common defense metrics.

The advanced offensive metrics use an expected run value for that event in order to calculate value. There is not inherent value in an offensive event except for a home run. Standing on second base instead of first base doesn't inherently get you more runs. A double is calculated as more valuable than a single because we know it has a higher expected run value. Context of runners on base or game situation is completely ignored (although you fault defensive measures for this). This is the quite literally the exact same concept that is applied to defensive metrics, including multiple possible outcomes exactly like you're talking about. The expected run value is first scaled to the specifics of that exact batted ball based on location, speed, etc. Then, the probability of that catch is applied with a 0-10% probability catch being the best with the most value and so forth along the scale. That unlikely catch prevented the expected run value of that specific event by creating an out (not every catch is the same!) while offensive metrics capture the expected run value generated when an out is not created. It's the same concept, they just mirror each other...

That's like saying a hit is a hit... But we know they aren't all the same just as we know not all catches are the same. I'm not quite sure what your point is here... UZR is wrong for not valuing all catches the same?

A home run has inherent run value so that's a tough comparison, but change that to a double and a single. A double is harder to hit than a single, but it's not a given that it was terribly valuable to the team... same issue.

The forest is still there... Why is it expected that advanced metrics like UZR and WAR will tell you what the player's actual skill level is when other stats don't do that either? The whole "3 years of data" thing gets misused as a way to dismiss defense stats. It's 3 years of data to give a good picture of actual skill level. It still tells us what happened (within the measurement accuracy, which is better than it gets credit for).

Except that's another example of how not to use WAR. Just like defense stats, home runs, RBIs or most other stats... accumulating a 6-WAR season, a +20 DRS, 50 HRs or 120 RBIs doesn't mean you're a "[fill-in any of the above] player". Why does that same characteristic make you distrust UZR or WAR? How many years of home run data would you want to see before saying a guy is a 20 HR player? More than one...

That's exactly right. I don't understand why there's a common expectation or perception otherwise. Even the most traditional stats are that way. That a guy hit 20 HRs in year doesn't tell us everything, but it tells us something. Same for defense stats...

Do defensive stats tell us what happened? Yes, I take that as factual. Do they accurately tell us what happened? That's what we debate. And while I love sushi, I don't love sushi while watching baseball. If I was at an NPB game though... it'd get real interesting.

Sure, in the most literal sense especially for HRs. Offensive counting stats reflect what happened, but even most of that measurement system has some degree of interpretation. Was it a strike or a ball? Was it a hit or an error? Was it a double or advance on throw? Most of them are easy calls, but then some aren't. And we accept that. My point is that defensive stats are doing the same thing. They are telling us what happened. Just as with the stats we are familiar with from yesteryear, there's going to be variation in those numbers and we shouldn't unduly dismiss defensive metrics based on that. Admittedly, there's some higher degree of interpretation, but I don't think it's as high as some skeptics currently believe. Most of the calls are easy, but then some aren't. However, the measurement system is designed to inherently capture and minimize that. We factually know whether or not the ball was caught at the same level of certainty we have on those offensive stats. The interpretation becomes how hard it was to catch and, again, the majority are easy. Even the hard ones though, we can pretty easily recognize it was a hard catch, so whether it's categorized one way and the guy gets a +.7 run value or another way and he gets a +.6 run value, we aren't talking about some giant gulf. If he missed that ball, we know it was a tough play and he's not getting any significant negative run value assigned. The most important factor is it's all grounded in that we factually know what happened in reality.

But isn't this true of even the most well established offensive statistics? Why would we hold defensive metrics to a higher (perhaps, unreasonable) standard? Why would we accept HRs as a stat when Chris Davis can go from 53 in 2013 to a flop of 2014?

While I'd agree with that statement, to be clear, an individual player's UZR and DRS scores specifically exclude plays where the defensive scorer recognizes a shift. That's doesn't mean he (or she) captures a step or three to the right direction as a shift.

To expand on this part a little further, UZR is based on averages from that player's position. Blatant shifts are excluded from the calculation, but if a LF is generally better positioned based on the batter than other LFers... he's going to have an advantage compared to the "average" LFer that is used for the calculations around what you'd expect the guy at that position to catch. (i.e., could explain a 2014 Alex Gordon) Just because he was better positioned in 2014 and had the chance to make some plays that the "average LFer" would've had no chance at, doesn't necessarily mean that's his true talent level. He could be way better than the other guys at his position, could be in a better position to make the play (excluding categorized shifts) or some combination. Simply, it's what happened compared to average. If he saved 20 runs compared to average, we shouldn't penalize him based on "how" it happened just as we wouldn't penalize the utility guy who miraculously got pitches in his wheelhouse and hit 20 homers.

In the most simplest sense, UZR is trying to capture what happened. Just like with offensive stats, what happened may not line up with what you'd expect based on a player's true talent level. For example, look at some of the short-term HR binges in more recent years by a Valencia or a Plouffe. A defensive measure like UZR can tell us that player made a series of crazy plays that had a lot of value. That doesn't mean he'll continue to do that or, to spy's point, even have the opportunity to do so. That summarizes why UZR doesn't capture "true talent level" without significant data samples. Although, to be fair, most commonly accepted offensive stats need good sized samples as well in order to become predictive.

How many pitchers are 6' 9"? Meyer is already an exception to start with...

That seems reasonable to me. It'd be hard to say Pressly or Tonkin are a step-level improvement over Graham if he's healthy. Might as well see if he has enough to offer to be worth keeping longer (or permanently).

Spot on. Even with better input data, there really isn't a better way to score it. We'll never know what would have happened after a different outcome of that exact play. However, there's been enough baseball games played to have a very precise idea of the average of what would have happened.

This is an interesting point, particularly of some teams don't value defensive data in the current state. Could part of the measured gap in defense be due to a wider skill range between best and worst? I think the effect could be very noticeable at "bat-first" positions like LF/RF (hence 2014 Gordon/Heyward WAR), but I'd think less so somewhere like CF/SS.

Any data or specific insights to show why not? I'd assume a 30 pitch outing would be matched with an appropriate amount of rest and, at that point, it just depends on what's harder between warming up and 30 pitches no more often than every other day or warming up for shorter outings in more rapid succession. I'd say the 30 pitches might actually be better.

For mid-inning changes, it can definitely be a challenge and almost becomes a bit of a 'great in theory' type idea in some situations. However, I do think there are others where it can be applied that aren't terribly common in today's game. If it's a starter tiring in the 7th of a one-run game for example, you should be able to get your best (or maybe even 2nd best) guy ready in time to get out of that jam instead of just using your "7th inning guy". You have to get someone ready either way and that's likely to be one of the highest leverage points in the game. Also, there should definitely be room between innings to consider the upcoming lineup. I think someone earlier mentioned a scenario like a lefty heavy middle of the lineup coming up in the 8th with a small lead. Absolutely it makes sense to use a guy like Perk there rather than wait for the 9th to face the 7-8-9 righties just so he can get the save.

I can appreciate that you're at least willing to hear and consider the debate despite being quite skeptical (which, btw, accurately describes every USAF/USA Chief I've ever known). A big trip up seems to be how many runs are being assigned to defense vs other factors such as pitching. To be sure it's clear, some of the math in the thread was too fuzzy and inaccurately used. If runs allowed were simply due to either defense or pitching, I'd be curious to hear what you think that split looks like... (we can make it far more complicated from there)

That's true. Using FIP in this instance is still going to be a far superior method compared to simply applying the positional defensive values to actual runs allowed. The intent is to get closer to the pitching staff's true expected performance. Do you have a better figure to propose? Better yet, do you have a better breakdown of defense vs pitching vs any other variables for assessing runs allowed?

And that stretch assumes a "real" base of only 80. Following up on the math in the other thread, if we split up runs allowed into simply defense and pitching (ignoring luck and other factors), that's saying defense is only something like 27% to pitching's 73%. Feels too extreme. Move that base to 110 and you're closer to a 1/3 to 2/3 split between defense and pitching. That feels about right to me. From there, some combination of measurement error and outliers stretching the gap to 140 seems reasonable. Even at a gap of 140 runs between the best and worst defenses, that's still only calling runs allowed a 40/60 split between defense and pitching. The impact of luck, park factors, sequencing, etc. will then serve to skew the real-world results from whatever that real split is.