[null, {"document_url": "http://hawknotes.blogspot.com/2012/06/statistics-with-gnuplot-i-correlation.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1314849600000&toggleopen=MONTHLY-1338523200000", "unformatted_src": "https://lh4.googleusercontent.com/-LW6Qofq9bmc/T8voB4BRO0I/AAAAAAAABwk/MUvJt-u3N3E/s690/avg0.png", "src": "https://lh4.googleusercontent.com/-LW6Qofq9bmc/T8voB4BRO0I/AAAAAAAABwk/MUvJt-u3N3E/s690/avg0.png", "formatted_filename": "avg", "alt_text": "Correlation of Runs with batting average"}, null, {"document_url": "http://hawknotes.blogspot.com/2012/06/statistics-with-gnuplot-i-correlation.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1314849600000&toggleopen=MONTHLY-1338523200000", "unformatted_src": "https://lh4.googleusercontent.com/-GEldxZmPCv4/T8voCq1aOwI/AAAAAAAABxE/9AqhoTSFfGo/s690/avg1.png", "src": "https://lh4.googleusercontent.com/-GEldxZmPCv4/T8voCq1aOwI/AAAAAAAABxE/9AqhoTSFfGo/s690/avg1.png", "formatted_filename": "avg", "alt_text": "Correlation of Runs with batting average and linear fit"}, null, {"document_url": "http://hawknotes.blogspot.com/2012/06/statistics-with-gnuplot-i-correlation.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1314849600000&toggleopen=MONTHLY-1338523200000", "unformatted_src": "https://lh4.googleusercontent.com/-NV66pvSY35s/T8voB__ZNoI/AAAAAAAABwg/zxOYh15be4I/s690/avg2.png", "src": "https://lh4.googleusercontent.com/-NV66pvSY35s/T8voB__ZNoI/AAAAAAAABwg/zxOYh15be4I/s690/avg2.png", "formatted_filename": "avg", "alt_text": "Correlation of Runs with average averages."}, null, {"document_url": "http://hawknotes.blogspot.com/2012/06/statistics-with-gnuplot-i-correlation.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1314849600000&toggleopen=MONTHLY-1338523200000", "unformatted_src": "https://lh4.googleusercontent.com/-CB9jMgKo2zw/T8voCNV0lGI/AAAAAAAABws/RrCXv35Rf5s/s690/avg3.png", "src": "https://lh4.googleusercontent.com/-CB9jMgKo2zw/T8voCNV0lGI/AAAAAAAABws/RrCXv35Rf5s/s690/avg3.png", "formatted_filename": "avg", "alt_text": "Correlation of Runs with standard deviation of AVG and RPG."}, null, {"document_url": "http://hawknotes.blogspot.com/2012/06/statistics-with-gnuplot-i-correlation.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1314849600000&toggleopen=MONTHLY-1338523200000", "unformatted_src": "https://lh4.googleusercontent.com/-jnupIwArohY/T8voCfu3eLI/AAAAAAAABw0/W-3uSgc_c5Y/s690/avg4.png", "src": "https://lh4.googleusercontent.com/-jnupIwArohY/T8voCfu3eLI/AAAAAAAABw0/W-3uSgc_c5Y/s690/avg4.png", "formatted_filename": "avg", "alt_text": "Correlation of Runs showing R = 0.814."}, null, {"document_url": "http://hawknotes.blogspot.com/2012/06/statistics-with-gnuplot-i-correlation.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1314849600000&toggleopen=MONTHLY-1338523200000", "unformatted_src": "https://lh3.googleusercontent.com/-UDpltNngpJg/T8voCrntTVI/AAAAAAAABxI/7JsuilUsWiU/s690/obpout.png", "src": "https://lh3.googleusercontent.com/-UDpltNngpJg/T8voCrntTVI/AAAAAAAABxI/7JsuilUsWiU/s690/obpout.png", "formatted_filename": "obpout", "alt_text": "On-Base Percentage versus Runs/Game"}, null, {"document_url": "http://hawknotes.blogspot.com/2012/06/statistics-with-gnuplot-i-correlation.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1314849600000&toggleopen=MONTHLY-1338523200000", "unformatted_src": "https://lh5.googleusercontent.com/-0DfCQ5hKgBY/T8voDH8VFcI/AAAAAAAABxY/zMivmTQvRb8/s690/slgout.png", "src": "https://lh5.googleusercontent.com/-0DfCQ5hKgBY/T8voDH8VFcI/AAAAAAAABxY/zMivmTQvRb8/s690/slgout.png", "formatted_filename": "slgout", "alt_text": "Slugging Percentage versus Runs/Game"}, null, {"document_url": "http://hawknotes.blogspot.com/2012/06/statistics-with-gnuplot-i-correlation.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1314849600000&toggleopen=MONTHLY-1338523200000", "unformatted_src": "https://lh4.googleusercontent.com/-tvhSDTJzAbA/T8voCkRGXuI/AAAAAAAABxA/YnBn00GVi-Y/s690/opsout.png", "src": "https://lh4.googleusercontent.com/-tvhSDTJzAbA/T8voCkRGXuI/AAAAAAAABxA/YnBn00GVi-Y/s690/opsout.png", "formatted_filename": "opsout", "alt_text": "On-Base Plus Slugging (OPS) versus Runs/Game"}, null, {"document_url": "http://hawknotes.blogspot.com/2012/06/statistics-with-gnuplot-i-correlation.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1314849600000&toggleopen=MONTHLY-1338523200000", "unformatted_src": "http://2.bp.blogspot.com/_WZHLzoZCfRI/R5KRm2GisLI/AAAAAAAAAKA/G5SyMQiquh4/S150/pengjay_w1.png", "src": "http://2.bp.blogspot.com/_WZHLzoZCfRI/R5KRm2GisLI/AAAAAAAAAKA/G5SyMQiquh4/S150/pengjay_w1.png", "formatted_filename": "pengjay w", "rendered_width": 150, "rendered_height": 150}, null, {"document_url": "http://hawknotes.blogspot.com/2012/06/statistics-with-gnuplot-i-correlation.html?widgetType=BlogArchive&widgetId=BlogArchive1&action=toggle&dir=open&toggle=MONTHLY-1314849600000&toggleopen=MONTHLY-1338523200000", "unformatted_src": "//2.bp.blogspot.com/_WZHLzoZCfRI/TAPIoeTDrPI/AAAAAAAAAwk/bUDjOM7aaqo/S220-s80/pengjay_w.jpg", "src": "http://2.bp.blogspot.com/_WZHLzoZCfRI/TAPIoeTDrPI/AAAAAAAAAwk/bUDjOM7aaqo/S220-s80/pengjay_w.jpg", "formatted_filename": "pengjay w", "alt_text": "My photo", "rendered_width": 80, "rendered_height": 80}, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]

Sunday, June 03, 2012

Statistics With Gnuplot -- I. Correlation

At work I've been using gnuplot to do some function-fitting for me. In the course of that I came across a page on computing basic statistics with gnuplot, and it got me to thinking about how to apply this to something meaningful — you know, like baseball.

Note: I'm not an expert in statistics. I haven't ever taken a statistics course. If I get something wrong, please correct me gently. Thank you.

That said, it's been often remarked that On-Base percentage (OBP), Slugging Percentage (SLG), and their offspring, On Base Plus Slugging (OBP) are more highly correlated with runs scored than the traditional batting average (AVG). But how do we quantify that? With statistical analysis, of course.

We need data. I went to Major League Baseball's team statistics database and pulled off the AVG, OBP, SLG, OPS, and Runs/Game data from 1996 through 2011. That gave me data for 476 team/seasons, all playing 161, 162, or 163 games. That should be enough data for a start.

First let's look at the relationship between batting average and runs per game. We'll go through this on in reasonable detail. The data top of the datafile, which we'll call runs.dat, looks like this:

The full file is available on request.

Plot out runs per game versus average:

which looks like this: OK. As we might expect, there is some correlation. Runs go up as the batting average improves. How much? We can use gnuplot's fitting routine to see. We'll assume a straight-line fit:

Which produces a couple of numbers:

and the plot The slope of the line is 37.06, which tells us that a change in batting average from 0.260 to 0.270 will add another 0.37 runs per game to a teams scoring (on average). (Note that the fit misbehaves if we get a low batting average, predicting a negative number of runs. That's because this isn't a great model for baseball at all levels. We're only considering Major League Baseball, where most batters are able to at least make contact with major league pitching, and so can be expected to hit above 0.200 the Mendoza Line. Don't worry about that for now, we'll look for better fits later on in this series, if it should continue.)

How good is this fit? One way to quantify a fit is by the sample correlation coefficient, which in our case can be written as

Where x is the data on the x-coordinate of the plot (here AVG), y the data along the y-axis (here Runs/Game), and the brackets mean take the average. Careful authors don't call σ the standard deviation, but I will:

The theory of R is simple. If R = 1, then all of the data in the last plot would fall on the line, and the line would slope upward. Then AVG and RPG would be perfectly correlated. On the other hand, if all the data fell on the line, but the line sloped downward, than R = -1, and AVG and RPG are prefectly anti-correlated. And if R = 0, there would be no correlation either way. So the closer |R| is to 1, the better AVG and RPG are correlated. (Standard disclaimers apply.) If R = 1 in the above plot we'd be able to perfectly predict how many runs a team would score if we just knew the team batting average. So what is R here?

To find R we'll need to find a lot of averages. Fortunately gnuplot is up to it. Suppose we wanted to fit the data in the last plot to a horizontal line. The formula for that is just f(x) = constant, and constant would just be the average value of the y (RPG) data. We can do the same thing with a vertical line for x. So to get the averages for AVG and RPG we write

The (1.0) in the fit routine tells gnuplot that the x-variable is a constant equal to 1. Actually it's a dummy. We don't care what x is, but we have to tell gnuplot something. The :1 or :5 tell gnuplot to get the y axis data (the functional values) from the first or fifth columns.

Which gives us output:

We can judge the reasonableness of this with a little addition to the plot:

(Using graph 0 on the x-coordinate has never worked for me in gnuplot.)

Which gives us a plot that looks like this: The standard deviations work similarly, we're just averaging things in one dimension again:

The $1 and $5 in the above tell gnuplot you want to use the data from column 1 and column 5 in mathematical formulas. If we just used 1 or 5 here, gnuplot would interpret them as numbers.

All that done, here's a busy plot with the standard deviations marked off: Finally, getting <(x - <x>)(y - <y>)> is a little trickier, since it's a fit to two variables. Fortunately, gnuplot can do that. The only trick is that we have to supply four parameters for a two-dimensional fit. The forth column is an error estimate, which we'll take to be one. Note that we have to also define a new functional.

Now we can compute the r factor, and print it onto the graph:

And here's the final plot, which shows that the correlation factor is 0.814. Good, but not perfect: Here's the entire script in one place, just to make it easier for me to cut and paste, and to annotate it:

And now for the heart of the matter. Let's use the same procedure to look at the correlation between On-Base Percentage and Runs: R = 0.903, somewhat higher than the AVG/RPG correlation.

How about pure slugging: Here R = 0.9026, where above R = 0.9031. OBS and SLG correlate equally well with Runs per game.

So what about everybody's favorite one-number way to evaluate a player? Here R = 0.952. So our conclusion is that OPS is very well coordinated with Runs scored per game. Better than OBP or SLG, and far better than the batting average.

Now we could do all of this with fancy statistical packages, but for simple stuff like this we can do it all with gnuplot and see everything graphically.

Now that's the story for a very simple set of statistics. What about something like Runs Created? Is it really correlated with Runs per Game? Next time …

Posted by rcjhawk at 6/03/2012 06:52:00 PM

Labels: Baseball, gnuplot, Mathematics, Software, Statistics


birdhouses said...

Just remember that correlation does not imply causality. Nice analysis.

1:53 PM
rcjhawk said...

True. However, we've got a few clues which indicate there is some causal relationship:

1) If you don't get on base, you score no runs

2) If you get a lot of people on base, you score runs as they walk in. If N+6 batters come to the plate in an inning, at least N runs score. If 4 batters come to the plate in an inning, one may score.

3) Hits, especially extra-base hits, score runs more quickly than walks do. E.g., in the 4 batters in the inning scenario, above, a home run scores the run, while a walk does not.

So there is strong evidence that OBP and SLG create runs. There's also some cross-talk, of course. Every at-bat that improves your slugging average automatically improves your OBP.

The question that needs to be answered is the relative value of walks, hits, doubles, stolen bases, etc. Runs created formulas attempt to answer some of that. Eventually I'll get to that, I promise.

5:46 PM
 email: email@example.com
 My Stuff What You Were Probably Looking For Subscribe To L&T Not Bored Enough?

Follow @rcjhawkku Notable Web Links Skeptical? You Should Be Useful Links: Linux