I worked on a project in a Complex Systems class where I wanted to know if there was any value in looking at the network of at-bats in baseball. To create this network, I assumed that if a batter got on base via hit, HBP, or walk, the batter won the at-bat and I drew a link from the pitcher to the batter. If the pitcher got the batter out, I said the pitcher won the at-bat and drew a link from the batter to the pitcher in the network. This created a directed graph that I could run networked statistics on, such as PageRank. I wanted to know how well the rank of a player in summary statistics (ERA, AVG, OBP, WAR, etc) matched up with the rank of the player in PageRank. PageRank, in this context, puts value upon beating other players with high PageRank. So, when playing the Dodgers, getting a hit off of Clayton Kershaw last year was worth more than getting a hit off of Chris Capuano. Did good players overperform or underperform against other good players? Does this have value for predicting playoff success? These were some of my questions as I started my study.

To get results of batter/pitcher matchups, I crawled Baseball-Reference.com. Their Play Index Tool lets you look up the results of any players’ batting/pitching matchups, possibly filtered by year. I wanted to download every at-bat for the year 2013.

To begin with, I’m not sure Baseball-Reference.com wanted me to crawl their records. They have disclaimers against this sort of bulk downloading, but I was using the data for a personal project and didn’t profit from it, so I went ahead. They didn’t kick off my IP as I went about crawling/downloading these matchups.

My code for this project is in Python. I used the screen-scraping package Beautiful Soup.

I first had to grab the usernames for all players in the majors in 2013. I went to this page to get the batters and this page to get the pitchers. Looking at the page source for the pitching page, you notice that the usernames start around line 1727. Download the page source for those pages and use some logic to grab all the usernames for pitchers and batters. Here is my ugly code to parse the usernames.

Once you have the usernames, you’ll want to crawl Baseball-Reference.com to get matchup data from every batter. Unfortunately, a batter’s matchup data (like this for Barry Larkin) creates the same page source whether you filter by year or not. Filtering by year only dynamically changes what is shown on the screen; it doesn’t change the page source, which is what we are going to crawl. So we have to use three steps to get only 2013 data:
-Parse the page source for a batter’s alltime matchups to see which pitchers he ever faced
-For each pitcher, see if that pitcher is in the list of 2013 pitchers
-If it is, crawl ‘http://www.baseball-reference.com/play-index/batter_vs_pitcher.cgi?batter=’+batter+’&pitcher=’+pitcher to get the line related to 2013. Add this line to your statistics that you are keeping.

Here is my Python code to download all at-bats from 2013. You’ll notice that I import urlopen from urllib2 to tell Python to open the webpages of interest. Then I use Beautiful Soup to parse the page source. Throughout the code, I added in lines like “time.sleep(random.random()*10)” from the time package to make the code delay a random amount of time. This kept me from overloading Baseball-Reference.com with requests and hopefully kept me from pissing them off. If you’re interested in using the code, note that you’ll obviously need to change your input/output folders to match your computer.

Hope this helps. I know it’s not 100% complete in its description, but post in the comments if you’re confused in some way.

The Reds Caravan rolled through Bloomington yesterday. This edition of the traveling side-show featured Marty Brennaman, Eric Davis, Assistant GM Bob Miller, new guy Brayan Pena, minor leaguer Tucker Barnhart, and Big Red Machine glue-guy Doug Flynn. After Marty introduced everyone, he opened up the floor to questions. I asked Bob Miller about the status of the Reds’ Analytics efforts. He tried to convey the vast amount of data that the team collects, including over 90 data points for every pitch thrown. The Director of Baseball Research/Analysis is Sam Grossman, who heads a team of three programmers. The team also employs over 20 scouts, which are especially necessary for understanding high school and foreign talent where the data on the player’s performance is sparser/non-existant.

While I appreciate the honest and helpful answer from Mr. Miller, I wonder whether having three people doing analytics for a team that is going to spend $100M+ each year on player payroll is enough. Do other teams have more analytics professionals? The Reds, under the Dusty Baker regime, tended to ignore a lot of largely accepted analytics wisdom:
-Baker consistently batted his shortstop in the top 2 spots in the order, despite the Reds not having an above-average bat playing shortstop
-The Reds left Aroldis Chapman, one of the most dominant pitchers in baseball, to languish in the closer roll for 2 years, where he pitched a total of 135 innings over 2 years, having a minimal effect on the game. Mike Leake, the Reds’ 5th starter, registered 371 innings in those 2 years.
-Baker wanted his hitters to be aggressive at the plate, which lowered their walk rate, sometimes to comical levels. Getting on base is important, and walks are a way to get on base.

I’d like to see the Reds become more cutting-edge in accepting data-driven wisdom that will improve their team’s performance. As a skilled analytics developer, its frustrating for me to see my team frequently mocked by those individuals who work full-time in baseball analytics. Maybe they’ll hire me as a consultant. I can fix them.

Maria wrote a wrap-up of the Bloomington Caravan stop for Redleg Nation. You should check it out here!