The purpose of this project is to scrape historical MMA data on fights and fighters, clean the data, and create new feature variables to make it as useful as possible. Project was written in R, using package rvest for scraping.
0-wiki_ufcbouts.Rscrapes the results of every UFC fight from Wikipedia (or updates an existing fight DB).1-wiki_ufcfighters.Rscrapes the details of every fighter that has ever fought in the UFC and has a Wikipedia page. Fighter details include: age, height, reach, wins/losses, nationality, team/camp association, etc.wiki_ufcbouts_functions.Rhouses all custom functions used in ufcbouts scrape file.wiki_ufcfighters_functions.Rhouses all custom functions used in ufcfighters scrape file.
Save folder mma_scrape to your current working directory. There are two scrape files
(0-wiki_ufcbouts.R and 1-wiki_ufcfighters.R), file 0 must be run prior to running
file 1. The two function files provide functions used for scraping and are
sourced at the top of each of the scrape files. The output of each scrape file is
a dataframe saved as an .RData file to the folder mma_scrape. If the ufcbouts scrape
file has been run in the past and the output .RData file exists in the directory mma_scrape,
then running ufcbouts again will ONLY scrape new fight records that have been added to
Wikipedia since the last time the script was run, appends the new records to the
boutsdf dataframe and saves it back to the same .RData file.
The majority of the code is performing text clean up, text extraction, tidying variables and creating new feature variables. I'm planning on add more to this in the near future (scraping historical judging data for all MMA fights, merging of datasets).
Bout Results File 0-wiki_ufcbouts.R:
str(boutsdf)> str(boutsdf)
'data.frame': 4033 obs. of 28 variables:
$ Weight : chr "Featherweight" "Lightweight" "Welterweight" "Flyweight" ...
$ FighterA : chr "Yair Rodriguez" "Joe Lauzon" "Ben Saunders" "Sergio Pettis" ...
$ VS : chr "def." "def." "def." "def." ...
$ FighterB : chr "B.J. Penn" "Marcin Held" "Court McGee" "John Moraga" ...
$ Result : chr "TKO" "Decision" "Decision" "Decision" ...
$ Subresult : chr "front kick and punches" "split" "unanimous" "unanimous" ...
$ Round : num 2 3 3 3 3 3 1 3 3 2 ...
$ Time : chr "0:24" "5:00" "5:00" "5:00" ...
$ TotalSeconds : num 324 900 900 900 900 900 177 900 819 461 ...
$ Event : chr "UFC Fight Night: Rodriguez vs. Penn" "UFC Fight Night: Rodriguez vs. Penn" "UFC Fight Night: Rodriguez vs. Penn" "UFC Fight Night: Rodriguez vs. Penn" ...
$ Date : Date, format: "2017-01-15" "2017-01-15" "2017-01-15" ...
$ Venue : chr "Talking Stick Resort Arena" "Talking Stick Resort Arena" "Talking Stick Resort Arena" "Talking Stick Resort Arena" ...
$ City : chr "Phoenix" "Phoenix" "Phoenix" "Phoenix" ...
$ State : chr "Arizona" "Arizona" "Arizona" "Arizona" ...
$ Country : chr "U.S." "U.S." "U.S." "U.S." ...
$ champPost : chr NA NA NA NA ...
$ interimChampPost: chr NA NA NA NA ...
$ wikilink : chr "https://en.wikipedia.org/wiki/UFC_Fight_Night:_Rodr%C3%ADguez_vs._Penn" "https://en.wikipedia.org/wiki/UFC_Fight_Night:_Rodr%C3%ADguez_vs._Penn" "https://en.wikipedia.org/wiki/UFC_Fight_Night:_Rodr%C3%ADguez_vs._Penn" "https://en.wikipedia.org/wiki/UFC_Fight_Night:_Rodr%C3%ADguez_vs._Penn" ...
$ over1.5r : num 0 1 1 1 1 1 0 1 1 1 ...
$ over2.5r : num 0 1 1 1 1 1 0 1 1 0 ...
$ over3.5r : num 0 0 0 0 0 0 0 0 0 0 ...
$ over4.5r : num 0 0 0 0 0 0 0 0 0 0 ...
$ ITD : num 1 0 0 0 0 0 1 0 1 1 ...
$ r1Finish : num 0 0 0 0 0 0 1 0 0 0 ...
$ r2Finish : num 1 0 0 0 0 0 0 0 0 1 ...
$ r3Finish : num 0 0 0 0 0 0 0 0 1 0 ...
$ r4Finish : num 0 0 0 0 0 0 0 0 0 0 ...
$ r5Finish : num 0 0 0 0 0 0 0 0 0 0 ...
Fighter Details File 1-wiki_ufcfighters.R:
str(fighters)Classes 'tbl_df', 'tbl' and 'data.frame': 1235 obs. of 45 variables:
$ Name : chr "Aaron Brink" "Aaron Riley" "Aaron Rosa" "Aaron Simpson" ...
$ Current Division : chr "Heavyweight" "Lightweight" "Light Heavyweight" "Welterweight" ...
$ Total Fights : num 52 45 23 17 22 30 11 17 89 36 ...
$ Total Wins : num 26 30 17 12 15 20 6 14 56 28 ...
$ Wins By knockout : num 21 6 6 6 5 6 3 8 7 13 ...
$ Wins By submission : num 5 13 4 1 4 6 3 1 38 3 ...
$ Wins By decision : num 0 11 7 5 5 8 0 5 8 12 ...
$ Wins By disqualification : num 0 0 0 0 1 0 0 0 0 0 ...
$ Wins Unknown : num 0 0 0 0 0 0 0 0 3 0 ...
$ Total Losses : num 25 14 6 5 6 9 5 2 29 8 ...
$ Losses By knockout : num 6 7 3 3 1 2 2 1 9 2 ...
$ Losses By submission : num 18 2 2 0 3 1 2 1 16 0 ...
$ Losses By decision : num 1 5 1 2 2 6 1 0 4 6 ...
$ Losses By disqualification: num 0 0 0 0 0 0 0 0 0 0 ...
$ Losses Unknown : num 0 0 0 0 0 0 0 0 0 0 ...
$ No Contest : num 1 0 0 0 1 0 0 0 0 0 ...
$ Draw : num 0 1 0 0 0 1 0 1 4 0 ...
$ Born : chr "1974-11-12" "1980-12-09" "1983-05-28" "1974-07-20" ...
$ Height in Inches : num 75 69 75 72 68 71 74 74 68 70 ...
$ Reach in Inches : num 75 69 77 73 70 NA NA 76 NA 72 ...
$ Team : chr "The Arena" "Jackson's Submission FightingAmerican Top Team (formerly)" "Team Punishment" "Power MMA Team" ...
$ Trainer : chr NA NA NA NA ...
$ Weight in Pounds : num 203 155 204 170 155 ...
$ Division : chr "Light Heavyweight (formerly)Heavyweight (current)" "LightweightWelterweight" "Light Heavyweight (205 lb) Heavyweight (265 lb)" "WelterweightMiddleweight" ...
$ Other names : chr NA NA "Big Red" "A-Train" ...
$ Rank : chr NA NA NA NA ...
$ wikilink : chr "https://en.wikipedia.org/wiki/Aaron_Brink" "https://en.wikipedia.org/wiki/Aaron_Riley" "https://en.wikipedia.org/wiki/Aaron_Rosa" "https://en.wikipedia.org/wiki/Aaron_Simpson_(fighter)" ...
$ Years active : chr "1998-present" "1997-2013" "2005 - present" NA ...
$ Fighting out of : chr "San Diego, California" "Albuquerque, New Mexico, United States" NA "Phoenix, Arizona, U.S." ...
$ Notable relatives : chr NA NA NA NA ...
$ Residence : chr "Roseville, California" NA "San Antonio, Texas" NA ...
$ Style : chr NA "Boxing, Wrestling" NA "Wrestling" ...
$ Nationality : chr "American" "American" "American" "American" ...
$ Notable school(s) : chr NA NA NA "Antelope Union High School" ...
$ Stance : chr "Orthodox" "Southpaw" NA "Orthodox" ...
$ University : chr NA NA NA "Arizona State University" ...
$ Wrestling : chr NA NA NA "NCAA Division I Wrestling" ...
$ Website : chr NA NA NA NA ...
$ Teacher(s) : chr NA NA NA NA ...
$ Ethnicity : chr NA NA NA NA ...
$ Notable students : chr NA NA NA NA ...
$ Children : chr NA NA NA NA ...
$ Spouse : chr NA NA NA NA ...
$ Occupation : chr NA NA NA NA ...
$ Died : chr NA NA NA NA ...