I didn’t know what web scraping was until last week. Now, I could probably write an entire tutorial on it.
Having received no positive response from the Carleton administration regarding data, I decided to look into web scraping as a way to gather the data we needed from the Alumni Directory. I downloaded a Google Chrome extension called Web Scraper and read through documentation and tutorials to learn how to use it. I then began the laborious process of configuring the Web Scraper for the Alumni Directory, which involved creating a sitemap and ordering different selectors. After many, many trials and errors, I finally got it to work. I performed a query for each job field on the Carleton Alumni Directory and scraped the results into a CSV file, gathering each alum’s address, zip code, major, and class.
After web scraping the entire directory, I combined the CSV files and used OpenRefine to clean up the data. I pretty much spent the entire last week data cleaning, as I combed through both the alumni and student datasets. The student dataset ended up being much more intensive work since many of the data points were not OCRed properly and location fields were not separated by commas, meaning I had to get creative and figure out other ways to splice the students’ location data.
To give an example of the process, I uploaded the CSV file with all the zoobook data from the 90s into OpenRefine. Every piece of data that has been OCRed displays as its own row, meaning that school, student names, locations, and extraneous words are all jumbled up in one column. I first have to go through and use a text filter to search for words relating to schools and remove them from the dataset to the best of my ability. I then use a text length facet to get rid of random characters. Finally, I separate the data using spaces as a separator so each word gets its own column. I then use text length facets to try and separate states and countries from one another. At the very end, I join multi-word city names together and run a Python program that adds spaces between them.
I’ve repeated this process for all the zoobook data from the 80s and 2000s. After the group cleaned up all the data and combined it, I also used the Google Sheets geocoder plugin to add coordinates to the zoobook data.
For the next steps, we will import the data into ArcGIS Online once it has finished geocoding. I will then use ArcGIS Online to create a map and visualize the data in a way that’s easy for the public to understand. I would like to do some sort of time lapse and density analysis if possible. At the very end, we will display our visualizations in ArcGIS StoryMaps.
You can read more about my group’s progress this week here.
