Web data scraping

When there isn't an API to connect to, or a database to query from, but you can see the data you want on the website, you turn to data scraping. It may not even be a traditional website, it may be a google sheet, or even a semi-structured PDF. Whatever the source may be, if it is readily available on the internet, it can be scraped and saved in a format that best makes sense. If you want an automated email sent out with scraped data, that is fine. If you want a database filled with values found on a website, that is fine. Do you need a copy of a bunch of images from a site, that is also fine.

Typically I will write custom code to get the data you need either as a one time service, or a scheduled service on whatever cadence you deem necessary. If your organization would like to purchase the code used to scrape the data, that can be arranged. You may even just want to buy the automated process (both the server hosting the automated code) and that can be arranged as well. If not, typically I can write the code to deliver whatever data you need, and we can set a servicing fee for me to maintain the code and ensure that everything continues to work in case of website changes, etc.

I have found python to be a great language for customized data scraping projects, that I have successfully used for several clients in the past. For more of a point and click tool that I could initially build and then deliver and train on, there is Mozenda.

ESPN

I was contracted to use a tool (Mozenda) to systematically scrape ESPN for NFL, NBA, MLB, College Football & Basketball, and NHL data going back a couple years, as well as going forward.

The project went really well, despite ESPN changing their website structure after running smoothly for several months.

Project: Create a deliverable for clients to use, monitor and easily tweak if needed to pull desired ESPN data, and host that data on client's website for users to download.

Job Sites

I built pythonic solutions that ran on Cron jobs to systematically scrape these sites, in absence of an API, to identify new jobs that met certain search criteria on a scheduled basis. Any new data was compiled into CSV files and dropped into clients Google Drive (their requested FTP delivery). This was then hosted on a Digital Ocean droplet (i.e. server), for the client to subsequently own and tweak.

Recommended Tool:

Mozenda

If you want an automated data scraping solution built on a tool that your organization can own, monitor, and visually see how the data is being scraped and stored for you, I would highly recommend using Mozenda. I have used the tool extensively in the past, and is one the quickest efficient ways to scrape web data. I can build the scraping process using the tool and ensure it is working properly before handing it off to your organization to monitor and maintain. Currently the tool is limited to working on a windows machine, so keep that in mind if you want to tweak things after I build the scraping process initially. The support team are a great asset to leverage making this is a viable, albeit more expensive, option to get the data you need.