At work we operate a number of Twitter Accounts, and we have been monitoring the number followers each has, on a monthly basis, for a number of years.

The collection of the data – which we normally put into a spreadsheet – has been, up to now, done manually which is, to be frank, a pain.

So, I decided a while ago to set up a scraper for it. I’ve only now got around to doing it – and it has taken me longer than I thought it would.

You can see it here https://morph.io/watty62/Count_twitter_followers although I might clone it from my personal account to a work one. The beauty of using Morph.io is not just that it does the scraping for you, but that you can also pull the data out using an API.

Morph runs using files on Github, so the Scraper.Py file is in my Github account. I’ve added comments throughout the code, so it should be self-explanatory.

Also there you will find some other files. including the ReadMe.md file which I’ve put together , and the followers.csv file which contains an export of the data from our original spreadsheet.

The Readme.Md explains how the code works and how it writes to Morph.io’s SQLite database.In putting together this project I’ve become quite a fan of SQLite and its elegant simplicity (even if I had to change to use Scraperwiki version for Morph),

And while the follower.csv file is not required by the code or Morph, it is there so you can see the data that I import in the first section of the scraper.

The scraper if set to run for the first time live on Monday 1st Sept, and monthly thereafter. I’ve tested it as much as possible but will watch nervously on Monday to see if it works as planned!

Hopefully you will find a use for a modified version of the scraper too!

Scraping Twitter data with Morph.io
Tagged on:             

Leave a Reply

Your email address will not be published. Required fields are marked *