Taming the Python – geting started programming on your Raspberry Pi

Posted on 18 June 2013 by ian

Last Christmas I got a Raspberry Pi as a present. As you may know it comes with the Python programming language as standard.

As a former programmer, who started on punch cards at school, wrote some Sinclair Basic for ZX81, and Spectrum, then Pascal, used VBA extensively, then Visual Basic / classic ASP, and finally a little .net, I was intrigued and hooked on this widely used, flexible, easily-extended and very powerful language.

So I set out to teach myself how to use Python.

My first port of call apart from the official documentation at the site above, was ebooks. You’ll find half a dozen free and useful ones  here.

Then I followed up with CodeAcademy. They have a great free Python track. I completed the whole track within a few weeks.

Most recently I finished in Coursera’s Introduction to Interactive Python. This was a challenging course, conducted over 9 weeks, in true MOOC style, using a GUI environment for user interaction, and culminating in programming a game of asteroids. It featured high quality video lectures, quizzes, stretching mini-projects, peer making of work and a very supportive online forum.

I cannot recommend this highly enough but would encourage anyone doing that to prepare for it using the CodeAcademy route first otherwise it could be a steep haul. The course will next be run in October 2013 – and you can sign up for it now.  I have to thank  the guys at Rice University, particularly Joe Warren and Scott Rixner, for a great experience – all the more amazing considering that this was absolutely free (unless you want certification ).

Finally, I can recommend for those seeking to move on from these online courses to look at the recent O’Reilly Publication – Python Cookbook which will really help you improve your skills.

Happy python programming!

Ian

 

Posted in MOOC, Programming, Python, Raspberry Pi | Tagged , , , | 1 Comment

Facts Are Sacred – a review

Posted on 6 May 2013 by ian

Facts are Sacred is the new book by Simon Rogers, the award-wining editor of guardian.co.uk/data and a news editor on the Guardian, working with the graphics team to bring figures to life on the page. He was closely involved with the Guardian‘s ground-breaking decision to crowdsource 450,000 MP expenses records, as well as the organisation’s coverage of the Afghanistan and Iraq ‘Wikileaks’ war logs.

The book, which is available in hard cover and a very inexpensive kindle edition, describes the changing world of data journalism, touching on big data, open data and citizen hacktivism.

Simon describes the methods and approaches taken by his colleagues on the Guardian, and shows how everyone can get involved in creating, analysing and visualising data.

It is clear that in the last four years things have changed dramatically. Governments, their agencies and local authorities have all started to provide open data with varying levels of commitment, standards and approaches; but the fact that they have these limited made inroads is positive and significant and we need to press to make open data and transparent government the norm. Nor does it mean that the job is done. The challenge remains for journalists and citizens alike in contextualising the data, analysing it, checking accuracy and uncovering what the data tells us.

That exposes a need for more, not less, citizen involvement, which itself necessitates better skills in data analysis, understanding statistics and being able to paint a picture (or at least create an infographic) with the data exposed.

Using examples from the 2011 riots, hurricane Sandy, MPs expenses, and more Rogers tells an engaging story of how the Guardian, in particular, interacted with its readers, challenged the government, used open source tools, and broke new ground in not only using data to source new stories, but also in starting to lay the foundations for live data reporting and analysis. This is something that could not have been imagined only 10 years ago, nor when CP Snow, celebrating 50 years as the Guardian’s editor, wrote in 1921 “Comment is free but facts are sacred” from which the title is drawn (let alone when the Guardian’s own first edition in 1821 carried some tabular data about Manchester schools).

This is an excellent book, which takes the pulse of data journalism as it stands in this early phase of open data. It offers us all a chance to develop our skills in data analysis and citizen journalism and reminds us that we all can hold authorities to account, and collaborate to develop further tools and crowd-sourced analysis.

Highly recomended!

Ian Watt6 May 2013

Posted in Collaboration, Data Visualisation, Hacking, Open Data, Open Government, Uncategorized | Tagged , , , , | Leave a comment

Aberdeen Cultural Hackathon 4th – 5th May

Posted on 8 April 2013 by ian

Aberdeen’s newest Hackathon is all set to go ahead! Mark your diaries for 4th and 5th May. Details of how to get involved in this coding weekend with a difference are below.

After several weeks of trying to agree dates with potential partners for a two-centre Open Data Hackathon in Aberdeen and Edinburgh, and stumbling over logistics, I finally had to abandon the original plan and alter my approach.

I changed the concept to a Culture Hack and pitched it to the bid team for the City of Culture 2017. Meeting with them, and Dr Bruce Scharlau of Aberdeen University,  it was obvious from the start that we had a commitment to get this of the ground, which makes me so pleased.

The following is taken from @aberdeen2017‘s latest news letter:

We are hosting a Cultural Hackathon at Seventeen over the first weekend in May.

The hackathon will see computer programmers brought together with artists, musicians, designers and writers in a competition to create something around one of the key themes of the UK City of Culture bid.

We will release full details of our themes on the day of the bid submission.

We have no idea what will be created.  Previous hackathons have resulted in new apps, new software and new ways of doing things on the web.

This is the first cultural hackathon we’ve had in Aberdeen and we’re looking for some creative people willing to spend a weekend in Seventeen experimenting away.We’ll provide refreshments, internet access and some top class prizes.

Programmers, writers, poets, designers or artists who might be interested in working on the hackathon should contact us at 2017@aberdeencity.gov.uk.
Does anyone else have  burning desire to work on some machine-generated poetry based on the social media outpourings of local citizens? Why not join me there?
Posted in aberdeen2017, Coding, Collaboration, Creativity, culture, Hacking, Innovation, Open Data | Tagged , , , , , , | Leave a comment

Adventures with a Raspberry Pi – Part 4

Posted on 11 January 2013 by ian

If you’ve read the previous parts of this article you’ll find I have set up my Raspberry Pi by formatting the SD card and installing the Raspbian OS, setting it up on my home network and assigning it a static IP address , and most recently configuring the PI with VNC so that it could be operated remotely and headlessly.

Thinking about the effort that has gone into the setup and the time that it has taken, I was starting to get concerned about the chance that the SD card, which contains the Raspian OS and all the Pi’s data might, at some point go phut!  Or what might happen if I was to try adding another package and screw thins up.

So I decided to back-up my SD card. There are two ways to do this: from within Raspian – using Linux commands, or to do it in Windows.  Being very rusty with Unix / Linux I took what I considered the safe way out and did it in Windows. And it worked like a dream.

I followed the instructions on this blog and used Win32diskimager ( which you might recall I used before in my initial set-up)  this time to read the SD card (which I’d taken out of the Pi and put in Windows’ SD card reader). It then creates an image file where you want it. I stuck mine on my Windows hard drive. The key thing, as the blog there says, is to remember that you use Win32diskimager to READ the image.

I then used Win32diskimager again but this time to WRITE the image from my Windows hard drive back to a spare SD card. I then put that card back in my Pi, booted from it and everything was there, working perfectly. Because this second SD card was larger I then used the following command to launch the set up menu:

sudo raspi-config

I then chose the second menu option to expand the root partition to fill the SD card.

So I now have two SD cards with copies of my set up, and a separate image on my hard drive that I can burn at any time to get back to where I was with the Pi yesterday.

 

Posted in Hacking, Linux, Raspberry Pi | Tagged , , , , | Leave a comment

Adventures with a Raspberry Pi – Part 3

Posted on 10 January 2013 by ian

I wrote in my last post about how I got SSH working on my Pi allowing me to connect remotely and run command line tasks on the Raspberry Pi.

The next step was to set up a means of running a remote desktop session from another PC, running any mainstream operating system be it Windows, Mac or Linux. This is best done using Virtual Network Computing (VNC) protocol.

There are many tutorials out there with instructions on how to install various VNC packages on the Pi’s Debian-based operating system. Most seem to favour TightVNC as the package for the Pi.

I chose the instructions on the Penguin Tutor site - but I could equally have chosen those on the Adafruit site as both seem to be equally popular in terms of people recommending them.

Getting your system configured should be a simple case of following the instructions (carefully) and it will work. I had some problems getting my Windows 7 laptop seeing the VNC client on my Pi but I think that was because I wasn’t careful in checking each step I followed.

Anyway, now have TightVNC installed on my Pi (per the Penguin Tutor instructions) and loading each time it boots up. I haven’t gone as far as securing access, as at present I am just running it on my own local home network.  It was when I started securing it that problems arose for me – but I am sure they were of my own making.

I’m also using VNCViewer (as recommended on the Adafruit tutorial) as my Windows client rather than the one recommended in the Penguin tutorial.

But the end result is that I am sitting now, at my Windows laptop, connected via VNC to my Raspberry Pi, using the Midori browser on the Pi to connect to my blog and write this post.

With VNC and SSH now up and running, I could in theory remove the Pi’s mouse, keyboard and monitor and use it entirely remotely.

Posted in Hacking, Linux, Raspberry Pi, Uncategorized | Tagged , , , , , , | Leave a comment

Adventures with a Raspberry Pi – Part 2

Posted on 2 January 2013 by ian

Having got my Raspberry Pi up and running, I decided to see what else I could do with it. How easy would it be, for example, to administer the Pi remotely? If that worked then it could be set up to run with no monitor, keyboard or mouse, making for less clutter on the desktop.

It turned out that this is surprisingly easy by using Secure Shell (SSH). There are loads of articles on how to do this so rather than put the full details here, I’ll outline the steps and provide some links below.

  • In most cases, when a computer joins a network it sends a request to the router for an IP address. The router uses DHCP to identify and issue the next free one. So each time you reboot the Pi  can be assigned a new address. This makes connecting remotely challenging.  So, the first step is to have your Pi use a static IP address on your home network. This will stay the same each time it connects. I’d suggest that you follow the steps here which are clear and work well.
  • Keep a note of the static IP address that you chose.
  • The next steps are summarised from those on the very useful AdaFruit site.
  • Using the LX Terminal, and enter the following command: sudo raspi-config which will bring up the configuration screen that you will have seen at first startup. Click on the SSH option, then choose enable. The Pi may have to be rebooted.
  • From another terminal window run the command ifconfig This will show your IP address and other network connection details.
  • Now go to the client machine from which you want to use to connect to thePi and set up a client. For a Windows machine you might want to use a client such as Putty.
  • Set up your client to connect to the Pi at the IP address you noted earlier. Log in with the user name Pi and your normal password.
  • You can now run commands on the Pi from your other machine. Try typing “ls -l” or “cd ..” without the quotes.

Now that I have that working I intend to try getting the desktop environment accessible from another computer using VNC or similar.

 

Posted in Linux, Raspberry Pi, Uncategorized | Tagged , , , | Leave a comment

Adventures with a Raspberry Pi – Part 1

Posted on 31 December 2012 by ian

For Christmas 2012 I received from my wife a Raspberry Pi – a credit card-sized PC which costs little more that £30 on its own. She had kindly bought it bundled with peripherals including a power source, a powered USB hub, keyboard, mouse, HDMI cable and an SD card which had the Raspbian OS pre-installed on it (similar to this kit).

This was a great start and meant that I could boot it up straight away. Before that, I installed the Pi in a case (similar to this one). This is important as it protects the computer from dust and stops the board flexing (eg when inserting and removing the SD card).

Raspberry Pi Photo

I plugged everything in (connecting it via the HDMI output to a monitor) and got it running straight away. The first time it boots up it opens a configuration menu. I read the  leaflet which had come with my kit and set the various options. The final option was to update the operating system to the latest version (which at the time of writing was 2012-12-16-wheezy-raspbian). Had I read a decent book or article on this I would have left the OS as it was at that point and gone into the LXDE desktop environment.

Not knowing that, I started the upgrade in a rush and didn’t allow it to complete, I now realise. So thinking it had finished I manually restarted the system by disconnecting and reconnecting the power (which is never recommended). The result was a corrupt OS on a card – and no idea how to fix it.

Trying to do work on it in Windows was a non-starter as the SD card had previously been partitioned by the installation of the Raspbian OS, so Windows could only see one partition and thought that my card was 78Mb rather than 4Gb. The best tip I found was to put the SD card in a digital camera and reformat it – and it worked. That meant I now had a 4Gb card again.

The next challenge was to re-install the Raspbian OS. I downloaded the img file from the Raspberry Pi site and saved it to my windows machine. Then using the instructions here, I  burned the Raspberry Pi OS image to an SD card. I chose the ‘Easy Way’ set of instructions which used Win32DiskImager to do the job. It went really straight-forwardly.

That done, I re-inserted the SD card into the Raspberry Pi and rebooted it. It worked straight away, and since the image I’d used was the latest downloaded from official site it didn’t require an upgrade to the OS.

In the next part I set up the Raspberry Pi on my home network including assigning it a static IP address and configuring SSH so that I can interact with it from other computers on the network.

 

Posted in Linux, Raspberry Pi, Uncategorized | Tagged , , , | Leave a comment

Use IFTTT & Readitlater to mine info from Twitter

Posted on 2 February 2012 by ian

While the following post uses #opendata as an example subject to pull information from Twitter, the methodolgy is generic.

The background is that I love to know what is happening in the world of Open Data, but there are so many people doing so many things in a diffuse environment that it is difficult to know where to look in order to keep up with the latest developments.

Separately, I’ve been playing with IFTTT (If This Then That) which is an incredibly powerful way of plumbing web accounts together to do some clever things.

So, here is an experiment that I’ve been playing with. If you want to follow it, to make your own project then you’ll need three things:

All are free, and the links above will take you to the sign-up pages. You can be up in running in minutes. Once signed up, you can follow the steps below.

My first task was to create task in IFTTT to search all or Twitter and find who is tweeting about #opendata. Luckily someone had already set up a similar task in IFTTT.  On that site, shared tasks are called recipes, and you can copy a pre-existing recipe.

Here is the recipe you want: http://ifttt.com/recipes/7540

Once you’re logged into IFTTT, just follow the recipe, and, under Step 1, enter the search term you’re interested in. In my case that was #opendata. Then, under step two, leave the first field as suggested then go to the second field and enter the name of the Twitter list you want to use.

Once you click Create Task, the event will fire every 15 minutes and each time, if it finds a new user tweeting about #opendata it will add them to your Twitter list.

So, you’re now building up a list of people who tweet about a specific topic – whcih might in itself be useful for those setting up hashtags for events, hackdays or conferences.

At this stage, before we move one, you might want to go to your Twitter account and add a user manually to the list just so there is something there to work with.

Now that that is set up and working you can set about searching the Twitter stream for that list and pulling out what people are saying about Open Data.  To do that you need to grab the RSS feed of the list. This is more difficult than you might expect – and certainly harder than it used to be with the ‘old’ Twitter.

However, someone has been through the pain for you already. You’ll find the details here on David Calhoun’s blog.

I found the method suggested by Pete Carrol on that page the most straightforward.

Go to a link such as this: http://api.twitter.com/1/lists.xml?screen_name=scibella

but change the screen_name to your own.

From that you’ll get back some XML with one or more <list> items. If there is more than one <list>, choose the ID of the one you want.

then use this link:

http://api.twitter.com/1/lists.xml?user_id=51496296 but substituting the ID for the one for you list

That is the feed for your Twitter list identified. Now, copy that link and head back to IFTTT and to another recipe – this time one that I created to search a Twitter list for a specific hashtag, and push the results to a Readitlater account. Here it is: http://ifttt.com/recipes/19356

Follow the instructions there and you’re done.

You’ve now be searching for the hashtag that you’re interested, from the Twitter list you set up to monitor who chats about it on Twitter. The results will be available in your Readitlater account.

Since I set this up yesterday the experiment has worked. My Twitter list of those who’ve tweeted about #opendata is growing.  Also, the task search the Twitter list for mentions of #opendata is working, and the tweets have been pushed to my Readitlater account allowing me to read them on my mobile phone. That said the IFTTT log shows that its been throwing up some errors as API seems to have failed a few times over night. I’ll have to keep an eye on it.

Also, you could adapt the latter recipe and push the alerts to mail, SMS or to other clipping services using the built in channels in IFTTT.

If you find this useful – or have comments on it, please leave comments below.

Ian

 

 

 

Posted in Coding, Hacking, IFTTT, Open Data, Open Government, Readitlater, Twitter | Tagged , , , | 1 Comment

Webteam Survey 2011 – The – Graphs Part 2 of 3

Posted on 24 December 2011 by ian

This is the second part of an article, which begins here.

The scatterplot diagram below shows the same data as in part one of this article but with the data for Essex removed for ease of plotting. Doing so means the spread of data is more easily visualised.

Webteam Survey 2011 – Essex eliminated

One might have expected a fairly linear distribution of data from bottom left, sloping upward to the top right somewhere, with bubble sizes (representing web team sizes) starting small at the bottom left and getting bigger as we move right and up. However this is not true. One of the largest bubbles is for Fylde council, down in the bottom left.

If you now move to the third part of this article, I’ll plot only the data for the comparator group that I selected for Aberdeen City Council.

Posted in Benchmarking, Coding, Data Visualisation | Tagged , , | Leave a comment

Webteam Survey 2011 – The Graphs – Part 3 of 3

Posted on 24 December 2011 by ian

This is the third part of an article that starts here, and continues here.

The Scatterplot diagram below shows only the data for the Aberdeen City Council and its six comparator councils. These were chosen as they are local authorities having both a local population and council FTE size within +/- 25% of that of Aberdeen.

So the clustering is understandable – but the the variation of team sizes, reflected in the bubble size is remarkable. Watch out for Falkirk and South Tyneside whose bubbles are almost exactly aligned. if you can’t see one or other trying refreshing the page (F5) as a jitter function moves the dots around a little.

Webteam Survey 2011 – Aberdeen Comparator group only
Posted in Benchmarking, Coding, Data Visualisation | Tagged , , | Leave a comment