We'll work through parallel processing examples on 1 machine with multiple cores and across several machines using:
- iPython cluster
- GearMan (job processing library with Python bindings)
- picloud (cloud-based parallel Python service)
I use these tools for various parallel processing problems, we'll do some worked examples and look at the good and bad points of each approach. At the end of the tutorial you should be able to decide between the popular approaches to run your code in parallel across local and remote clusters.
If we get time I may also demo some other recent high performance python tools that have come to the fore.
The first two topics will be adapted from my PyCon High Performance Python tutorial (http://ianozsvald.com/2012/03/18/high-performance-python-1-from-pycon-2012-slides-video-src/), the next three topics will be new.
I will prepare a small VirtualBox image using Ubuntu with the tools pre-installed, I'll link on this page to the download location about a week before the tutorial. The link will also include a list of the tools you need if you want to install them by hand.