Yesterday’s London Python Dojo

Yesterday was the March Python Dojo, hosted as usual by the ever-generous Fry-IT, with a book donated by O’Reilly. We started with a couple of not-so lightning talks from Tom Viner — talking about his team’s solution for last month’s puzzle — and Nicholas Tollervey — talking about bittorrent. An artfully-worded late question had @ntoll on his soapbox for a while on the subject of copyright and payment to artists, until someone spoiled it by suggesting that maybe we ought to write some code in Python!

After the usual, only slightly convoluted, voting experience, we decided to pick up one of last month’s runner-up challenges: creating a compression-decompression algorithm. Naturally most people started from some kind of frequency table, replacing the most common items with the smallest replacement. The approaches ranged from a hybrid Huffman-UTF8 encoding to an attempt to replace common words by a $n placeholder, where the n would increase as the word became less common. The winner for the most optimistic approach was a lossy algorithm which dropped every other word on compression, replacing it on decompression by the most likely from a lookup table. Tested against a corpus of Shakespeare’s works it produced some quite readable poetry.

As an aside, I can assert after a wide-ranging survey that (a) the preferred editor background is dark (black or dark-grey); and (b) in spite of all the tech at their fingertips, programmers still reach for pen and paper when they need to work something out!

The Great Thing About Open Source Is…

(No, this isn’t one of those New Year memes; it’s just a bit of a verbal ramble).

I wanted to get an app up-and-running on WebFaction. And, for reasons not necessarily entirely connected with reality and hard requirements, I decided to go for Python3. WF offers CherryPy 3.2 against Python 3.1 but — again, for no very good reason — I decided I wanted the very latest.

So: download cherrypy.latest.tgz and python.latest.tgz. Dum de dum, configure –prefix=$HOME && make && make install. Bit of fiddling about with WF’s cherrypy setup and… we have a problem. The cherrypy engine is, for some reason of its own, using threading._Timer rather than threading.Timer and it’s not there any more in Python 3.3.

Investigation shows that this was fixed in cherrypy a few months ago. Meanwhile things will probably go ok with Python 3.2. What to do? I decided on backtracking on Python (truth be told, I hadn’t checked the cherrypy logs at this point). Download python.32.tgz. Dum de dum… and… error with _posixsubprocess. Looks like it’s fixed in the 3.2 branch but not yet released.

I’m now in the situation where I’ll either have to use a hg checkout of cherrypy or an hg checkout of Python. Or both.

Lest this be seen as a dig at either the cherrypy or the Python guys, it’s certainly not. I’m grateful to them both for the work they’ve done and for the open source infrastructure they use which gives me the information and options I’ve outlined above. I could most certainly backtrack a couple of published versions of either perfectly well for my own needs.

It’s more of an endorsement of what open source offers you if you want (and I haven’t even talked about the fact that I could fork cherrypy or Python if I needed and still maintain a version in sync with the canonical one for my own use on WF).

All I need to do now is clone hg.python.org/cpython, config & make and I’m away.

Aide-memoire for Python hg clones

(This is because I use mercurial rarely enough and commit to Python even more rarely; so I always forget what incantations I used last time…)


hg clone http://hg.python.org/cpython hg.python.org
hg clone hg.python.org issue1234
cd issue1234


hg up 3.3
hg import --no-commit http://.../fixedit.patch

# Do whatever
# Edit Misc/NEWS


hg commit -m "... (Patch by ...)"
hg up default
hg merge 3.3

# Stuff happens including, probably, Misc/NEWS conflicting
# Copy Misc/NEWS.orig back to Misc/NEWS and re-edit


hg resolve -m Misc/NEWS

# Do whatever


hg commit -m "... (Patch by ...)"

(Watch out for push races if other devs have committed…)


hg push ssh://hg@hg.python.org/cpython

PyConUK 2012

Well that was fun!

Four days of community-oriented Python. Talks, open spaces, sprints, teachers, Dojos, tubas… PyConUK 2012 had it all.

I won’t retail it all. You can look at the schedule or the videos for yourselves. A few things, though, merit a special mention.

First, the organisers and the venue both managed extremely well. It wasn’t without a few very slight issues, but the venue staff were professional and helpful and the Python West Midlands organisers made sure everything went to plan (adjusting the plan, if necessary!).

Second, the education track. This was a new venture as was fronted by Nicholas Tollervey, himself a former teacher and the prime mover of the London Python Dojo. (Note to self: really must do something about a ldnpydojo website…). We had ICT and other teachers who had a tutorial on the Saturday as an introduction to Python and a sprint on the Sunday with groups of developers to see how to translate technical know-how into practical education. I think most of them also joined the Saturday afternoon Dojo. The most enthusiastic was “Miss P” who also participated in the PyConUK panel.

Then we had Alex Bradbury from the RaspberryPi foundation. He gave a keynote speech on the Saturday and led the very popular RPi sprint on Monday. He’s very enthusiastic about Python and the RPi and education and there was a lot of cross-discussion throughout the conference on those themes.

Finally among many lightning talks — ably compered, as always, by Lightning Talk Man himself — the standout was a performance on two tubas, playing the Sousa march which will be familiar to any Monty Python fan. I’m hoping someone has a picture of the performance which I can link to: the video doesn’t seem to be up yet.

A last shout-out to Tim, the video man, who stood patiently at the back of the main hall throughout the conference, recording every talk and panel, streaming it live and uploading very quickly with your slides synchronised in.

Python 3.3 is on the way

You’ve probably seen the mailing list announcements and the tweets which declare that Python 3.3 is entering its beta stage. For core developers this means that no more features can be introduced into this version of Python (at least not without the connivance of the Release Manager: Georg Brandl). Bug fixes are still allowed — which sometimes leads to rather creative labelling of changesets.

Have a look at the preliminary “What’s New?” doc.

Up to now, Python core developers have been championing Python 3 over Python 2 largely on the basis of a significant amount of cleanups, rationalisation and cruft-clearing. Now all this is very good, but mostly pleases people who have to work with the underlying Python code & data or who are fond of housekeeping. Clearly there have been other additions. (In fact the more I read through the “What’s New?” documents the more I realise that we’ve been doing something of a poor job of advertising our own achievements). But there hasn’t perhaps been anything which is really structurally groundbreaking.

Now we’ve got built-in virtual environment support, proper namespace packages, a much clearer OS exceptions hierarchy, and — at long last — a way of yielding from other iterators. (That thing where you had to loop over an iterator yielding its values: you can now just “yield from”). Any one of those alone is surely worth the entrance price. It’s actually starting to become attractive to use Python 3.x not just because it’s cleaned-up (on the moral highground but without much of a change of scenery) but because it’s got cool new features that I actually need (wow! look at that view).

To be sure there are literally hundreds of other small changes, ranging from one-off bugfixes to large-scale rewrites: bz2 has been rewritten from scratch; Brett Cannon’s finally achieved his objective of reimplementing the import mechanism in Python; Unicode support under the covers is cleaner and faster; the useful-but-slower decimal module is now useful-but-faster thanks to a version written in C, the indefatigable Stefan Krah getting the credit there. And after much bouncing around, Peter Moody’s ipaddress module is now integrated in the stdlib.

For Windows users, two significant changes have made it in (although they haven’t reached the What’s New? docs yet). One is the implementation of PEP 397 — a Python launcher for Windows, conceived by Mark Hammond, with implementation work by Vinay Sajip and worked up into its final form by Martin von Loewis. The other, implemented by Brian Curtin, is the addition of python.exe to the system PATH. This one is a long-standing gripe especially for novice users who can now just type “python” at the command prompt and get the latest version up. In fact, the PEP 397 launcher makes this perhaps less important but it’s still good to have the option there in the installer.

I’m trying to help along a couple of changes to the Win32 implementation of some os pieces under the guise of bugfixes, but even if I don’t get them past the eagle eye of the release manager, this’ll still be a version to look forward to.

PEP8 it is, then

Thanks to everyone who commented on my previous PEP8-related posts. It’s been quite an eye-opener for me to see how many people felt strongly that PEP8-compliance was a real must-have, not just for core Python itself but for any Python project. My own use of Python just about predates PEP8 itself, and certainly predates its more widespread adoption within the community. And, while I was aware that it was a default recommendation for someone seeking style guidelines, I don’t think I’d appreciated just how wide and strong its uptake was.

Of the comments made to my previous posts, only one outright dismissed PEP8; several spoke neutrally, more or less echoing the sentiments I’d expressed; but the majority which expressed a clear opinion made it strongly advisable, not to say mandatory, to follow PEP8 for one’s own (published) work.

So I’ve decided to switch to near-PEP8 style. (Note that pretty much everyone and every project varies a little, most commonly on the line length issue). The two key points which seemed — startlingly, from my perspective — to bother people the most were the two-space indents and the space-before-brackets. If you cared to peruse my winshell & active_directory github repos, you’ll note that those are now in line with PEP8. Others will follow as and when.

You’re probably not surprised to hear me say that, having made the switch, I can’t understand how anyone can prefer the PEP8 style. If I’ve switched, it’s not because I’ve had a sudden aesthetic revelation which makes me wonder why I wasn’t using 4-space indents all along. Quite the reverse: for the life of me I can’t see why anyone would prefer it. Rather, it’s because I recognise the value of a commonly-held standard within a community. And because, when I weighed it up, there was no reason — beyond my own preferences — not to switch.

I’ve tweaked my SciTE options and added the PEP8 module to my test scripts, turning off certain of the checks and allowing for slightly longer lines. Let’s see what I feel about this in a month or so’s time.

More on PEP8

Well I certainly got some robust comments on my previous PEP8 post. A few people chimed on my side, so to speak. Someone else found it “discouraging” when faced with “non-compliant code”. But it was Luke who obviously felt most strongly about the matter, and it’s his points that I’m really addressing here.

First of all, it seems that you’ve got me coming and going: on the one hand, I’m unreasonable to expect people to adopt my project’s style when contributing to it; but on the other I’m unreasonable if I expect to program the same way in every environment. Let me say again what I said in the previous post: if I contribute to, say, core Python, then I *do* expect to have to change my style to match the project guidelines. Likewise, if I contribute to Pyro — a fairly well-known Python project — then I use camelCase method names, because that’s what its maintainer uses. I don’t find that so very difficult and I doubt it anyone else does either. (Of course my “anyone else does” here is as anecdotal as the generalisations in Luke’s comments and can be treated the same way).

Luke’s basic point seems to be that all Python code *should* look the same, regardless of any other consideration, and that to do otherwise is not to be a team player, to indicate that you don’t care, and to leave open the suspicion that your code is of as poor quality as… well, your PEP8-ness. And I find that position far beyond what I consider reasonable. If that leaves me turning people away at the door then please write and tell me. Perhaps I’m being a single-minded curmudgeon on this issue, but I am honestly bemused at the idea that *all* Python code should follow what is, basically, one person’s preferences.

In short, I would be genuinely interested to hear from other people if they think that publishing Python code which follows a consistent style but which is not PEP8-compliant is that much of an issue. For them, for would-be contributors, or for the image of Python.

PEP8? Or not?

From time to time someone comments on some code example I’ve produced (or, indeed, one of my published modules) and makes the point that the code is not PEP8-compliant. The most common complaints are that I use spaces before function-call brackets — spam (”foo”) rather than spam(”foo”) — and that I use 2-space indents rather than 4.

Since I’ve written pretty much the same answer whenever this has been raised, I thought I might as well write it here and then point people towards it.

PEP 8 is the Python project’s code style guideline. I quote from the opening paragraph: “This document gives coding conventions for the Python code comprising the standard library in the main Python distribution.”. It is not a guideline for every piece of Python code ever written by anyone. On the odd occasion that I contribute code to the Python core I naturally ensure that my code complies with PEPs 7 & 8 for C & Python code respectively.

I’ve been using the format described above for well over 25 years of coding on different platforms & languages. Along with other aspects, it constitutes my personal “house style”. If anyone wished to contribute to any of my projects I would ask them to follow the same guidelines (or at least, not to complain when I reformatted their code to match my own).

FWIW I find the spam(”foo”) style cramped; no doubt some people find my spam (”foo”) style too loose. So be it. I’m genuinely astonished when people advocate 8-space indents (and I’d be a bit startled by 1-space indents). I find 4 spaces push the relevant code too far away too easily so I use 2 spaces. Barring manifest absurdities, such things are largely subjective.

So now you know: if you see code examples of mine, or are perusing my source code, be prepared for (gasp) non-PEP8 compliance!

winshell 0.6.1

In a fit of energy over the weekend, I added some basic Recycle Bin functionality to the winshell module. (And thanks to Steve Reiss for the suggestion and for ideas about convenience functions). As of version 0.6.1 you can now:

  • Iterate over items in the recycle bin
  • Determine what versions of a file there are in the recycle bin
  • Undelete the latest or a specific version of a file
  • Empty all recycle bins

You could already delete to the Recycle Bin (via delete_file). Now you can undelete as well!

winshell 0.5.4

Ahem. I promise I did test it, but then testing on developers’ machines is notoriously untrustworthy. In short, I failed to include the versioning module in the winshell sdist. Several fumbled attempts and struggles with PyPI later, I bring you: winshell 0.5.4 whose sole purpose is to fix this problem.

Many thanks to Steve Reiss who took the trouble to let me know it was failing to install