I keep half an eye on various of the channels through which new or updated Python projects are announced: planet.python.org, the PyPI feed, and c.l.py.announce. While I am occasionally irritated by the noise which PyPI produces (nested lists, anyone?) it’s helpful to see projects go past, just to remind myself what’s out there.

One phenomenon I’ve experienced is that I’ll not even have *heard* of some underlying library or web service until someone writes a Python interface to it. Glancing randomly at the current PyPI front page, I see “Python yubico client” which refers to some kind of one-key logon service which I’d never heard of. Then there’s “sunburnt - a Python interface to Solr” which, it happens, I have heard of. And a project which refers to the “Comprehensive Knowledge Archive Network” which, even after I’ve visited the website, I’m still not very clear why I might want to use it. But I’m sure it’s really good for the people who do.

This is great news for two reasons at least: people are using Python for all sorts of things, and not just the big-named everyone-does-that stuff; and there’s an inadvertent advertising effect when they publish their work, which brings a possibly interesting project to the attention of a wider audience.

Which brings me to the main point of this post: from time to time I see a project go past which really grabs the programmer in me, and I just wish I had an excuse to use. Occasionally I actually get to fulfil this wish, but usually I just look wistfully on and check as each new update is released to see if there’s any way I can squeeze it into whatever I am paid to do (or have time to do otherwise).

• Kamaelia - I’ve followed this one since its owner (Michael Sparks) presented it at a London Python meetup some years ago. I love the idea of simple objects and network pipelines between them. (Who wouldn’t?). I recall a thread a few years ago where Michael wondered why more people hadn’t taken it up (and why they’d gone for Twisted instead which seemed so much less intuitive).
• Celery - obviously not unrelated to the previous entry. My interest in lightweight distributed message passing probably goes back to my days using Modula-2, the language I used at University and which led to my discovering Python. (Modula-2 has built-in support for coroutines).
• requests - bit of a newcomer but it it exemplifies one of the things I admire most about Python code: it makes it really easy to do something which is conceptually simple. In this case, making HTTP requests. I think this is a key differentiator when comparing Python to other useful and successful languages: it just doesn’t take much code to get something done. It’s something I try for in my own libraries, and is, sadly, one of the reasons why the built-in logging module gets such a bad press. (In spite of its author’s ready willingness to answer questions, address issues, improve things, and update docs).
• Whoosh - a pure Python search engine. For no very good reason, the idea of doing this in pure Python appeals to me. As it happens I’ve rolled my own search facilities for several small projects. (And who hasn’t?). Unfortunately, the cognitive overhead of replacing those by Whoosh is just too great at the moment to justify the small risk in changing. One day…
• PyMongo (or any other of the several interfaces to MongoDB). As a daytime database developer, I’m as intrigued as the next man by the various schemaless / distributed databases currently going under the “NoSQL” umbrella. I’ve played with CouchDB (very RESTful interface but map-reduce all the way down) and MongoDB (less pure but more practical) and I even toyed with re-implementing our Helpdesk system using one of them, but as ever the risks were too great and there’s no real benefit to the business. (Only to me :) ).

There are loads more; these are just some that occur to me. Others include the kind of popup notifiers inspired by Growl, lkcl’s recently-discussed pyjamas and even some Windows-specific functionality within the pywin32 modules.

If only time and opportunity allowed…

Yesterday’s Dojo was as entertaining as ever. It almost went wrong before it started, as Fry-IT — who continue to provide the venue and the food & drink — had no working projector. Fortunately, my company were willing to lend one for the evening which got over that hurdle.

We started with a couple of lightning(ish) talks. Ravi had been on the StartupBus and he and the team had produced the final winner: TripMedi. Ravi explained the way the thing works and various of the challenges involved. He also explained how YCombinator works — something I’d only been vaguely aware of. [On a personal note, I can’t even imagine the kind of person who’d actually want to make use of what TripMedi provides, but I suppose it takes all sorts…] UPDATE: Ravi has just tweeted that the team has made it through to the next stage of YCombinator. Congratulations!

I then gave a presentation on contributing to Python core development and how you could go about it. Thanks to the efforts of Jesse Noller, Doug Hellmann and others, there is now the Python Insider blog, whose intention is to give greater visibility of some of the wider-reaching discussions taking place on python-dev. And the new core-mentorship mailing list is specifically intended to welcome would-be contributors who aren’t too sure how to go about things, or what the ground rules and policies are.

Once all the talking was done, the trouble started :) Nicholas, the indefatigable organiser of the London Python Code Dojo, has been quite keen on promoting XMPP (aka Jabber) to implement some sort of network-ability for our nascent multi-Dojo game engine. For the previous Dojo, his team had produced a simple XMPP Component which implemented a to-do list: the challenge for that Dojo. This time, Nicholas was keen for everyone to have a go with the same thing. He’d put code on Github, added a readme, and recommended the lua-based Prosody server and SleekXMPP for the Python end.

So we broke up into small groups (as they say), fired up git, pip and whatever, and dove in with about 75 minutes to produce something.

About 45 minutes later, most of us were still diving in, trying to get Prosody to listen to anything and failing to get the component talking to Prosody.

At this point, Mr Tollervey very wisely decided that we should cut our losses. Fortunately, John Chandler was willing to go up front and ask for a code review of the code he had produced. Which turned out to consist of two lines, one of which was “import random”. Being social coders, we were more than willing to advise him on the many ways in which he could make this simple and correct code into an over-engineered mess of obfuscation.

Since we’re in the middle of PyWeek, it seemed apt to have a demo from Daniel Pope of the game he’s developing on the theme of “Nine Times”. Impressive as always. (If you haven’t seen it, check out Bamboo Warrior, one of his previous efforts).

As always, we ended with a raffle of name-stickers for an O’Reilly Book and — this time — an additional donation from Mr Tollervery himself. Ironically Jon “I don’t like XMPP” Ribbens got the O’Reilly book on XMPP.

Watch the python-uk mailing list or follow ntoll on Twitter to know when tickets are available for the next Dojo (it usually takes place on the first Thursday of the month).

There’s work afoot by Mark Hammond with encouragement from others to produce a Python launcher for Windows. The work’s being specified under PEP 397 (not yet in the PEP index) and a reference implementation is being tracked under issue11629 on the Python issue tracker.

The need arises from the difficulties of having an easy way to associate Python .py/.pyw files with the appropriate version of Python on Windows, exacerbated by the advent of Python 3. The idea is that rather than having .py associate with c:\pythonxy\python.exe, it would associate with \py.exe which would attempt to read a shebang line (#! python.exe) from the top of the file dropping through various levels of default to choose a meaningful version of the Python executable to run.

From what sounds like a simple requirement there are lots of details to iron out. The (draft) PEP explains the levels of drop-through which the launcher will employ to decide what to run. There’s what seems to be a sort of awkward stand-off between not caring about cross-platformability and honouring common conventions such as /usr/bin/python. For my part I rarely if ever write code which needs to run on *nix — it’s so Windows specific that it wouldn’t even import all its dependencies, let alone do anything useful. So that side of things matters less to me.

The other detail which has caused much discussion is the mechanism by which the launcher process starts the Python interpreter. Ignoring such esoterica as COM instantiation, you have basically two options: have the launcher load the appropriate Python .DLL; or have the launcher call CreateProcess on the appropriate python .exe, passing through the parameters, open handles, etc. Each has its pros and cons but Mark’s currently plumping for creating a child process and using the Windows Job Object API to ensure the child dies with its parent.

If you’re at all invested in developing or deploying Python under Windows, have a look at the PEP and try out the Python reference implementation.

I’m fairly new to github; to git, in fact. Truth be told, I wouldn’t even have bothered with it if it weren’t for the fact that ntoll organises the code for the London Python Dojo around it. And that the game engine we had decided to use was hosted there. While I’m all in favour of DVCSes (you just *know* what’s coming when someone says “I’m all in favour…”, don’t you?), the problem is that there’s only one Subversion, but there are at least two if not three — depending on your view on bzr — DVCSes with fairly widespread support, and they’re all subtly or radically different. I more-or-less committed to learning Mercurial, because it’s what Python’s moving to, but that doesn’t help me when other people are using git. Oh well…

Previously on London Dojo: we cloned the crawlr engine from axedcode’s github repo. And found, ironically, a maze of twisty passages all alike. We went north, south, east and west in search of treasure but found only darkness… Fortunately, the man himself spotted that his game engine was flavour of the month and got in touch to ask why, and to offer a newer, better version: axengine2. Which is a lot cleaner, even if there are some questionable design choices…

So we did that. And I spent half an hour at the beginning of last Thursday’s Dojo going over key pygame concepts — rects, surfaces, sprites, and sprite groups — and giving a whistle-stop tour of the codebase which, slightly surprisingly, reimplements the Python class inheritance model in JSON, backed up by Python classes. Sort of. There was half a chance we might even have a live Skype connection to the author at the beginning of the session — before he started work over in the States — but timing didn’t work out so it didn’t come off. Shame.

We set to work again, in teams as usual, each team having a “Dojo Quest” and 90mins to complete it. Naturally, everyone was still frantically coding at the 89-minute mark. (I know we were, having made a critical breakthrough in our attempt to introduce a joystick-controlled second player [*]). We had the usual show-and-tell of: a menu system, some background sound, the two-player mode, hidden treasure, and something else I’ve forgotten. By which time it was getting on for 10.30pm and there was only just time to pull a name out of the hat for the winner of the O’Reilly Book-of-the-Month on XMPP, which we hope to use for the inter-dojo gameplay.

The Dojos are always fun; they’re also nothing like the disciplined Dojos which others favour. Basically, it’s a social coding exercise with a fair bit of laughter and passenger-seat driving. We seem to be nudging the 30 mark every month at the moment, so watch the Python UK mailing list for the announcement that tickets are available and get in there quick.

[*] We delayed ourselves a little by confusing the KEYUP / KEYDOWN constants — which indicate depression and release of keys — with the constants indicating the up and down-arrow keys :(

What technical stuff am I trying to pick up at the moment? I suppose many techy types are like me in that they keep a more-or-less close eye on the various technologies and libraries just to know what’s around. Without an immediate need; and without the time or perhaps the inclination to dive into any one of them. But from time to time some kind of incentive or motivation (or need (or job)) comes along and you start to pick something up to see if it’s worth pursuing further.

At the moment, then, I’m reading up on and trying out:

• Jabber / XMPP - for a future London Python Code Dojo
• C# - for possible work and for career prospects (and just because…)
• BI / Data Warehousing - for work
• Django - just because

Things I want to find the time for (and the money to buy the books):

• Windows Internals - to have a better understanding when I’m helping people out
• New aspects of WMI & AD - for work and to help other people

All of which adds up to a fair amount of reading and so on. And that’s on top of everything I’m trying to do. Good thing I’ve got a 50-minute train journey in the morning.

There were sounds of fear and screaming courtesy of Team 1 at the Dojo last night, hosted as ever at Fry IT offices in Southwark — an area whose eclectic mixture of shiningly-new and uncertainly-old architecture I continue to enjoy. We kicked off with the idea, promoted by Bruce (aka @otfrom), of producing the basis of a game to which other London-based Dojos could submit bots. Nicholas (@ntoll) had tweaked an existing PyGame-based game and proposed specific “Dojo Quests” for each team to work on independently.

Which was fine, except for two things. the sheer number of people — just below 30, a record I think, and leaving people perched on tables or even on the floor for want of chairs; and the relative complexity of the code. Very much to its credit, the modules were well-structured, consistently named and fairly well laid-out. But there were layers of complexity in there which meant that, in less than 90 minutes, certain of the teams were struggling to understand enough to start the task, let alone complete it. Including ours.

Fortunately — and with the help of Rene (@renedudfield) — most teams managed to get at least something to show in the “show & tell” session at the end. I think Nicholas was relieved that it came off as well as it did: we’d had a previous bad experience in trying to get to grips with an existing library in the course of a Dojo. Team 1’s Dojo Quest involved adding music & sound effects, which they did with gusto. The end result was a character wandering around a rural landscape with randomly-generated screams and cries of fear filling the air. Never has a walk in a forest been so terrifying.

In addition to helping out generally, Rene also generously provided a bunch of game controllers. And O’Reilly continued their “sponsorship” of the Dojo by providing their XMPP title for our customary raffle… which I won! (The idea is to use XMPP to connect these hypothetical Dojo-generated gamebots).

It was great to see so many people, although it does pretty much illustrate the physical limitations of our venue. I enjoyed it as always, and people I chatted to afterwards did, too. Looking forward to next month.

UPDATE: Forgot to mention that we *really* kicked off with a truly lightning talk from Tom (@tomviner) showcasing Michael Foord’s recent e and oo modules.

If I might essay a generalisation: Windows-based users of Python are less likely to have become involved in the development of Python itself. This might be for the same reasons that other people don’t get involved: a lack of time, of inclination, or of confidence in their ability to help. But it might be because Windows users on the whole aren’t so accustomed to delving into the source code of the things they use, nor are they used to checking out from source control, testing fixes and rebuilding. It’s not so much part of the culture.

I’m keen to see more Windows users & developers active in core CPython development. Python has been supported on Windows for a long time by the core development team but it can sometimes feel a little bit like a poor cousin. The majority of Python users (and of key Python developers) are Unix-based: traditionally Linux & BSD, these days often OS X. Even if they’re not unsympathetic to the needs of Windows users, they may not be be in a position to help. But Python is a volunteer effort and if Windows-savvy developers aren’t available, Windows issues will remain unfixed and untested and better support for Windows-specific issues will not be forthcoming.

There are practicalities involved: Subversion or Mercurial, Visual Studio, being able to build Python from scratch and so on, but let’s start from the other end: you’re well-intentioned and motivated, but what should you do? I’m focusing on Windows-specific issues, but some of what I say here applies to anyone who’s interested in helping Python’s core development.

How should I start?

You don’t have to be an über-hacker or even a developer at all to contribute to Python. (Although you can do more if you are). You can start by submitting bug reports or requests for improvements to the issue tracker:

• Good: Submit a bug report: send an email to bugs@python.org detailing what version of Python you’re using and on which version of Windows, what you did and what went wrong. Or what you think should happen. If you’re able to re-test with a newer version of Python that will help, in case the bug’s already been identified and fixed. If it’s a documentation issue you’re describing, be clear what alternative or additional text you’re proposing.
• Better: Submit a bug report or feature request and a failing test case: send an email as above, but attach the simplest possible self-contained script which will cause the failure you’re describing or which will illustrate what should happen.
• Best: Submit a bug report with a failing test and add a patch which you believe will solve the issue plus a patch to the relevant Python test module plus a doc patch if applicable. If this is a documentation issue, the ideal is a patch against the reStructuredText docs; at the least, a clear plain text version of what you’re proposing.

Things to bear in mind:

• Don’t be disappointed if no-one takes any action. Ideally – and often – someone quickly notices the new issue, does a little categorisation on it and assigns it to a developer who’s known to be the owner of the area where the problem occurred. At the least, one or more developers may be added to the nosy list, and one of them may take ownership. Even if that doesn’t happen, you’ve definitely done the right thing by reporting the issue and offering a patch. If it’s Windows-related, feel free to add me to the nosy list. But even if a developer is assigned, developers are volunteers with their own availability, pressures and interests.
• Don’t be surprised if the issue isn’t quite as straightforward as it appears at first blush, particularly if backwards compatibility is at stake. There may be some history of which you’re unaware, or some ramifications in other areas of the codebase which aren’t obvious.
• Don’t give up in despair if the issue is rejected. This isn’t a hard-hearted manoeuvre by the core developers to reject all outside help. It just means that things aren’t always as simple as they might appear and some bigger-picture issue might have put the kibosh on your bright idea. If you feel that a valid point has genuinely been misunderstood you’re free to ask that the issue be reopened; but please don’t do this frivolously.
• Not all versions of Python are under active development. At the time of writing, the 2.x series is at the end of its life with 2.7 the final release. Clear bugs may be fixed in that branch, but little else. It’s unlikely that anything categorised as a new feature will make it in there. The same goes for 3.1. The latest development branch is version 3.2 where new features will be targetted. Any earlier releases will either see no changes whatsoever or only critical security fixes.

I’m a Windows developer, but where are the problems?

Being a core Python developer on Windows can mean several things: doing any kind of Python core development when Windows is your primary platform; working on Windows-specific modules, such as msvcrt or winreg, or the Windows-specific aspects of other modules, such as os and subprocess; or assessing and maintaining platform-specific subtleties such as compiler-sensitive datatype sizes or line-ending issues.

A lot depends on your own inclinations, motivations and areas of expertise. If you’re an experienced Visual Studio user you might focus more on some of the compiler-specific issues or build file configurations. If you have experience of sysadmin-type work you might want to look at the os module which wraps some of the Win32 API in posix-like calls or the winreg module which exposes the registry functions. If you’re a general purpose C coder who just uses Windows as a platform then you could look at the core Python code or the parts of the stdlib written in C. If you’re an MSI guru, look at some of the outstanding issues in Python’s own MSI and the bdist_msi distutils subsystem.

Ultimately involvement probably comes from one of three things:

• Scratching your own itch: is there something in the docs which you’ve never really understood because it relies on knowledge of Unix-specifics? Or just because it’s confusing? Have you been tripped up by a stdlib function which assumes that Windows paths always act like Unix paths? Have you tried to use os.access only to discover that it’s essentially useless on Windows? Do you think you could reimplement the signal module using the Win32 API? Or sponsor the merge of a 3rd-party Windows port of the curses module?
• Scratching someone else’s itch: have a look at bugs.python.org and choose an issue which is unresolved. The issue tracker has recently had new quick-searches added to it down the left and you can easily build your own searches. This search, for example, looks for unclosed issues which include Windows in the component list. It can be a useful exercise if you simply confirm that an unclear issue still obtains on current versions of Python and, possibly, on recent versions of Windows. Or if you produce a failing test case for a merely textual or vague report. (Something which can take quite some time). But be wary of producing “me-too” noise as people following that particular issue will receive notification emails which don’t add usefully to the known state of affairs.
• Finding an itch to scratch: if you’re keen to contribute but haven’t experienced any problems which need fixing, and haven’t found any existing bugs which take your fancy, then why not just browse the source code seeing how things are done and looking for ways to improve. Be careful here since cosmetic, trivial or frivolous changes are generally frowned upon. But, for example, the subprocess and multiprocessing modules both involve some reasonably serious Windows-specific plumbing. The module maintainers aren’t necessarily Windows experts and even if they are, the more eyes the better. There are other modules which have Windows-specific elements if only in small ways.

Where should I be helping?

• The issue tracker: bugs.python.org hosts the Python issue tracker, an instance of roundup which keeps track of all bug reports and feature requests. It’s broadly followed by core developers, but a small team keeps track of incoming bugs and tries to categorise them appropriately and to assign them to a known developer if possible, or to add to the issue’s nosy list developers known to be interested in that area of the system. By helping out in this way, you get a fairly good idea of the issues being reported and the work being done on them. And on which core developers have expertise in which areas of the system. It’s also relatively easy to get higher privileges on the tracker than he default logged-in user has so that you can help classify and manage open issues.
• Documentation: The documentation is maintained as reStructuredText files within Python’s version control repository and is published using Sphinx. It’s very easy to change and patch and there’s a batch file to rebuild it on Windows although you may need to tweak it a little. Doc patches are welcome, whether for small fry such as typos or slight clarifications, or for weightier rewrite.
• stdlib Python: The Python-only parts of the standard library. A lot of the stdlib is written in Python itself; that’s what makes the whole package so portable. Even if you’re not a C developer, you can still debug and test issues in the stdlib itself. It’s especially possible to do this without a C compiler or even a Subversion checkout: you only need to put an altered version of the module in a test directory from which you run the Python interpreter and the interpreter will see the local version first.
• stdlib Python & C: Some of the standard library relies partly or wholly on extension modules, written in C against Python’s C-API. The C is reasonably readable and is often concerned only with moving Python objects to and from some system library, which makes it a fairly restricted subset of C. To do anything useful with this, you will need a C compiler and, ideally, a version-controlled checkout.
• Core Python: The core of CPython is written in C: the objects, the interpreter, the underlying data structures, key built-in modules. You’ll need to be a fairly good C programmer to work here, not least because the impact is high if a mistake is made.
• Infrastructure: CPython needs a certain amount of scaffolding to build, especially on Windows where the standard autoconf/make tools don’t apply. This scaffolding tends to be under-supported, not least because it’s not very public-facing and doesn’t solve most people’s problems. But it still needs to be done. This includes Visual Studio solution and proj files, msi build mechanisms, buildbot scripts and documentation make files.

This is Part I; Part II should include information about checkout out the source, building it, creating & applying patches and working within the development community.

A python-dev post from Steve Holden gave rise to a wider discussion about the effectiveness of the buildbot system. But also to the question of having more people involved in core Python development who are knowledgeable about the Windows aspects of the core and the stdlib.

To get involved you certainly don’t need to be a Windows guru, nor a Python guru. You don’t need to have oodles of time on your hands. (Although all of the above would help). You do need a willingness to chase down occasionally obscure failing code paths and propose and implement patches. Or workarounds. Or documentation improvements. At present there is a very small number of us watching and fixing win32 issues and, speaking for myself, I have very little time available to help. The more people there are, each of which may have very little time, the more time is available overall to help Python develop and improve.

If you’d like to get involved but haven’t bothered previously, please bother now.

If you’d like to get involved but aren’t sure what to do, ping me or drop into #python-dev on IRC or just go to the Python bug tracker and look for Windows-related problems. (That search isn’t exhaustive as it relies on someone setting the Component field, but it certainly brings some results back).

Guido’s recently expressed a wish to give more people commit privileges, and MSDN have in the past generously provided free licences to allow core developers to develop and test on different Windows and Visual Studio versions.

There’s documentation on python.org for developers, and a new in-progress document by Brian Curtin on the Python sprints site.

Even if your contribution is to confirm that a bug still applies, or has been fixed, or to add tests to an undertested module which you use yourself, or to clarify or add some documentation, it will still be a very useful addition.

I’ve already made one module Python3-compatible. I decided to bite the bullet and make my not-really-released winsys module completely Python3. I’d tried the 2to3.py approach but it left just enough false positives & negatives (sorry, don’t remember the details) that I decided to go by hand, including translating the unit tests from nose format to the updated unittest format.

Although I was reasonably clued-in on the changes from 2 to 3, I was caught out by a few things, which I list here in case they help anyone else:

• __nonzero__ is now __bool__: I knew this perfectly well, but it took me a while to find out why my tests were failing. 2to3 would have picked this up straightaway.
• __hash__ isn’t inherited if __eq__ is defined: this one surprised me and took a bit of research. (All right; I actually read the docs for __hash__).
• bytes vs str: well, obviously you knew this, but it’s bitten me in some slightly unexpected places when interfacing via pywin32. The error message speaks of expecting a buffer object, which threw me slightly.
• assertTrue doesn’t bool its param: UPDATE - of course this isn’t true at all; I was confusing this and the __bool__/__nonzero__ issue above. Thanks to Marius Gedminas for pointing this out.
• pywin32 generates tz-aware datetime objects: for a long time, pywin32 rolled its own datetime type (as Python had none). Last year, Mark H canvassed the python-win32 list about committing to the Python datetime type. And now it’s there. But it’s tz-aware. And the core Python type isn’t. Fortunately, pywin32 has for a while shipped with the win32timezone module which solves the issue.

I’m sure there’ll be more which I’ll add here as they come along.

Congratulations to Benjamin Peterson and everyone else who’s contributed to the release of Python 2.7. I’m already trying to push myself over to 3.x for everyday use and I’m confident that this psychological milestone will give other library developers the motivation to start looking at the 3.x series for their live code and newer releases.

Irmen de Jong has released a Python3-compatible version of the newly-rewritten Pyro4 (as well as maintaining for 2.x). The developers of pyodbc (a module I use extensively) have a Python3 branch which is due to be merged soon. pywin32 has been 3.x for quite a while now (thanks, Mark & Roger). Once I finish porting my own win32-specific stuff, I’ll be well-placed to see 3.x rolled out here at work.