No programming language offers what Python does philosophically.

Jeremy Jones has been working with Python since 2001. He uses Python for test data generation, task automation, and general utility. He spends a significant portion of his day in writing and tweaking code to test specific areas of functionality.

In this interview, he tells us about his open source projects and about his favorite features in Python. He also tells us about the hurdles in developing quality software and how to negotiate them.

PythonThreads >> Hello Jeremy. We are glad to have you on PythonThreads. Could you introduce yourself to PythonThreads readers?

Jeremy Jones >>My name is Jeremy Jones and I was working at The Weather Channel as Quality Assurance Engineer, but just transitioned to Software Engineering at the beginning of the year. I’ve been working with Python since at least 2001. I have used Python in my “day jobs” over the past 5 years to primarily write functional test harnesses for a broad range of applications written in C, C++, Java, and Python.

In my free time, I’ve also contributed a couple of open source projects to the community. I am a regular contributor to the O’Reilly Network’s online publications, writing mostly on Python-related topics, but also on general Linux topics, as well. I have a blog on O’Reilly, as well. I’ve also recently acted as a technical reviewer for “Beginning Python” written by Magnus Lie Hetland and published by APress.

PythonThreads >> How would you say has python developed over the years?
Jeremy Jones >>Very positively. I’ve seen the addition of list comprehensions, generators, decorators, a logging module in the standard library, and tons of other stuff I can’t recall at the moment. I’ve also seen Python advance more into the mainstream. There appear to be more jobs specifically for Python programmers than there were 5 years ago. There are also a respectable (and growing) number of well-written books available for Python. In the “first they ignore you, then they laugh at you, then you win” progression, I’m not totally sure where Python currently is. While I typically think it’s beyond the “they ignore you” stage, there are times when I think it doesn’t get the notice in the enterprise, which is a shame.

Other times, I think we’re at the “they laugh at you” stage. I often hear it mocked for its use of significant whitespace, dynamic typing, and failure to require explicit type declarations for method signatures and return values. I think some could point to “success stories” to say that we’re at the “and then you win” stage. I would label that as premature declaration of victory. Overally, while there has been considerable publicity at times, I think Python is doing a great job flying under the radar and acting as the “secret weapon” for a number of developers. All that being said, I believe that the progression has been positive and in the right direction.

“There appear to be more jobs specifically for Python programmers than there were 5 years ago…”

PythonThreads >> You have spent a lot of time on Python but also tried out various other languages. What do you think gives it an edge over other languages like Java or Ruby?
Jeremy Jones >>Answering that for Java is really easy. Just the interactive shell gives Python a huge “up” on Java, specifically a shell like IPython. Add the fact that you don’t have to declare everything as some type before you use it and your getting-started productivity is superior in Python. Add to that the ease of iterating through a list of objects in Python as opposed to Java. And it’s really easy to remember how to do common tasks in Python. Now, was that “System.out.println” or “System.out.printline” or what was that? “print” in Python. And how do I open a file? “open” in Python. Every time I’ve had to do file IO in Java, I’ve had to go running for the docs. Some folks will argue that an IDE will answer the “System.out.println” question for me. But I hate being locked in to using an IDE. I feel that I am as productive with “just” vim and an IPython prompt as others are with a full-blown IDE.

“The interactive shell gives Python a huge “up” on Java…”

I haven’t ever done more with Ruby than to browse a tutorial or two, so I’m reluctant at this juncture to claim Python as being superior to it. Ruby is doing really well in the marketing area, which could actually be to its detriment. If I were to say that Python has an advantage over Ruby, it would be the clean syntax. I’m sure those in the Ruby camp will disagree, and that’s OK. I think a lot of it boils down to personal preference.

“If Python had an advantage over Ruby, it would be the clean syntax…”

PythonThreads >> You mentioned that functional programming languages fascinate you, can you tell us more about it.
Jeremy Jones >> FP languages are fascinating because they require such a different way of thinking than OO/imperative languages. As an example, recursion as a looping construct used to make my head hurt, but once you get the hang of it, it’s not so bad and actually makes you be a little more particular lest you wind up with an unintentionally infinitely recursive function. Given the choice, I think I still prefer a “for” loop, but there are times when recursion makes more sense. The fascinating thing, though, is just the way you have to think. I believe that having to think in different ways is like mental exercise and is beneficial even when you aren’t “exercising”. So, I guess it’s more of a psychological fascination than a technical fascination.

“I think I still prefer a “for” loop, but there are times when recursion makes more sense…”

PythonThreads >> Do you see Java, Perl developers transitioning to Python or is Python going to be adopted by a new generation of developers?

Jeremy Jones >> I think you’ll see some of both. I came from Perl, so it’s easy for me to think that others like me will get on a Python life boat and come on over. But maybe when Perl 6 actually comes around, Perlers will have less of a reason to come to Python. The big thing for me in getting away from Perl and coming to Python was the syntax of utilizing OO constructs (in Perl). And, actually, I see Ruby as maybe a better transition for Perl folks seeking a syntactically similar language with easier OO support, not that I want to discourage them from Python.

“The big thing for me in getting away from Perl and coming to Python was the syntax of utilizing OO constructs”

Java folks probably have more of a reason to come to Python than Perl folks. Perl folks at least have an alternative language (Ruby) that is easy to transition to that has nice strong OO support. All the languages like Java have the same warts that Java has. But, if they want much of the benefit of Java (the huge libraries available, some of the cool deployment mechanisms such as servlet containers and application servers), Python, specifically Jython, could be really beneficial to them.

” Java folks probably have more of a reason to come to Python than Perl folks.”

This raises an interesting question of why someone would willingly (not as a result of a work requirement) transition to another language and what Python has to offer them. I feel as though some of my previous answers sound like they aren’t promoting Python as much as I’d like. To me, the single most compelling aspect of Python has nothing to do with any syntax of the language, supporting technology, web framework, or deployment mechanism. To me, the best thing that Python has to offer is its guiding philosophy. “The Zen of Python” resonates within me as a thing of truth and beauty. The language and the best of the supporting technologies are outgrowths of that philosophy.

“The best thing that Python has to offer is its guiding philosophy”

That being said, I think Python will draw those to whom the philosophy resonates most true. This could be a new generation who are doing initial exploration of languages and haven’t found a language that has gripped them. Or it could be those who have used language “X” for years, but it just hasn’t gripped them and they feel that there must be more somewhere, so they start looking for another language. While there are plenty of usable good languages out there, I don’t feel any of them are able to offer what Python does philosophically.

“No programming language offers what Python does philosophically.”

PythonThreads >> You are the author of the open source project “Munkware” , can you tell our readers about it.
Jeremy Jones >> Munkware is a transactional persistent queueing mechanism. When I created it, I was working in a J2EE shop which relied heavily on JMS queues. While I didn’t try to implement the JMS spec in Python, I was inspired by JMS. It’s usable now, but it’s not where I’d like it to be. I’ve built at various times XMLRPC and RESTful interfaces for it as well as the standard non-networked Python API. I really am not set on how to make it available to remote processes, so that’s one area that needs some work and thought.

Another area, which is iterestingly related, is the internal structure for the queued data. Right now, I have a home-grown locking and data persistence mechanism. I’ve toyed with the idea of using BDB, but haven’t successfully integrated it as I’d like. This is related to the “remote process” area because BDB database files can be shared among multiple processes within the same operating system. So, I could create a Python API, let users create multiple processes all pointing to the same BDB file, and they could queue the data to the file between processes. Anyway, it’s something I’m still working on.

“Munkware is an open-source queueing mechanism inspired by JMS and usable today…”

PythonThreads >>  How do you use Python for your everyday software tasks?
Jeremy Jones >> At work, I’ve written a functional test harness/framework to test some of the applications from the Software Engineering group. Without getting into too much boring detail, it is XML driven, networkable, and very customizable. The test scenarios are written in XML, which I rely on Python’s DOM parser as well as ElementTree to parse. The framework walks through the XML test scenario, figures out which Python test code to instantiate, executes the test code, and checks to see if the actual result meets the expected result. I’d like to open source this some day. So, I spend a significant portion of my day writing and tweaking code to test specific areas of functionality. I also use Python for test data generation, task automation, and general utility.

At home….it varies. I use it for system administrative tasks. I’m also working on a website for my wife and am creating it in TurboGears. It’s basically a storefront, so it will contain a product catalog, shopping cart, and payment system (currently I’m integrating it with PayPal).

“I also use Python for test data generation, task automation, and general utility…”

PythonThreads >>  Any interesting projects you would like to tell us about?
Jeremy Jones >> Yes, three things. The first, I’ve already mentioned: the test framework. The second, I’ve already mentioned as well: the TurboGears store. I’m planning on releasing this to the community once I’m done with it and I’ve refactored it a bit. The third is related to some of the cool stuff I’ve been working on the broadcast side of The Weather Channel. But I’m not sure how much I can talk about that.

PythonThreads >> What developments are you keenly following in Python or even other languages?
Jeremy Jones >> The project I’ve been following most closely is TurboGears. I’ve written an in-home photo management application out of it and was thoroughly impressed with it. As I mentioned, I am also working on a website for my wife’s online store in TurboGears. TurboGears is one of the projects which I was mentioning earlier which embodies the “Zen of Python” philosophy. At least, it facilitates coding in such a style.

I also monitor activity around CherryPy itself as TurboGears has access to any and all CherryPy goodness.

“TurboGears is one of the projects which embodies the “Zen of Python” philosophy…”

I keep my eye on mod_python in hopes that it will help provide a way for hosting companies to offer cheap hosting for frameworks such as TurboGears (which sits on top of CherryPy) which typically requires a long-running standalone process. Last week I was able to deploy my wife’s website to a hosting provider using FastCGI. I think it would probably be a little cleaner with mod_python. Regardless, the community really needs to get some momentum behind a single good option for facilitating usage of commodity hosting for applications like TurboGears.

This one isn’t Python, but a project I’m interested in and keep an eye on is Asterisk. I’ve been working to setup an Asterisk server in my home and discovered that it is scriptable with Python (and pretty much any other language).

On a non-Python front, I run Ubuntu everywhere I can’t run Windows, so I check in with developments there frequently.

“I’ve been working to setup an Asterisk server in my home and discovered that it is scriptable with Python…”

PythonThreads >> You recently wrote that you are very disappointed with Python. Can you elaborate?
Jeremy Jones >> Sure. This was a really bad joke. I wrote about that in http://www.oreillynet.com/pub/wlg/7940″>this</a> blog entry. I stumbled across a song that someone had created for Ruby and lamented (jokingly) that Python had no such song and that I was disappointed with it. Some of my readers “got” it. Some others didn’t get that I was just joking.

PythonThreads >> Coming to the software quality aspect of your work, what do you think are the biggest hurdles in the path to delivering good quality software?
Jeremy Jones >> I don’t know if this is the biggest, but it’s a good place to start. Lack of good business requirements is a huge hurdle in developing quality software. Most programmers whom I have worked with are capable of doing a good job writing code if they know what it’s supposed to do. If they have requirements which don’t do a good job outlining what the system is intended to be, their ESP can sometimes fail them (typically due to lack of caffeine) and they can miss the mark for what the system is supposed to be.

“Lack of good business requirements & unit tests are huge hurdles in developing quality software…”

Another huge hurdle is lack of unit tests. It’s really easy to convince yourself that you don’t have time to write unit tests, but when you look at how much you can catch with them, it’s hard to justify not writing them. I don’t known for certain, but I suspect that well over half of the bugs that I find could have been prevented by good unit tests. I’ve never worked somewhere that was “test infected”. I have worked places where they have started “doing” unit testing, but it has always fallen by the wayside.

Every time I read about Test Driven Development, I get a tingle of excitement and think, “Man, this is how it ought to be done!” So, part of my assertion of unit testing promoting quality is based on experience, but I admit that some of it is based on purely romantic notion.

“Half of the bugs could be prevented by good unit tests…”

Something else that can be tied to unit tests is building an application with an eye toward maintainability. If you have a good suite of unit tests around an application, you can have confidence that the system will endure the hardships of time and maintenance (so long as the tests themselves are being maintained). But if an application is not nurtured with unit tests, some pieces of the code fall into disuse and people forget what they were even for. Unit tests provide documentation for their use as well as confidence that they still work as they should. If an application is not written so that it is maintainable, as the application matures, crud will invariably creep in. And crud creep tends to produce all kinds of flakey bugs that are a pain to fix and even more of a pain track down in the first place.

“Some pieces of the code fall into disuse and people forget what they were even for… “

PythonThreads >> Do you think that programming languages are missing a few things, which makes delivering good software quality difficult?
Jeremy Jones >> Rather, I think many programming languages include things that make it difficult to create quality software. When choices are abundant for performing any given task, some choices are less clear than others. When programmers choose the less-clear-choices, they are inviting problems. In my opinion, this is one of the things that makes Python so appealing. While there is frequently more than one way to do something, there is typically a single obvious way of doing the task. Reducing options for completing a task can improve a programmers productivity because he doesn’t have to figure out which way is the best. It can also help with overall software quality. If there is one obvious way of doing something, then presumably the whole team will be coding similarly. This makes inter-programmer software maintenance much smoother. Any programmer can (theoretically) pick up another’s (Python) code and start fixing bugs or enhancing it.

“Reducing options for completing a task can improve a programmers productivity…”

It would be nice if languages included built-in version control systems which integrated with the defect tracking system which integrated with the requirements tracking system which integrated with the build system which integrated with the deployment system. And it would be nice if you could simultaneously deploy two versions of the same code, test on the new version and maintain the older version in case you have to roll back to it. But I think most of those things are in the domain of standalone applications to facilitate re-use among multiple languages.

The inclusion of a standard unit testing framework will go a long way in enabling the creation of quality software. Python has had one for a while now. If something is available in the standard library and doesn’t require any work to install, it will get used more than if it weren’t included in the standard library. This was a great call to include as the standard tool set. I’m starting to wonder if there should be a push to include a web-application unit testing framework, though, since web development is quickly becoming the dominant development arena.

“I wonder if there should be a push to include a web-application unit testing framework…”

PythonThreads >> How well does Python figure, in terms of delivering good quality software?

Jeremy Jones >> Very well, I believe. It has a built-in unit testing library. It also has a philosophy which is conducive to writing quality code. Bugs will happen, but these two things will make them happen less often and easier to find when they do happen. With Python, it has been so easy to troubleshoot and debug problems.

If you get a traceback, you can typically figure out what the root of the problem is in very little time and fix it promtply as well. If you don’t get a traceback, it’s still easy to just read through code, understand it, and figure out what may be misbehaving. Python code reads easier to me than Asimov (who is really easy to read, anyway). Plus, there’s a really good debugger built in to Python which can help find and fix bugs. There are also a number of GUI debugging tools which I have heard good reports of.

“With Python, it has been so easy to troubleshoot and debug problems…”

That addressed tools for preventing, fixing, and finding bugs. But I think Python’s more appealing feature is how the language helps prevent bugs in the first place. Clear, straightforward code is much less error prone than clever, unclear code. Python promotes code of the clear, straightforward variety. Plus it’s really easy to think about a problem in Python. The closer a means of expression is to your native cognitive map, the easier of a time you’ll have correctly expressing yourself in that given medium.

“Python promotes code of the clear, straightforward variety…”

I’ve often heard it said that Python “fits the brain well”. It’s true. And it is because it “fits the brain well” that it can be used as an effective means of expression in order to instruct a computer on precicely what you want it to do. When you’re dealing with a piece of code, you’re really dealing with congealed thought. And if that congealed thought is easily parsable and pretty well matches your own native thought, the easier time you’ll have manipulating it and producing code that acts as it should. And code that acts as it should is really foundational to “quality software.”

“Python fits the brain well…“

PythonThreads >> Thanks Jeremy. It’s been good talking to you. Could you share your blog url or email address with our readers?
Jeremy Jones >> It has been my pleasure. My blog is at www.oreillynet.com/pub/au/2118 and my email address is zanesdad at  bellsouth dot net