The good and bad of XP and Agile notions of software testing
XP and Agile techniques not only have an impact on the way you develop software but also on the way the software is tested. While there are some who are excited about XP notions of 100% Test automation, there are others who feel that such an approach does not work in the real world.
Jonathan Kohl talks at length about the good and bad of the XP and Agile notions of software testing. Jonathan is a well known writer on software testing issues and has worked extensively as a tester on conventional software projects as well as scrum and extreme programming (XP) teams. He also shares his views on Ruby and the Ruby based Watir testing tool.
IndicThreads >> Hi Jonathan! It’s a pleasure to have you on IndicThreads. A few words about yourself?
Jonathan Kohl >> I’m a consultant based in Calgary, Alberta, Canada focusing on software testing. Software testing, for me, is in the intersection of three disciplines: business, technology and philosophy. Testing provides me with opportunities to investigate and problem-solve in a unique way, which is enhanced by working with talented people on great teams. I’m known for my collaboration work on software teams, particularly working with developers. My business background also comes into play, and I can be as at home in a development shop working with technical people, or working with executives in the board room. Software testing provides a lot of variety that way for me.
I believe software testing is a challenging, intellectual craft, and I’m dedicated to help move the craft forward. Skilled software testing is a relentless pursuit of learning, and there is a vast amount that we can learn. I hope we can keep pushing the envelope. You can find out more about me here: http://www.kohl.ca which has links to my articles and my blog.
“Software testing is a challenging, intellectual craft, and I’m dedicated to help move the craft forward.”
Recently, I’ve been trying to encourage more testers to work on Agile teams, and to encourage Agilists to bring more testers on their teams. These articles describe some of my experiences in this space:
*
Conventional Software Testing on a Scrum Team
* Conventional Software Testing on an Extreme Programming Team
* Exploratory Testing on an Agile Team
 |
| Jonathan testing his guitar 🙂 |
IndicThreads >> You have worked as a software tester with scrum as well as extreme programming teams. How do you distinguish between conventional software testing and the kind of testing required as part of a Scrum or extreme programming team?
Jonathan Kohl >> My experience on a wide variety of teams has taught me this: skilled software testing fits in and adds value on any team, no matter what the methodology is. There has been a lot of hype surrounding Agile methods about the tester role going away, and I spent almost a year working on Agile projects to see if this was true or not. What I found was that the XP notions of 100% test automation, no tester role, etc. didn’t always hold up when skilled software testing was applied on those projects. There are XP teams with enough skilled testing done by developers and customers that they don’t necessarily need a dedicated tester. However, I have yet to hear of a skilled software tester not providing value on a team.
“XP notions of 100% test automation, no tester role, etc. didn’t
always hold up when skilled software testing was applied on those projects”
In XP, you have two roles on a team: the customer, and programmer. So, the term tester in the XP context usually means programmer, and as such, the testing activities engaged in are programming-centric. To someone who has been in a software testing role for a number of years, so-called Agile-Testing is a specialized form of software testing – test automation. So the view of testing is often quite narrow in the Agile Testing context.
“Agile teams miss out by having a narrow focus on testing…”
It’s difficult right now to break into Agile teams if you don’t have test automation skills, and I think Agile teams miss out by having such a narrow focus on testing. I want diversity on test teams, so testers with programming skills are welcome, but so are any other skilled testers. Some of the best testers I’ve worked with could not program; many were technical writers for example. I’ve heard of tech writers fitting in just fine on Agile teams once they adjust. The people who struggle the most are Process Police they aren’t used to having to demonstrate skill, and keep up with the demands of a fast-moving team. Talented Agile teams demand skill, and testers often find they now have to explain what they are doing and why. If it provides value, most Agilists I’ve worked with were pragmatic and didn’t care if it broke the rules.
“I found Agile Testers discovering things that software testers/QA folks have known and practiced for years…”
Typically in technology fields, innovation is forward looking, and we rarely draw on past lessons from other disciplines, or even our own. I found Agile Testers discovering things that software testers/QA folks have known and practiced for years. They thought my testing was magical, and were impressed by skilled software testing, so I saw a huge opportunity for Agilists to learn from people who make a living as software testers. The mutual learning that takes place in these pairings, and the results that can be achieved are tremendous. We are barely scratching the surface and I want to see more of a convergence between these groups.
“Ridiculous claims that software testers have heard for years were once again trotted out as wisdom…
Agile methods have some great things going on with regards to testing, but we’ve been set back by a lot of bad testing folklore that was rehashed under the Agile Testing banner in early writings. Sometimes ridiculous claims that software testers have heard for years were once again trotted out as wisdom, such as manual tests are harmful, or every test must be automated, or testers are the Quality gatekeepers on a project. I found that actually working with Agilists, particularly Test-Driven Developers provided some great experiences, and I learned a lot from them, and they said they learned a lot from me. The most important lesson I learned was that my software testing skill worked just as well on Agile teams as other teams. I came to an Agile team already prepared, due to my experience as a software tester.
“My software testing skill worked just as well on Agile teams as other teams…”
“How to adopt agile techniques while testing software?”
IndicThreads >> How do you go about the task of adopting Agile techniques on your testing projects?
Jonathan Kohl >> I learn the most by doing, by trying and failing, trying and succeeding. I work with smart people with a lot of skill, and if we find a particular technique works well, I’ll try it on a testing team as well. I have found that there isn’t necessarily a direct-correlation, but I regularly focus on communication feedback loops, which is something I learned from Scrum. I’ll have a daily Scrum standup with a testing team, even if we aren’t on a Scrum team. That is one practice that seems to work well just about anywhere. I’ve also adopted using 3×5 story cards for testing tasks, which was a habit I developed on XP teams where we logged bugs on those cards. I also like testing in pairs there is a spark of creative learning and thought that occurs in pair-testing, which is something that XP taught me a lot about.
“The developers on one project discouraged me from doing anything but manual Exploratory Testing…”
 |
| Contexts and Collaboration |
Some test automation practices I learned from Agilists were a lot of fun, but they didn’t provide the information decision makers on the project needed. The developers on one project discouraged me from doing anything but manual Exploratory Testing at one point because that skill was something I could bring to the project that was missing. We can automate the tests, but we can’t test the way you do. They said to me: We love your Exploratory Testing. I was doing virtually none of the generally accepted Agile Testing practices at this point, and this was working with hardcore XPers. We finished this story, do your magic and show us where the problems are they’d say. I took that as a glowing endorsement of how skilled software testing can work on an XP team. Those are the moments you shoot for.
“Do your magic and show us where the problems are…”
IndicThreads >> Agile and Extreme Programming are often confused, so is there a difference between testing using Agile techniques and testing using Extreme Programming techniques?
Jonathan Kohl >> Extreme Programming has excellent execution guidelines, and a vocal community of practice, so it can be hard to distinguish Agile Testing using another methodology from XP. This is one reason I wrote about testing on a Scrum team. Extreme Programming focuses a great deal on automated testing. Early on when I started working on a Scrum team, I didn’t have pressures against manual testing, or towards automating all tests. When I started on an XP team, I was discouraged to do anything but automate tests. Testers will find it difficult when they start on a team that expects them just to program, or work on test automation. I have found that demonstrating skill using other testing techniques quickly overcomes the initial suspicion of a tester role.
“On an XP team when someone said tester, they often meant programmer doing testing,
and test was almost always an automated test.”
There is a lot of confusion though because a lot of what we do involves vague words. I deliberately made a distinction between conventional software testing and what you might find on an XP team because I was finding there was confusion. On an XP team, when someone said tester, they often meant programmer doing testing, and test was almost always an automated test. Testing usually meant developing and executing automated unit tests or automated user acceptance tests.
 |
| Jonathan and Elizabeth |
Now if you ask someone like me who has been a software tester for a while what these words mean, you will probably hear a different perspective. Test will include automated and manual tests in all sorts of forms; tester will mean anyone who is doing testing, or someone who works as a professional software tester. Testing, as James Bach would say is asking questions of software so we can make an assessment of it. We can express those questions in tests in all sorts of forms. Automated testing is just part of the picture, and like any technique, it comes with its own pros and cons.
As I started working on more projects, and writing and speaking about agile projects, more people started coming up to me and asking for help. On one project, agile developers and dedicated software testers on another team would have meetings, and they would use a seemingly shared language. They would agree, leave the meeting and work on tasks, and then get frustrated. A development lead on this team pulled me aside and asked me what was going wrong. I asked him to start clarifying what the testers were agreeing to, and to avoid jargon. Within a couple of days they found the source of the problem was miscommunication. Once they realized what each other were talking about, and what they needed, they were all able to meet the needs of the team.
” If I end up testing in a new environment that feels uncomfortable, I remember that good communication and skilled testing always prevail.”
I have a rule of thumb: If I end up testing in a new environment that feels uncomfortable, I remember that good communication and skilled testing always prevail. If as a software tester I’m not adding value, I’m completely willing to move on to a different project. In my experience, skilled testing seems to transcend the various methodologies and stand on its own.
“Skilled testing seems to transcend the various methodologies and stand on its own…”
“Pair Testing and Developer Tester Collaboration”
IndicThreads >> You have also been involved in pair testing with developers. Can you tell us more about pair testing and the dos and donts of effective pair testing?
Jonathan Kohl >> Pair testing for me is about learning, and about joining forces with another skilled practitioner. I’ve written an article that people have found helpful Pair Testing: How I Brought Developers into the Test Lab There are some dos and don’ts in there that you might find useful.
“Timing as a tester is really important…”
Something I’ve learned since I wrote that article, particularly from working on Scrum and XP teams, is that timing as a tester is really important. I am a musician, so I think of a project in terms of a song, or a compositional piece that a band is performing. Each role has a part to play, and those parts change over the course of a piece. The same thing happens every day on a project.
“I learned that there is a time and place for bug finding and test idea generation.”
When I first started working all day, every day with Test-Driven Developers, I put too much pressure on myself to constantly come up with testing ideas. I overwhelmed developers at certain times, and at other times they thrived on my ideas. I learned that there is a time and place for bug finding and test idea generation. Sometimes they want to bounce ideas off another systems thinker, and aren’t ready to think about bugs. Other times, they want a second opinion on a design, so if I go into full-on bug-finding mode, I will frustrate them. Once a task is complete, the developers would say: Do your worst! Tell me where the problems are, and they learned that bugs are a good thing for us to find. They appreciated the work I could do to expose problems. That isn’t all of what testing is about though focusing all the time on test idea generation and finding bugs can be like playing the same part over and over in a song. Your band leader and your audience are going to get frustrated.
“Focusing all the time on test idea generation and finding bugs
can be like playing the same part over and over in a song….”
 |
| Jonathan and Heidi |
I work on my timing when it comes to testing activities, and see how testing can help developers throughout the project. This can vary depending on the team and the people I work with. In some cases, I spend time teaching developers how to do functional testing, and they teach me more about unit testing. Recently, I’ve worked with more and more developers who were misled by the Agile-Testing folklore that automated unit testing encompasses software testing.
I’ve worked with more and more developers who were misled by the Agile-Testing folklore that automated unit testing encompasses software testing.
They are shocked that even though all their automated unit tests pass, the program fails basic, manual scenario tests. So I work with them some of the time teaching them how to think more like a tester, and to do manual functional testing in conjunction with their TDD work, as well as my own testing. I usually see a marked improvement in the code they deliver once they start doing more testing on their own work, testing in contexts other than the source code.
IndicThreads >> Testers Vs Programmers is a conflict situation we see quite often in teams. What’s your take on the subject?
Jonathan Kohl >> There doesn’t need to be a conflict. My testing career is a testament to the fruitlessness of that conflict. Here’s just one example:
I was testing a web application that had an intermittent bug. I was working on getting a repeatable case so we could fix it and I joined forces with a programmer. For about an hour a day, we would sit at one computer, and test together. His knowledge of the code and automated unit tests, coupled my Exploratory Testing were a great combination. We quickly got to the bottom of some of the issues we were investigating by working together. We learned from each other, and ideas that we realized when we understood the other’s perspective were a catalyst. It was a lot of fun, even though we were under a lot of pressure. When we worked together like this doing exploratory learning, the ideas would rapidly chain together like a kind of quicksilver. This collaboration sparked a creative energy that lead to vital discoveries in software design as well as testing.
We then spent time away working on other tasks, and investigated issues based on our skills and view of the program. Thanks to his insight and our testing sessions together, I was able to amass a great deal of information very quickly. Within a couple days of a productive pair-testing session, I found a repeatable case. Without our pair-testing, I’m sure we would have found the problem, but it would have taken much longer if I would have been on my own. That exchange of ideas carried forward now I knew a lot more about the application I was testing, and I was able to test more effectively later on. I also knew how to relate to the programmers better, and to be able to speak their shared language better. That helped me develop a better relationship with them.
“The more knowledge I’m armed with from collaborating with developers, the more creative I can get with testing…”
There is an enormous potential to learn from each other and to complement each other’s work. If developers are testing more, then I can do different kinds of testing and work on more interesting problems. The more knowledge I’m armed with from collaborating with developers, the more creative I can get with testing. I thrive on working in that collaborative space.
“Watir Web Application Testing in Ruby”
IndicThreads >> Could you tell us more about the functional software testing tool that you are involved in developing?
Jonathan Kohl >> I have been involved in Watir Web Application Testing in Ruby. Bret Pettichord and Paul Rogers have done most of the development work, and deserve a lot of credit. Chris Morris who wrote the original IEController has also been involved. Watir is a Ruby library that facilitates testing web applications using Internet Explorer. I’m also working to help start a new project that does the same thing with C#.
“Watir is a Ruby library that facilitates testing web applications using Internet Explorer…”
Watir came about because there were a few of us who were frustrated with these popular testing tools that were buggy and didn’t do what we needed for testing web apps. We independently found that capture/replay didn’t work well, and ended up using the tool as an IDE and writing our scripts by hand. Most tools had some proprietary language that had their share of problems as well. It was hard to get help, or bounce ideas off coworkers. Developers didn’t want to touch these weird languages, and I would find myself working on solving problems alone. This whole rich community grew up around Perl and Python, and there were mailing lists, forums, all kinds of tools being written, and a great exchange of ideas.
“Open Source communities are wonderful spaces of collaboration…”
So we decided to take a real programming language, develop a web testing library that we could use ourselves, and make it freely available to anyone else who wanted to use it. Open Source communities are wonderful spaces of collaboration, so we benefited from a growing community who wanted to contribute to the tool to solve their own testing problems.
IndicThreads >>Interesting that you chose Ruby. Any particular reasons why you did that? Did you find Ruby better suited to the task?
Jonathan Kohl >> We chose Ruby because it’s a powerful scripting language, and it was something we all wanted to work in. Paul Rogers and I found we could get things done quickly and easily in Ruby, so we got hooked on the language quite early on.
“I found we could get things done quickly and easily in Ruby…”
“Ruby works the way I think about programming in a scripting language…”
Chris Morris and Bret Pettichord were big fans of the language, so we had some smart people who were willing to contribute. We could have just as easily chosen Perl or Python. I like to use Ruby because it works the way I think about programming in a scripting language. We also had a community of practice growing up around Ruby, which was important. We didn’t know a group of like-minded people who were doing the same thing in Python or Perl.
“Context Driven Testing”
IndicThreads >>You are a member of the Context-Driven Testing School. Can you give us an insight into what context driven testing means to you?
Jonathan Kohl >> The Context-Driven Testing School is at the cutting edge of software testing. Some of the greatest minds and most influential figures in software testing are founding members. Like many Open Source projects, it is a self-critical community, so when you share ideas, you expect to get good, constructive criticism. Disagreement, debate and learning fuel this community. Being a member of this community is an opportunity for me to learn from the best testers in the world.
“The Context-Driven Testing School is at the cutting edge of software testing…”
|
| Think of a project |
in terms of a song
I am frequently confronted by new great ideas on testing that I can try out, instead of hearing the same old testing folklore we get from third-rate consultants and authors. I also know that if I’m the least bit sloppy when presenting an idea, I have several other Context-School members who will call me on it. We keep one another honest, but at the same time provide a tremendous amount of encouragement. No good idea is left aside just because it doesn’t fit a model that is popular at the time. Above all, skilled testing and ideas around developing and teaching skilled testing is valued.
IndicThreads >>Thanks Jonathan for sharing your views with us. Any final notes for our readers?
Jonathan Kohl >>Â Always remember this rule of thumb: no matter what situation you find yourself in as a tester, good communication and skilled testing will prevail. If you keep on developing your testing skill and working on being an effective communicator, you will add value to any team you are a part of.
IndicThreads >> Readers can keep up with Jonathan through his blog at http://www.kohl.ca/blog/
Republished with kind permission from QThreads.com
