“Can automated testing replace all manual testing?” The answer is “No”

Danny Faught has been involved with software testing since 1992 and he is working as a consultant since 1999. Danny Faught is a founder of Tejas Software Consulting. Danny has written many articles and papers on software testing.

In this interview Danny talks about software testing. He shares his views on evolution in the field of software testing, automated testing techniques and tools.

QThreads >> Hi Danny. It’s a pleasure to have you on QThreads. A few words about yourself?

Danny Faught >> Hi, I’m honored to be asked to share my thoughts. About myself, I got a testing job by chance after I received my Bachelor’s degree in Computer Science. I’ve been testing ever since. I’ve been a consultant since 1999, and I really enjoy doing consulting.

QThreads >> How do you think has testing evolved over the years?

Danny Faught >> That’s a complex question. My own practices have actually regressed, in a way. When I was first learning about testing, I eagerly learned about standards and formal processes. I learned about statistical quality control and process improvement. I automated practically all of my functional tests. But I got frustrated trying to apply all these things on my project. And as a consultant, I see few clients who want and need all of these ‘œbest practice’ techniques. Most of my work now follows a very pragmatic approach and does not assume that the project is following a formal process.

Now with that said, my ability to get useful testing work done has evolved greatly. I use tools to assist with many of the things I do, even though I don’t do much functional test automation. I look at statistics and processes, not for their own sake, but in order to deal with real issues on a project. And I do much more exploratory testing than I ever did before. I’m finding bugs faster than ever before, too.

“As for the evolution of the testing field,

there have been some innovations over the years.”

As for the evolution of the testing field, there have been some innovations over the years. What’s striking about the innovations, however, is how few people know about them, and even fewer people are using them. In terms of evolution, I think we’re in the Dark Ages of testing. Many test teams work in isolation, knowing little of the existing literature on the subject, and providing little input to improve the state of the craft in general. I think that testing will evolve into a proper engineering discipline, but I don’t think that will happen for several decades yet.

” I think that testing will evolve into a proper engineering discipline, but I don’t think that will happen for several decades yet.“

QThreads >> As for test automation, do you think the tools deliver what a tester wants?

Danny Faught >> In some cases they do, though often what a tester wants is not what they really need. When most people talk about test tools, they think only of GUI automation for functional test execution. I think there is a place for automating functional test execution, but I believe that there are many more important priorities for a project, such as unit test automation and exploratory testing. Most projects I see are not ready for functional test automation. Because they’re irrelevant to most of my work, I have little experience with the big GUI automation tools and no opinion about how well they work.

“When most people talk about test tools,

they think only of GUI automation for functional test execution.“

I do use other kinds of tools on a regular basis. James Bach and I wrote about this in ‘œNot Your Father’s Test Automation: An Agile Approach to Test Automation‘, and it’s a core part of what I do. I use mostly open source and freeware tools, along with lots of small scripts that are easy to develop and make my job a lot easier.

QThreads >> If implemented properly, can automated testing replace manual testing?

Danny Faught >> The simple answer to the question ‘œCan automated testing replace all manual testing’ is ‘œNo.’

Automated functional tests can be used for regression testing (which is a small part of the overall testing effort). If an organization is running the same manual regression tests repeatedly, then the automated tests can replace some of that effort, but they also add the effort to maintain the tests, which is sometimes more than the work required to just run the tests manually. When I say some of the effort, I mean that test failures from an automated test run still must be analyzed manually. Also, any part of the process of provisioning and setting up the machine to run the tests, kicking off the test run, and babysitting it along the way that isn’t automated will still require manual attention.

“The simple answer to the question ‘œCan automated testing replace all manual testing’ is ‘œNo.’

You said ‘œif implemented properly,’ which is a big if. Automating functional tests in a way that keeps the costs of maintenance reasonable requires all of the best ideas that we have about test automation, including modularization techniques such as keyword-driven testing. It requires strong programming skills, combined with expert test automation skills. And if the user interface for the application is changing frequently, a wise automation engineer will recommend that the tests not be automated at all.

I generally recommend starting automating functional tests by developing a smoke test suite ‘“ a small set of tests that touch broadly on several of the application’s features. This will be the test suite that gets run the most, giving the greatest return on the investment in automation effort. And if the automation effort fails, like most of them do, the wasted effort is minimized.

“If done well, automation can increase the number of tests that get run.”

Danny swordfighting with his daughter, Alex

Another important point ‘“ I’ve never seen the introduction of test automation reduce the number of people on a test team. If done well, automation can increase the number of tests that get run. But maintaining that automation requires staff.

QThreads >> It’s a notion that automation testing means just recording the scripts and playback them. What do you think?

Danny Faught >> The only thing that capture/playback is useful for is learning the scripting language for a tool. You can record a script and then study it to see what the valid syntax is for one way of getting a test done. But if you believe a tool vendor who says that you just need to capture one round of manual tests and then the playback mechanism will handle all of your testing, you’re going to waste a lot of money. Hopefully the scrupulous sales reps are no longer making this pitch.

“The only thing that capture/playback is useful for is learning the scripting language for a tool.“

Test automation requires programming skills. Plain and simple, no way around it. No tool can get around that. If you don’t have programming skills on your team, you shouldn’t be automating. You can have a few programmers support the test design efforts of a larger team, but the skills still have to be there on the team somewhere.

QThreads >> Many companies don’t go for automation testing because of the inhibitive costs. So are they any free and open source alternatives?

Danny Faught >> There are hundreds of free test tools, and most of them are open source. I’m counting all kinds of tools ‘“ bug tracking tools, unit test tools, screen capture tools, functional test drivers, load test tools, static analysis tools, etc. The free load test tools are promising, because of the high cost of the commercial load test tools. They are especially appealing when you significantly ramp up the number of simulated users, which usually makes commercial load test tools much more expensive. However, the fancier free load test tools I’ve looked at, like many free test tools, are often buggy, poorly documented, and lacking necessary features. I tend to use simple load test scripts that I write using freely available automation libraries.

“There are hundreds of free test tools, and most of them are open source.“

There are not any feature-for-feature replacements for the big GUI test tools. There are a few free tools that can do GUI testing; most of them are programming libraries. There are several for applications developed in Java. And I’ve used a Perl GUI test library that works on any Windows program. Its features are limited, but it works well enough that I was able to write a random test monkey that found quite a few bugs in a Windows GUI application.

QThreads >> What do you think are the keys to successful automation testing?

Danny Faught >> If you’re asking about automating functional test execution, the key is not to assume that this kind of tool is the only and the most important tool for your team to put to use. I think that organizations that figure this out are going to have a competitive advantage over those that are still stuck in the GUI automation quagmire.

“The key is not to assume that this kind of tool is the only and

the most important tool for your team to put to use. “

I prefer to use many different small tools to help with my work. Every day I use small scripts that I wrote to launch the application under test in various configurations. I use an open source test data generation tool when I do boundary testing for input fields. I use a commercial video recording tool to monitor my monkey test when it runs unattended. I use open source automation libraries in Perl to do web load testing. If I focused all my tooling efforts on deploying automated functional tests, I might not have time to do any of these other things, and the project would suffer.

In fact, I am not automating any functional tests at all on my current project, because the application is early in its life cycle and the interface is changing frequently. It’s also important to note that I don’t feel guilty about not having automated functional tests. I sometimes wish we had a smoke test suite, but I don’t feel like there’s something wrong with the project because we’re doing manual testing. It’s the right thing to do for this project at this point in time.

James Bach and I wrote about this small tools approach in our article, ‘œNot Your Father’s Test Automation‘.

By Danny’s daughter, Kelly

QThreads >> Could you tell us more about keyword driven approach to test automation?

Danny Faught >> The keyword-driven approach shows promise for making tests more maintainable, and also for allowing non-technical people to write test cases (as long there are technical people available to support them). I don’t have enough long-term experience with it to know with confidence what situations it works well in. I’ve just participated in a few brief experiments.

“The keyword-driven approach shows promise for making tests more maintainable“

Here’s the basic idea ‘“ you develop a very simple high-level language that non-programmers can understand. Test designers use the keyword language to specify test actions and test data. Often the keywords are put into a spreadsheet. Ideally the language is not specific to any particular test driver, so you can switch to a different test driver without rewriting all of the tests; you only have to reimplement the layer that implements the keywords in a particular tool. Toolsmiths do the programming to implement the keywords and make them work with the tools that drive the user interface. So the programming work is separated from the test case development.

There are a handful of tools, some commercial and some open source, that add a keyword layer to a traditional test interface driver like a GUI test tool. And some tools integrate a keyword engine with a GUI test driver. A few of them claim to work without any programming, but I feel that they’re too closely tied to the design of the application under test.

QThreads >> How would you compare keyword driven testing with data driven testing?

Danny Faught >> On the surface, it’s fairly simple. If the data and the test code are separate, and typically the data only works by cycling through one test case with different data, that’s a data-driven test. If the test case and the data are specified in one place, but not in a full-fledged programming language, that’s keyword-driven testing.

“I used to think that keyword-driven testing was a further evolution of data-driven testing, but now I don’t think it’s that simple.“

But there’s a lot of gray area. One style of data-driven test specifies an entire test case in a single row in a spreadsheet. This is very similar to data-driven testing. Another style uses multiple statements for each test case, more like a programming language. I used to think that keyword-driven testing was a further evolution of data-driven testing, but now I don’t think it’s that simple. Some test frameworks that are described as data-driven also have keyword-like elements.

“Some test frameworks that are described as data-driven also have keyword-like elements.“

Many people are developing different styles of keyword languages, and most people who use this technique seem to develop their own tools to support it rather than using an existing commercial or free tool. Some keyword languages use keywords that invoke low-level GUI objects, making them vulnerable to maintenance problems in the test cases when the GUI design changes. Others are more abstract, focusing on performing actions from the user’s point of view. These isolate much of the maintenance to the keyword implementation. Add the fact that commercial vendors tend to invent their own terms to describe keyword-like techniques, even insisting that they’re not using a keyword approach but something new and improved, and we have a wild frontier that’s still in its infancy.

QThreads >> A Forrester study says that the next generation of automated testing tools is expected to support keyword driven testing in a big way. What features would you expect from these tools?

Danny Faught >> The biggest thing is something we’re not likely to get from the vendors who integrate the keyword support with the GUI test driver. I want independence between the keyword engine and the test driver. So not only do keywords make the test cases resilient against minor changes in the GUI, but you also can switch to a totally different GUI test driver, reimplement the keyword engine, and then use all of your test cases without changing them. I don’t want teams to get locked in to a vendor because they would have to rebuild all their test cases from scratch in order to switch.

“I want independence between the keyword engine and the test driver.”

I also want the tools to encourage testers to build an abstraction layer between the GUI and the test cases. It’s nice that some tools can scan the GUI objects in an application and automatically provide keywords that manipulate those objects without doing any programming, and some vendors consider this a big selling point. But just like capture/replay, this approach makes your test cases a maintenance problem. You need an abstraction layer in between, and that’s what you should modify when a GUI object is modified, not the test cases. This can typically be done in any tool with keyword support, but the salespeople don’t tend to mention this.

“I also want the tools to encourage testers to build

an abstraction layer between the GUI and the test cases.”

The tester’s devious hand at work

Here’s a desire that will definitely go unfulfilled ‘“ I want every tool vendor and open source tool project to start using the same terminology to describe what’s essentially the same technique. Sure, add your own flowery language to your company’s unique angle on it, but let people know that it’s a keyword engine. Otherwise, when they’re looking for a tool to support keyword-driven testing, they might pass yours by without a second thought.

That’s more of a wish list than a prediction. I’ve given some information to Forrester myself, but I’ll refer you to them for predictions on what vendors will actually do.

QThreads >> Thanks Danny. Any final notes for our readers? Also could you share your blog url / email?

Danny Faught >> I’d like to establish some sort of toolsmith’s guild, where people who build, acquire, and maintain test tools can support each other, especially those who want to broadly consider all the different kinds of tools that they can use. If anyone is interested in this, please check my blog or contact me.

You can reach me at (faught at tejasconsulting dot com), and my new blog is at http://www.tejasconsulting.com/blog.

Related:

>> An Introduction to Software Test Automation

>> Using Intelligent Test Automation Techniques

>> Next generation software quality tools from Mercury