Using Intelligent Test Automation Techniques

In part one of this series of articles, we had a look at software testing, basics of test automation, the types of test automation and myths and realities about it.

• An Introduction to Software Test Automation

In this second article Saket Godase looks at why the test automation projects get shelved and how the intelligent test automation techniques can be used to make test automation project a success-story.

Abstract
If you have been on a test automation project, it’s very likely that you have heard one or more of these comments. ‘œWhat??? Only 45 % of the manual test cases have been automated.’ I thought this would be compatible with all the browsers?’ Why can’t I run this on different versions of the Windows OS?’ ‘œI thought there would not be any need for manual testing after we completed the automation.’ You mean we will have to upgrade the scripts each time the product changes. I thought it was a one-time job.’

These statements illustrate several problems that plague test automation projects:

• A single tester performing the dual role of manual and automation. This keeps it from getting the time and focus it needs.
• Unrealistic expectations from test automation. The most common one being, test automation a simple activity that required only record and playback.
• A lack of experience in the testing team in which things are made worse by a high turnover.
• An unhealthy shift of focus from testing the product to automating the testing just for the fun of it. Many find automating the testing more interesting than testing it. The outcome in such cases rarely contributes to the test effort.

In this article we will try to address issues faced by the test automation team and attempt to present a few feasible solutions. We will look at why test automation projects get shelved regularly and how the Generic Approach and the ‘œNeural Networks’ approach could be the answer to your test automation woes.

Why do test automation projects get shelved?

There are primarily two reasons for test automation projects to get shelved. The first and the foremost are all the unrealistic promises dished out by the people managing test automation projects. People are often under a wrong impression that test automation is a piece of cake. Typical approach is to record and playback the test scripts, without giving any thoughts to the selection of tool(s), automation architecture and feasibility of the entire exercise. The second and equally important one is the inability of test automation to effectively replace manual testing.

Misconceptions surrounding test automation.

Some of the issues that we discussed in the previous article were’¦

• Test Automation tools are difficult to use.
• Test Automation is very easy. Just record the scripts at any given point of time and replay them whenever you want to.
• An unreasonably long time span is required to train the manual test team in usage of the tool(s).

While a more detailed explanation of these can be obtained at ‘œAn Introduction to Software Test Automation‘, this list can be further extended’¦

• Automating manual test cases even before the unit testing is complete. Although this might seem strange and practically impossible, it is in fact quite common, thanks to an over zealous test automation team.
• Constructing test automation suites on the fly, without bothering to correlate the test automation scripts to the existing state of the product or the test cases.
• In case of situations where the automation scripts fail / terminate abnormally, they are often unable to start from the point where they left off. This causes the scripts to be rerun from scratch, resulting in loss of valuable testing time / effort.

Inability of test automation to effectively replace manual testing

In most of the organizations test automation is not considered as a feasible alternative to manual testing. Usually, it is a marketing gimmick used to entice customers or a prop supporting the organization’s claims of being a company that works on cutting edge technology. In reality, with each passing day the test automation software is shelved and the focus shifts back to manual testing.
Reading this it is but natural for a few questions to popup in one’s mind. Is test automation worth the investment? Can test automation ever replace manual testing?
Test Automation can be a feasible alternative to manual testing, provided that the architecture of the test automation suite is good enough and the scripts posses a certain amount of intelligence. Successful “Test Automation” is not rocket science; it just requires the right blend of planning, technical know-how and innovation.

The following section will address these issues and attempt to provide a feasible solution for the same.

Intelligent Test Automation

Test automation in its current form is not an alternative to manual testing. It is crude, ineffective and possesses no inbuilt intelligence what so ever. If test automation is to ever replace manual testing (even to a reasonable extent) it must be adaptive and intelligent. These are the two main qualities that separate a manual tester from a test automation script. Let’s now look at two complementary approaches that attempt to bridge this gap between manual and automated testing.

Generic Test Automation

I’m sure that reading these words, the first question that will come in mind is: – Generic Scripts? Is this guy reinventing the wheel? No!!! I am not. Writing generic scripts does not only mean clubbing the common functionality together. It means writing scripts, which are essentially adaptive in nature. This is illustrated by the following example.

Consider a scenario in which you have been given the task to automate manual test cases used for testing a website. This website consists of half a dozen modules and you need to automate test cases for all of these. In the traditional approach using the functional decomposition method, the scripts are usually written in the following manner.

Figure 1: Traditional approach to test automation

Figure 1

Using the generic approach the same module(s) could be automated in the following manner.

Figure 2: Generic approach to test automation

Figure 2

The above two figures clearly illustrate the differences between the two approaches to test automation. The traditional approach is analogous to an automated assembly line requiring manual intervention at a dozen places. This rather defeats the purpose of automation, ultimately resulting in even lesser productivity and efficiency. On the other hand, the generic approach derives all the benefits of test automaton resulting in less manual intervention and a marked increase in productivity and efficiency. Like any other methodology the generic approach has its own share of disadvantages. However, these are more of research areas rather than limitations.

Advantages of Generic Approach:

1. A single script can be used for majority of the test cases, barring a few exceptions.
2. Error handling is much more robust in these scripts, allowing unattended execution of the test scripts.
3. Turn around time for such scripts is extremely fast.
4. Only the test data and not the scripts need to be updated for future releases of the same product.
5. This is a one-time job in the true sense.

Disadvantages of Generic Approach:

1. Technical and highly skilled personnel are required to create and maintain the scripts.
2. Creation of the initial framework and libraries can be time consuming.
3. In case of complex products, embedding business logic in such scripts may prove to be a bit tedious. Such scripts are more suitable for sanity checks.
4. Though these scripts are generic and highly reusable, they are still dependent on manual input and they lack the intelligence of a manual testing team.

Thus, we can see that ‘œGeneric Test Automation’ not only reduces the scripting efforts, it also offers a chance to the test automation team to deviate from the traditional approach, resulting in a greater scope for innovation. Such scripts are truly adaptive in nature as they are constantly learning on the job. Such scripts are ideal for regression testing products across a large number of builds. The only thing these scripts lack in is ‘œintelligence’. Though these scripts are adaptive in nature and bring us a step closer in our goal to replace manual testing with automation testing, they are not intelligent enough to completely replace manual testing. The following section outlines a complementary approach in an attempt to bridge this gap further.

Test Automation using Neural Networks

Before we dwell on the usage of neural networks in test automation let us have a quick look at neural networks. Before we proceed please keep in mind that this article is about the usage of neural networks in test automation and not about neural networks. The sole reason for including this section in the article is to provide a brief introduction of neural networks.

Neural Networks

A Neural Network is a computer technique modeled on the supposed structure and operation of the brain and is involved in processing information, making rational decisions and initiating behavioral responses.

In other words, a neural network is a computer software / hardware that attempt to simulate a model of the neural cells in animals and humans with the sole purpose of acquiring the intelligence embedded in these cells. The biggest strength of a neural network is its ability to learn by example. Used in many commercial applications for pattern recognition, neural networks could be of tremendous use in test automation.

Essentially, a neural network is nothing but a group of neurons connected together. Think of a neuron as a program or even better as a class, that accepts one or more inputs and produces one output. A typical neuron has 2 modes of operation: the training mode and the using mode. In the training mode a neuron is taught how to output a value, based on the input value(s). In the using mode, a neuron scans a given input and its associated output equals the current output. If the neuron does not recognize the input as a part of its training program, it applies firing rules to determine the output. Firing rules are nothing but a set of instructions to help the neuron react sensibly to all the inputs, irrespective of whether they were a part of its training program. Some of the popular neural networks are Forward Connection Neural Network, Hopfield Network, Brain-State-in-a-Box, the Kohonen Network and the Back Propagation Network. The Back Propagation Network is a type of neural network in which a training sample is presented to a neural network.

In order to bridge the gap between manual and automation testing it is quite important for the automation scripts to possess a certain amount of intelligence. Naturally, this intelligence has to be fed into the script(s). The easiest and possibly the most efficient method would be to use the test automation scripts in conjunction with neural networks.
Apart from being an active area of research, today, neural networks are used in a variety of commercial applications including character recognition and image recognition, stock forecasting etc. Also, neural networks are a concept, therefore tool/technology independent. Hence, effective usage of neural networks is possible, regardless of the software test automation tool/technology being used. Secondly, neural network programs are typically modular in nature. Hence, the same set of functions and procedures can be plugged in across various scripts. The subsequent part of this section will provide an insight on the usage of neural networks in test automation.

Neural Networks in Test Automation

Neural networks can be used very effectively in test automation, provided a strong architecture is in place, complemented by robust and generic scripts. The following section will explain the application of neural networks with respect to the previous test scenario.

Figure 3: Usage of neural networks in test automation

Figure 3

Traditionally, a neuron consists of 3 parts: input(s), weights assigned to each of these inputs and an output. Mapping the structure to the above architecture, it can be observed that neural networks can be embedded in each layer of the test automation suite:

• The test data acts as an input to the neural network / neuron(s).
• Each input has a value assigned to it, termed as the weight of the input. These weights are nothing but real numbers, which describe the relevance or importance of a particular input to the hidden neuron or the output neuron. The greater the weight that is assigned to an input, the more important the value of that input is to the neuron that receives it. These weights can be negative, which implies that the input can inhibit, rather than activate, a specific neuron.
• The output of the test scripts will be determined by the inputs along with their weights. In some of the neural networks, there are additional layers between the input and the output layers, called as hidden layers. In the above architecture, a part of the common libraries can constitute the hidden layers. In these types of networks, the hidden layers are embedded between the input and the output layers. These hidden layers have the freedom to form their own representation of the input data before it is passed on to the output layers, making these networks / programs very complicated and equally powerful.
• Each test case / script in a test automation suite can be mapped to a neuron. Ultimately, all these scripts constitute the neural network.

Import of Neural Networks in Test Automation:

Provided they are used correctly, neural networks would address some of the most common issues in test automation.

1. The most important of these is verifying the correctness of the test result. Currently, this is the biggest drawback of any test automation script. Existing test automation scripts are unable to verify the correctness of the test results and hence require manual intervention time and again. The inherent intelligence and pattern recognition abilities of neural networks would solve this issue to a large extent. Such a test automation suite would be fed with huge amounts of test data. Over a period of time, the test scripts would be able to distinguish between the different test data and be in a position to verify the correctness of the output, based on the input data. In this case, the main functionality of the test automation suite would be:
   1. Monitoring discrepancies between the expected and the actual output while performing regression testing.
   2. Adapting to subsequent releases of the software and updating the output of the test results accordingly (after the initial learning is complete).
2. Differentiating between genuine errors and ‘œfalse alarms’ and vice versa. This can be very useful in extremely complex software where errors and intricate functionalities are often transposed. An appropriate combination of hidden layer(s) and firing rules is the right tool to handle such a scenario. As stated previously, the hidden layers are free to construct their own representations of the input, whereas the firing rules help a neural network (automation script) to react in a sensible manner to any input that it did not encounter in its training program. This can help the automation suite to logically differentiate between functionality and defect. An appropriate combination of logic, intelligence and channeled learning would make such an automation suite a far superior tester.
3. Traversing paths and examining possibilities previously untouched. It is of common knowledge that it is impossible for any manual tester to build and execute each and every possible test case. In other words, software is never completely tested in the true sense. Scripts with built-in artificial intelligence would be learning and adapting continuously. Over a period of time they might build up on the existing test cases and ultimately, 100 % software testing might become a reality.

Advantages of Neural Networks:

1. These scripts, if designed, developed and implemented correctly, can substitute for manual testing.
2. The inherent parallel nature and real time response of neural networks makes them extremely robust.
3. Training neural networks does not require domain knowledge, only the correct training data.
4. Unlike the traditional approach, neural networks are not based on a fixed set of rules. Hence, these can be used in complex scenarios involving a lot of dynamics.
5. Such scripts have a high degree of fault tolerance.

Disadvantages of Neural Networks:

1. Technical and highly skilled personnel are required to create and maintain the scripts. Resources with a programming experience are more suited for this kind of automation.
2. Teaching and mentoring neural networks can be a time consuming activity.

Thus, we can see that implementing neural networks in test automation not only retains all the advantages of generic test automation, it also trains the automation scripts to think intelligently. Such scripts not only know what to do, they also know how to go about things without manual intervention. Requiring minimal manual intervention, these scripts are capable of completely replacing manual testing to a large extent.

Conclusion

In this article we discussed 2 complementary methodologies to test automation, generic test automation and test automation using neural networks. Each of these has its own distinct advantages and a few disadvantages, if at all any. However, the advantages far outweigh the disadvantages. Both these methodologies will take the test automation scripts a step ahead in becoming something much more than dumb work horses performing mundane tasks. Successful test automation is not just about writing scripts, it is about writing intelligent and adaptive scripts that can provide output comparable to manual testing. Sadly, test automation still has a long way to go.

Resources

• An Introduction to Neural Networks By Prof. Leslie Smith
• NEURAL NETWORKS by Christos Stergiou and Dimitrios Siganos
• Neural Networks by Genevieve Orr, Willamette University
• Introduction to Neural Networks by Nici Schraudolph and Fred Cummins, Istituto Dalle Molle di Studi sull’Intelligenza Artificiale
• Neural Networks for Pattern Recognition by Christopher M. Bishop
• An Introduction to Neural Networks by James A. Anderson