Sunday 14 February 2016

Why Fully Automated Sofware Testing can never replace the human tester’s craftmanship, as it misses this tester’s hunch for bad parts in the software and less than obvious bugs

My name is Ernst Labruyère. I am a functional acceptance tester of software... and proud of it. It is what I did since 1998... And it is something in which I want to remain involved until the day I stop working.

My last project was at a large international bank. There my colleagues and I were involved in the creation and maintenance of a data warehouse, which had to collect and process monthly balance data and daily non-financial data on behalf of the European Central Bank and the Federal Reserve. 

The reports from this data warehouse gave these two central banks valuable insights in the financial position of the bank and the amount of risk involved in all the on-balance and off-balance items that the bank had under maintenance.
 As it were, these reports put a daily thermometer in the bank, in order to measure the bank’s health, regarding finances and risk.

The software in the data warehouse put a thermometer in the financial 
and risk health of the bank, including all its global subsidiaries
Picture by: Ernst Labruyère
Click to enlarge
The vast amount of daily and monthly data coming from subsidiaries of this bank from all over the world – multiple files per month consisting of hundreds of thousands of lines of balance data and numerous files per day containing hundreds of thousands of lines in non-financial data – forced the bank to adopt a strategy of automated software testing.

Automated software testing was an indispensable part of the test strategy of this international bank, for the simple reason that there is no way in which only one or two human testers can process such large amounts of data on a daily basis. Even the best and fastest testers in the business could only do 5 to 10% of the testing work that ought to be done per day every day, with so much data coming. This made automated testing so important for the bank.

The bank (i.e. my colleagues) had established an testing infrastructure that checked three things during every testrun of the incoming data from all over the world:
  • Domain values – did all the incoming data from the subsidiaries all over the world fit in the domain values of the target fields in the data warehouse?
  • Mandatory fields – where all the fields in the data warehouse that were truly mandatory indeed filled by the incoming data;
    • Fields that were always mandatory (i.e. key values), like identifiers and indispensable information?
    • Fields that were mandatory in the context of the product or trade that was dealt with in the balance line (f.i. fixed interest rates on a mortgage)?
  • Referential Integrity – When a balance line contained a facility or a financial instrument, then the accompanying files containing the necessary additional data regarding instruments and facilities, must contain a line with this particular facility or financial instrument. When a particular instrument was mentioned in a balance line in the balance file, but not in an instrument line in the instrument file, it would be an error.

In this case the automated test execution did a really good job, as it filtered an awful lot of information and found (potential) problems at a blistering speed. Problems that then could be investigated by the human testers and for which a root cause analysis could be found. This was work that could have never been done by human testers alone.

In a situation like the one at this bank, it is nearly impossible to do the daily testing work without an automated test infrastructure, as the human testers would always trail the increasing amount of daily information coming in from all over the world. So automated testing was a genuine success here.

There is a certain pitfal to this, however...

When there is a automated software test infrastructure at a certain test site that operates at a satisfactory level, like there was at this particular bank, then the reflex of many CIO’s (i.e. chief information officers), program managers and project leaders is to strip the amount of human testers to the bare minimum. 

To their eyes human testers are superfluous. They think that with a few test automation engineers the whole testing job can be done at an excellent level. Their argument is: 

Why would you maintain human testers in an organization when the automated testing software can do all testing stuff and find bugs so much faster and so much more accurate than human testers can?” 

And it is actually a very good question, for which there are a few answers however.

Test automation, irrespective of how accurate and advanced it is, is always the product of the perception of – mostly – a software developer or an automated test engineer. These are people that often know everything about developing and building software and designing test automation, but at the same time know very little about the business at hand in most cases.

The results is that developed sets of test automation test the underlying software very thoroughly, while using a certain set of pre-determined parameters delivered by business experts, without forgetting one simple thing. 

Yet, the test automation will never test more than to the extent of what it has stored in its parameters. Parameters that thus must be maintained on a regular basis to not become obsolete fairly quickly. In other words, an automatic test tool is just as good as the set of parameters and scripts on which it is established: when these scripts become outdated or when they don’t cover enough ground, the testing will become flawed and increasingly incomplete.

And there is more... As my former teacher and genuine test guru Erik van Veenendaal said during my testing classes during the ISEB Practitioner course: automated test tools tend to find certain kinds of software bugs. When these bugs are repaired or mitigated, they obviously won’t be found anymore, unless they re-emerge in a new software delivery (the latter happens quite often actually - EL).

Nevertheless, after the same group of software developers has been confronted with the same (kind of) bug on multiple occasions, it will be in their ‘system’ to not cause this particular bug anymore. Yet, this bug will be tested until eternity, unless this piece of automated testing script will be changed or replaced with something new. Eventually this means that the value of such a piece of test automation diminishes over time until it reaches the level of close-to-nought. 

Other bugs that have remained undisclosed, however, will be repeated over and over again, unless the automated testing software is changed in order to also find this kind of bug. In other words: when automatic test scripts are not kept up to date 24x7, their value for the testing process will diminish with a certain pace, until very little value is left eventually.

So a finished set of test automation is never complete, but must always be maintained, updated and refurbished to remain useful over the months and years. This means that the efforts of test automation engineers are never a one-off, but an enduring process. As experienced test automation engineers are very expensive people, however, they remain a substantial source of expenses for the testing budget. 

It is therefore a very good idea to not cover all parts of the software at hand with test automation, but leave some remote parts of it open for manual testers, who can test these parts very well, without the company having the need to develop automated test scripts for everything. 

The last argument for the usage of human testers is that most software engineers and designers of test automation are not capable to grasp the true meaning and reasons for the sheer existence of the software. They look at the software as a set of lines of codes and parameters with a certain goal, that they think they understand, but often don’t.

Mostly they don’t know enough of the business for which it was designed, to fully comprehend the philosophy and design reasons behind the software. Design choices and usage options that they consider as futile, can make daily usage of the software a drag for the end users, of they could compromise the quality of the outcomes of certain logic in the software.

A human tester – especially an experienced one or one with a lot of business logic – can follow his instincts and hunches to investigate things that seem ‘smelly’ to his/her eyes or that give him an uncomfortable feeling, without him being able to explain why that is. And when the human tester thinks he has got hold of a dangerous bug, he will test relentlessly until he fully uncovered it.

And so a human tester focusses his concentration on one part of the software on the first day and than on a totally different part of the software on the second day. Although this working method is less quickly, much less thorough and seemingly much less efficient than automated testing, it tends to find the not-so-obvious bugs that are not found by the automated software testing process.

Human testers – especially the best ones – have the capability to make strange associations and choose combinations that would not be really possible in real life, ‘unless’... And especially this unless-situation, this one-in-a-million, nearly impossible outcome of certain processes can cause the most serious and far-reaching bugs, for which every business manager is anxious.

Other human testers can put themselves in the minds of the daily users of the software and look at it through their eyes. So they can disclose design flaws and bad choices in the software, before they compromise the usability of the software after it is deployed in production.

Besides that, test automation is mostly at its best in the unit test or system test ‘in the small’ phase of software development, when the software is not yet deployed to its full extent. 

However, it is very hard to deploy automatic testing during in an end-to-end test situation, as it would lead to a simplification of testing, where testing in an end to end phase must be akin to real life as much as possible.

Only humans can test an ATM (i.e. automated teller machine), a medical tool or a game computer to the full extent of tweaks, hacks and abuse of such a machine. And there is nobody and no test automation that can put a website similarly to the test as a 19 year old adolescent with pimples and braces. And only humans can do things with software, websites and games of which every developer hopes that they will NOT do it, because...

My advice is therefore for companies to adopt automated testing to a certain level, as it will dramatically improve the quality of testing and make testing results available for the company in the blink of an eye.

Yet, companies should never try to remove the human factor out of testing. 

Although it might seem a waste of valuable time and money to the uninformed eye, a few human testers on top of the automated test infrastructure will turn this type of ‘hybrid’ testing into  a winning combination. 

Where the test automation will find the obvious bugs and small disarrays in the software and give it a very thorough testing treatment, the human testers will focus on exploring the outskirts of the software, following their instincts and hunches by putting everything to the test where they expect bugs to emerge. 

Combined humans and test automation become an unbeatable combination, where the one without the other is incomplete and could never lead to optimal results.