My name is Ernst Labruyère. I am a functional
acceptance tester of software... and proud of it. It is what I did since 1998...
And it is something in which I want to remain involved until the day I stop
working.
My last project was at a large international bank. There
my colleagues and I were involved in the creation and maintenance of a data
warehouse, which had to collect and process monthly balance data and daily non-financial
data on behalf of the European Central Bank and the Federal Reserve.
The
reports from this data warehouse gave these two central banks valuable insights
in the financial position of the bank and the amount of risk involved in all
the on-balance and off-balance items that the bank had under maintenance.
As it were, these reports put a daily thermometer in
the bank, in order to measure the bank’s health, regarding finances and risk.
 |
| The software in the data warehouse put a thermometer in the financial and risk health of the bank, including all its global subsidiaries Picture by: Ernst Labruyère Click to enlarge |
The vast amount of daily and monthly data coming from subsidiaries
of this bank from all over the world – multiple files per month consisting of hundreds of thousands of
lines of balance data and numerous files per day containing hundreds of thousands of lines in non-financial
data – forced the bank to adopt a strategy of automated software testing.
Automated software testing was an indispensable part of
the test strategy of this international bank, for the simple reason that there is
no way in which only one or two human testers can process such large amounts of
data on a daily basis. Even the best and fastest testers in the business could only do 5 to 10%
of the testing work that ought to be done per day every day, with so much data coming.
This made automated testing so important for the bank.
The bank (i.e. my colleagues) had established an
testing infrastructure that checked three things during every testrun of the incoming
data from all over the world:
- Domain
values – did all the incoming data from the subsidiaries all
over the world fit in the domain values of the target fields in the data
warehouse?
- Mandatory fields – where all the fields in the data warehouse that were truly mandatory indeed filled by the incoming data;
- Fields that were always mandatory (i.e. key values), like identifiers and indispensable information?
- Fields that were mandatory
in the context of the product or trade that was dealt with in the balance line (f.i. fixed interest
rates on a mortgage)?
- Referential Integrity – When a balance line contained a facility or a financial instrument, then the accompanying files containing the necessary additional data regarding instruments and facilities, must contain a line with this particular facility or financial instrument. When a particular instrument was mentioned in a balance line in the balance file, but not in an instrument line in the instrument file, it would be an error.
In this case the automated test execution did a really
good job, as it filtered an awful lot of information and found (potential)
problems at a blistering speed. Problems that then could be investigated by the
human testers and for which a root cause analysis could be found. This was work that could have never been done by human testers alone.
In a situation like the one at this bank, it is nearly impossible
to do the daily testing work without an automated test infrastructure, as the
human testers would always trail the increasing amount of daily information coming in from all over the world. So automated testing was a genuine success here.
There is a certain pitfal to this, however...
When there is a automated software test
infrastructure at a certain test site that operates at a satisfactory level, like there was at this particular bank, then the reflex of many
CIO’s (i.e. chief information officers), program managers and project leaders is
to strip the amount of human testers to the bare minimum.
To their eyes human testers are superfluous. They think that with a few test automation engineers the whole testing job can be done at an excellent level. Their argument is:
“Why
would you maintain human testers in an organization when the automated testing software can do
all testing stuff and find bugs so much faster and so much more accurate than human testers can?”
And it is actually a very good question, for which there
are a few answers however.
Test automation, irrespective of how accurate and
advanced it is, is always the product of the perception of – mostly – a software
developer or an automated test engineer. These are people that often know everything about developing and building software and designing test automation, but at the same time know very little about the business at hand in most cases.
The results is that developed sets of test automation test the underlying software very thoroughly, while using a
certain set of pre-determined parameters delivered by business experts, without forgetting one simple thing.
Yet, the test automation will never test more than to the
extent of what it has stored in its parameters. Parameters that thus must be maintained on a regular basis to not become obsolete fairly quickly. In other words, an automatic test tool is just as good
as the set of parameters and scripts on which it is established: when these
scripts become outdated or when they don’t cover enough ground, the testing
will become flawed and increasingly incomplete.
And there is more... As my former teacher and genuine
test guru Erik
van Veenendaal said during my testing classes during the ISEB Practitioner course:
automated test tools tend to find certain kinds of software bugs. When these
bugs are repaired or mitigated, they obviously won’t be found anymore, unless
they re-emerge in a new software delivery (the latter happens quite often
actually - EL).
Nevertheless, after the same group of software
developers has been confronted with the same (kind of) bug on multiple occasions,
it will be in their ‘system’ to not cause this particular bug anymore. Yet,
this bug will be tested until eternity, unless this piece of automated testing
script will be changed or replaced with something new. Eventually this means that the value of
such a piece of test automation diminishes over time until it reaches the level of close-to-nought.
Other bugs that have remained undisclosed, however, will be repeated over and over again, unless the automated testing software is changed in order to also find this kind of bug. In other words: when automatic test scripts are not
kept up to date 24x7, their value for the testing process will diminish with a
certain pace, until very little value is left eventually.
So a finished set of test automation is never complete, but must always be maintained, updated and refurbished to remain useful over the months and years. This means that the efforts of test automation engineers are never a one-off, but an enduring process. As experienced test automation engineers are very expensive people, however, they remain a substantial source of expenses for the testing budget.
It is therefore a very good idea to not cover all parts of the software at hand with test automation, but leave some remote parts of it open for manual testers, who can test these parts very well, without the company having the need to develop automated test scripts for everything.
The last argument for the usage of human testers is
that most software engineers and designers of test automation are not capable
to grasp the true meaning and reasons for the sheer existence of the software.
They look at the software as a set of lines of codes and parameters with a
certain goal, that they think they understand, but often don’t.
Mostly they don’t know enough of the business for which
it was designed, to fully comprehend the philosophy and design reasons behind
the software. Design choices and usage options that they consider as futile,
can make daily usage of the software a drag for the end users, of they could
compromise the quality of the outcomes of certain logic in the software.
A human tester – especially an experienced one or one
with a lot of business logic – can follow his instincts and hunches to
investigate things that seem ‘smelly’ to his/her eyes or that give him an
uncomfortable feeling, without him being able to explain why that is. And when
the human tester thinks he has got hold of a dangerous bug, he will test
relentlessly until he fully uncovered it.
And so a human tester focusses his concentration on one
part of the software on the first day and than on a totally different part of
the software on the second day. Although this working method is less quickly, much
less thorough and seemingly much less efficient than automated testing, it
tends to find the not-so-obvious bugs that are not found by the automated
software testing process.
Human testers – especially the best ones – have the
capability to make strange associations and choose combinations that would not
be really possible in real life, ‘unless’... And especially this
unless-situation, this one-in-a-million, nearly impossible outcome of certain
processes can cause the most serious and far-reaching bugs, for which every business manager is anxious.
Other human testers can put themselves in the minds of
the daily users of the software and look at it through their eyes. So they can
disclose design flaws and bad choices in the software, before they compromise
the usability of the software after it is deployed in production.
Besides that, test automation is mostly at its best in
the unit test or system test ‘in the small’ phase of software development, when the software is not yet deployed to its full extent.
However, it is very hard to deploy automatic testing during in an end-to-end
test situation, as it would lead to a simplification of testing, where testing in an end
to end phase must be akin to real life as much as possible.
Only humans can test an ATM (i.e. automated teller
machine), a medical tool or a game computer to the full extent of tweaks, hacks
and abuse of such a machine. And there is nobody and no test automation that can
put a website similarly to the test as a 19 year old adolescent with pimples
and braces. And only humans can do things with software, websites and games of
which every developer hopes that they will NOT do it, because...
My advice is therefore for companies to adopt automated
testing to a certain level, as it will dramatically improve the quality of
testing and make testing results available for the company in the blink of an
eye.
Yet, companies should never try to remove the human
factor out of testing.
Although it might seem a waste of valuable time and
money to the uninformed eye, a few human testers on top of the automated test
infrastructure will turn this type of ‘hybrid’ testing into a winning
combination.
Where the test automation will find the obvious bugs and small
disarrays in the software and give it a very thorough testing treatment, the
human testers will focus on exploring the outskirts of the software, following
their instincts and hunches by putting everything to the test where they expect
bugs to emerge.
Combined humans and test automation become an
unbeatable combination, where the one without the other is incomplete and could
never lead to optimal results.
