Considering all of its shortcomings, we are lucky that testing existing functionality isn’t really testing. As we said before, real testing is questioning each and every aspect and underlying assumption of the product. Existing functionality has already endured that sort of testing. Although it might be necessary to re-evaluate assumptions that were considered valid at the time of testing, this is typically not necessary before every release and certainly not continuously. Testing existing functionality is not really testing. It is called regression testing, and although it sounds the same, regression testing is to testing like pet is to carpet—not at all related. The goal of regression testing is merely to recheck that existing functionality still works as it did at the time of the actual testing. So regression testing is about controlling the changes of the behaviour of the software. In that regard it has more to do with version control than with testing. In fact, one could say that regression testing is the missing link between controlling changes of the static properties of the software (configuration and code) and controlling changes of the dynamic properties of the software (the look and behaviour). Automated tests simply pin those dynamic properties down and transform them to a static artefact (e.g. a test script), which again can be governed by current version control systems.
The takeaway is that testing is a process requiring human intervention. Bas Dijkstra, an experienced test automation consultant, describes how even the term “test automation” is flawed unless you understand what is and isn’t automated. The actual “learning, exploring, and experimenting” involved in manual, human-performed testing cannot be automated, according to Dijkstra. He writes:
Although all accounting software products have invoicing tools, FreshBooks' are incredibly easy to use. The WYSIWYG format is intuitive, and with just a few clicks you can add billable time and expenses, customize the look of the invoice and set up recurring invoices, automatic payment reminders and late fees. This accounting software connects to your bank, generates several reports, tracks time and helps you manage projects. [Go here for a full review of FreshBooks.]
I think we can all agree that automation is a critical part of any organization's software delivery pipeline, especially if you call yourself "agile." It's pretty intuitive that if you automate testing, your release cycles are going to get shorter. "So, if that's the case," you might say, "why don't we just automate everything?" There's a good reason: automation comes with a price.