Problems with Test Automation and Modern QA

AlexQin

Problems with the Process

Let’s see how the above problems manifest themselves in a typical development scenario:

• Product owner writes user stories with either no or minimum acceptance criteria.
• Not enough time dedicated to story refinement sessions to discuss various scenarios for a user story.
• Acceptance criteria are interpreted as acceptance tests – Yes, there is a difference between the two!
• Testers only automate the acceptance criteria in the user stories mostly using Selenium and/or Cucumber.
• Automated testing is almost always the responsibility of “automation testers”.
• Developers have no idea what is covered in the test packs or don’t even know how to execute the automated tests.
• The automated tests are added to an ever-expanding “regression pack” therefore taking longer and longer to run each time.
• The UI automated functional tests are integrated into the build pipeline, which is good but…
  
  

A developer pushes a simple change and has to wait 30 minutes for the automated tests to go green before the new feature or bug fix can be deployed to production. The 30 minutes wait is only if the tests pass the first time. If they fail due to some test or environment issues, it can take longer.

As the automated tests are running and the QA is investigating random failures, the developer and/or the product owner have verified the new implementation and are happy to release, but they can’t because the build is not green.

After a while, either the build goes green or the management becomes frustrated with the failing tests and makes a decision to release anyway. Then, BOOM, after a few minutes in production, there is a spike in 500 server errors.

AlexQin

Infrastructure Failures

The failures seem to show a similar pattern

• Failure in integration points.
• Failure in communicating with 3rd party apps.
• Web services not being “up” and requests to the API endpoints failing.
• A wrong configuration on one of the VMs or nodes, thus resulting in intermittent issues.
  
  

And yet, there is no process in place for checking these issues as part of the development or delivery process.

The focus of the test automation engineers is to automate functional tests. There is no focus on performance, security or resiliency. And there is certainly no testing of the infrastructure!

AlexQin

Summary:

Time has come to shift our focus from automating functional tests that have little chance of catching functional issues to the more serious and common environmental issues that plague development.

Test Automation, if done wrong or with no thought process, is a waste of time and provides no value to anyone. Crappy automated tests can incur huge maintenance costs and impede development. In the end, the only solution is to bin the tests.

In the modern software development, most of the effort of the “Test Automation Engineers” is spent battling with automation code and getting the “tests” to work rather than focusing on proper testing and exploring the system.

There is literally not enough time to write automation code and perform exploratory testing. We automate story after story and forget about integration tests, forget about the big picture.

Often we end up executing tons of automated tests, yet exploratory testing finds the majority of bugs. Then retrospectively, we write an automated test for the bugs that were found by exploratory testing, to catch regression bugs.

We should be selective on what to automate and judge our decision based on risk. What can go wrong, what is the likelihood of it going wrong and what will be the impact on the user or the business if it did go wrong?

If you are in the business of “Test Automation” then please don’t use Selenium to test the functionality of APIs or UI components. Instead, use Selenium to automate only a handful of useful and business-critical scenarios to provide confidence in business continuity before each release.

And lastly, every time you spend automating a “test”, think about the time you are wasting by not testing!

AlexQin

Source