The Ultimate Guide to Automation Testing

AlexQin

What Is an Automation Testing Framework?

An automation framework is a common set of tools, guidelines, and principles for your tests. Having a framework helps to minimize test script maintenance.

I like to break down an automation testing framework into specific areas of concern, or what I call the “four Ps” of an automation framework: people, planning, process, and performance.

The first one is the people aspect of a test automation framework.

People

As we already covered earlier, you want to make sure that you have set the expectations of your managers and team about your automation strategy. To help ensure that your automation is collaborative and a whole team effort, I recommend that you include automation on your sprint team’s definition of done.

The next stage is the planning piece of your automation framework.

Planning

Before writing one line of code, always check to see if there is an existing library or tool you can use before inventing your own. Break your automation framework into abstraction layers so that if anything changes, you just need to make the change in one place. Using established automation testing design patterns like the ones we cover later in this post should be part of the planning stage.

Separating your tests from your framework will also help when you have to make changes to your framework.

Also, plan on making your methods and utilities reusable to avoid code duplication. Making your test and code as readable as possible (they should read like English) will go a long way to prevent confusion and code duplication.

Finally, when planning your test, it's important to be aware of what test data your tests need. Many times, tests run against different environments that might not have the data you expect, so make sure to have a test data management strategy in place. Including support for mocking and stubbing in your framework can also help with some test data issues.

Process

Having a process in place that holds team members accountable for automation is another vital piece of a framework. Since automation is just like any other development project, make sure to use the same process and best practices that developers already follow, like using version control and performing code reviews on all automated tests.

Performance

Always start your automation framework with the end in mind. Tests need to be reliable and maintainable. They also should run as fast as possible.

Along these lines, make sure that your developers are creating unique IDs for each element that you will have to interact with within your tests. Doing this will help you avoid resorting to lousy automation practices like relying on a coordinate-based way to communicate with an element.

Another top killer of test automation script performance is the failure to use proper synchronization/wait points in your tests. Too many hard-coded waits will slow down your test suite. Use the preferred explicit wait method for synchronization.

As you write your test scripts, think about how they would perform if you had to run in parallel. Thinking about possible parallel issues beforehand will avoid problems when you start to scale the running of your test suite again on a grid or a cloud-based service like Sauce Labs.

To ensure that your tests are as preformatted as possible, refactor slow or poorly written code whenever possible. Including reporting and logging in your framework will help you quickly identify poorly running code.

To make sure that teams follow all these guidelines, determine a strategy for training and retraining your framework users.

That’s not all …

AlexQin

Automation Testing Design Patterns

It’s a given that your applications are going to change over time. And since you know change is going to happen, you should start off right from the beginning using best practices or design patterns. Doing so will make your automation more repeatable and maintainable.

Here are some common automation testing design patterns that many teams use to help them create more reliable test automation.

AlexQin

Page Objects

One popular strategy to use when creating your test automation is to model the behavior on your application. Creating simplepage objects that model the pieces of your software that you are testing against can do this.

So, for example, you would write a page object for login or a page object for a homepage. Following this approach correctly makes use of the single responsibility principle.

If anything changes—say, an element ID—you just need to go to one place to make the change and all of the tests that use the page object will automatically pick up the changes without you doing anything else. The test code needs to be updated in only one place.

Page objects also hide the technical details about HTML fields and CSS classes behind methods that have easy-to-understand names. Being mindful when naming your methods has the extra benefit of helping to create a nice readable test application programming interface (API) that a less technical automation engineer can quickly start using for automation.

AlexQin

Screenplay Pattern

Page objects are a good place to start making your test maintainable, but if you’re not careful, they can grow out of control over time. TheScreenplay pattern takes page objects and chops them down into really tiny pieces. Some testers tell me that this has made their tests much more maintainable and reliable.

Another significant benefit is that it makes test scripts more readable.

AlexQin

Ports and Adapters

Theports and adapters design strives to make sure you are using the single responsibility principle so that an object should do only one thing and have one reason to change.

When you apply this to test automation, make sure to decouple your test code to allow you to swap slow components with fast simulators so that you can run your test and the app you’re testing in the same process.

Remove all networking and input/output so nothing is slowing down the test suite. Of course, this is not easy to do, but the more you try to do this when creating UI automation, the better off you will be.

AlexQin

Presenter First

Presenter First is a modification of the model-view-controller (MVC) way of organizing code and development behaviors to create completely tested software using a test-driven development (TDD) approach.

I first heard of this pattern when interviewing Seb Rose, one of the contributors to the Cucumber project and author of The Cucumber for Java Book.

He mentioned that if you draw out the MVC pattern as blocks and arrows, you can see that the view, which is your UI, has well-defined channels of communication with the model and the controller. If you can replace those during runtime with models and controllers that your test creates and controls, then there is no reason why you can’t just test that the UI behaves in the way that you want.

You can also set your model and controller to mimic all sorts of odd behaviors, like a network going down.

AlexQin

Test Automation Process

I like to follow a six-step cyclical process when planning my test automation efforts that have the following phases: Analyze, Write, Execute, Evaluate, Communicate, and Repeat/Refactor.

• Analyze—understand your functional testing objectives. Understand what test data is needed. What needs to be tested? Know what needs to be verified. If you are part of an agile team, an excellent place to start would be in your definition of ready meeting. This will allow you to look at your upcoming sprint and see if all aspects of your automation have been taken into consideration before you start developing new functionality to test.
• Write—turn the requirements into an automated solution. Know what the start and end conditions are for each test. Tests should be completely independent of other tests. Add proper assertion checks to ensure your application is behaving according to your specifications. Each test should have a particular purpose.
• Execute—your tests should be reliable. Run each test at least three times in a row before checking in code. If you plan on using a continuous integration tool like Jenkins, start by getting your first test to execute in the environment you plan on using.
• Evaluate—verify that the automated script is doing what you expect it to do. Have manual testers prove that it is working as expected. Remember—if it’s not asserted, it’s not checked. Is the test reliable?
• Communicate—be sure that everyone on your team is aware of the results. Flaky tests should be fixed ASAP, or you’ll risk your team ignoring your test results.
• Repeat/refactor—if you notice a flaky test, refactor it to make it more reliable. Most importantly, delete any tests that are not reliable and haven’t been fixed within a given time frame. When looking at your automated regression test, ask the team if it is still needed or if it’s adding value. Pruning old tests will save you time in maintenance in the long run and ensure you are only running tests that give your team value.
  
  

AlexQin

Test Automation Best Practices

Here are some high-level good automation practices you should follow in more detail:

AlexQin

Atomic Test

An atomic test is a strategy for ensuring that each test is entirely self-contained. That is, it should not depend on the outcome of other tests to establish its state, and another test should not affect its success or failure in any way.

Also, when an automated test fails, you need to know why. Having a well-named atomic test that tests only one thing will help you quickly identify what broke if your test fails.

Furthermore, you should endeavor to get feedback to your developers as quickly as possible, and the best way to do that is with a fast, well-named test.

This is also critical if you plan on running your automation test in parallel in aSelenium Grid.

AlexQin

Test Sizing

Test size matters because tests need to run quickly.

At this point, many people visualize a traditionaltest pyramid, which has unit tests as its base, integration tests in the middle, and graphical user interface (GUI) tests at the top.

But I think more in terms of test size. By test size, I’m referring to tests that are faster than others.

While I understand the need to run UI tests if you have to create one, make it as fast as possible.