The Ultimate Guide to Automation Testing

AlexQin

Readability

A quick point on test code readability—did you know that developers spend more time reading code than actually writing it?

It is rare that the person who wrote code will also be the one that needs to modify it. Even worse, how many times have you written code only to come back to it months later and have no idea what it is doing?

Since, as we mentioned, automation code is software development, you should create your test code with the reader of the code in mind—not the computer.

This will help not only to make your test more maintainable, but also will help ensure that you do not duplicate code because you didn’t know what an existing piece of code was doing.

This might seem like a minor issue, but ignore readability of your automation test at your peril.

The importance of code readability really hit home for me after watching Cory House’s Pluralsight course calledClean Code: Writing Code for Humans.

AlexQin

Testability

Testability needs to be baked into our applications right from the start. As a regular part of sprint planning, developers should be thinking about how they can make their application code more testable. They can do this by providing things like unique element IDs for their application fields and APIs to help create hooks into their application(s) that can be used in their automated tests.

They should also be thinking about how any code changes they make to the application are going to impact existing automated tests, and plan accordingly.

If you don’t do this, you’re not going to be successful with automation for very long.

Remember, you can’t automate that which is not testable.

AlexQin

Stable Environment

This one is a really common hindrance for many teams with their automation efforts.

Without a stable test environment that is always in a known state, it will be tough for your teams to make progress with their automation efforts.

Tests failing due to environmental issues rather than actual application issues will cause your teams to lose confidence in your test feedback quickly.

Once teams start ignoring automation results, your test efforts become useless.

Some tools that can help create more stable and repeatable environments are Docker,Vagrant, and Skytap.

AlexQin

How To Pick an Automation Testing Tool

There is no “correct” test tool for automation testing. Ultimately, it all depends on your team’s unique needs and skill set.

I always recommend that you run a two-week proof of concept (POC) for each tool that you are considering and include your team’s feedback in the process before committing to a tool.

The first place to start is to look at the product roadmap and make sure the tools you select will handle future features and technologies. To avoid future compatibility issues and tedious framework refactoring due to false notions, don’t skip this step.

Next, you should evaluate the cost, including maintenance. If you plan on having your whole team help out with the automation effort, make sure to use a tool that leverages the same tools and languages your developers use.

Don’t just assume a tool will work for you. Create a small POC for each tool and get team feedback before committing to anything. Ask the team: Is the tool extensible? How easy is it to use and get started? Does it provide reporting and debugging capabilities? Does it recognize all the objects in your application? Can it integrate with other tools like version control, test management tools, and continuous integration tools?

Most importantly, find out if the tool has an active user base and select tools that other companies are using. You don’t want to select an open-source solution that is not actively maintained by the community. You may need to ask how much training it will take to get your teams up to speed with the tool. Finally, determine how easy is it to hire folks that have the skills needed to create your automated tests.

AlexQin

Test Automation Metrics

Coming up with metrics that teams can use to make sure they are on track is tough. These should be used just as a quick way to monitor your team’s progress and shouldn’t be used as hard and fast rules.

Mean Time to Diagnosis (MTD)—How long does it take you to debug a failing automated test? A high MTD is an indicator that your automated test code is not high-quality.

Bugs Found by Automation—This can be helpful to determine how much value your automation efforts are bringing.

Flaky Rate—Ideally, this should be zero, but this is a good indicator to know if your automated tests are reliable or not.

Automated to Manual Ratio—The more manual tests you have, the longer it will take to tell if your application is ready for release. This helps you keep a pulse on how long your release efforts will take. It also enables you to gain visibility into whether teams are automating (or not automating) the right things.

AlexQin

Automation Testing Frameworks and Tools

This is not an exhaustive list, but rather a quick summary of some of the more popular test tools that I’m aware of.

AlexQin

Open-Source Tools

• Selenium—This has arguably become the de facto test tool standard for browser-based testing. Please remember: You cannot use Selenium for non-browser applications; not everything can be an automation testing Selenium script.
• Appium—Appium is automation for apps. Appium seems to be the winner in the mobile testing space so far.
• Watir—This is an open-source Ruby library for automating tests. Watir interacts with a browser the same way people do: clicking links, filling out forms, and validating text.
• Sikuli—What’s cool about SikuliX is that it allows you to automate anything you see on your screen using image-based testing.
• WinAppDriver—Windows Application Driver is a service to support UI Test Automation of Windows Applications.
• White Framework—White is a framework for automating rich client applications based on Win32, WinForms, WPF, Silverlight, and SWT (Java) platforms. It’s .NET based and doesn’t require the use of any proprietary scripting languages. In fact, test automation programs using White support your writing with whatever .NET language, integrated development environment, and tools you are already using. White also provides a consistent, object-oriented API, hiding the complexity of Microsoft’s UI Automation library (upon which White is based) and Windows messages.
• AutoIt—AutoIt v3 is a freeware BASIC-like scripting language designed for automating the Windows GUI and general scripting. Many teams integrate AutoIT with Selenium to work around non-browser windows that appear in an automated test workflow.
• Serenity—This is one of my favorite automation frameworks around. Serenity is a great open-source tool because it acts like a wrapper over Selenium and behavior-driven development (BDD) tools like jBehave and Cucumber-JVM. That means that there’s a lot of built-in functionality available to you in Serenity that takes care of many things you would normally have to code from scratch if you had to create your own BDD framework. What Serenity is really awesome at is creating unbelievable reports. Out-of-the-box Serenity creates living documentation that can be used not only to view your Selenium BDD test results, but also as documentation for your application.
• Gauge—Gauge is a test automation solution that’s being built by ThoughtWorks; it’s cross-platform and open-source. It supports multiple languages including Ruby, Java, C#, Python, and JavaScript, and has upcoming support for other languages—like Golang—as well.
• Sahi—The first thing you need to know is that Sahi comes in two flavors: open-source and a pro version. Sahi Pro is the enterprise version of the open-source project. It includes lots of features coveted by larger organizations like pro style reporting.
• Robot Framework—If you want to use Python for your test automation efforts, you can’t go wrong using the Robot Framework. The Robot Framework is a mature solution that was created for testers and uses a keyword-driven approach to make tests readable and easy to create. It also has many test libraries and other tools you can use for editing, running, and building your tests.
• RedwoodHQ—This takes a little bit of a different approach from the other tools on this list. It creates a website interface that allows multiple testers to work together and run their tests from one web-accessible location.
• Galen Framework—If your automation efforts are focused on user experience design/layout testing, Galen might be a perfect fit for your needs.
• Cypress—Cypress is a more developer-centric test automation tool. It is aimed more toward making TDD a reality with developers.
  
  

AlexQin

Vendor-based Automation Test tools

• Applitools—Applitools integrates with both vendor and open-source solutions. Where most of the tools listed here are for functional test automation, Applitools helps you with visual validation testing from a user’s perspective.
• Unified Functional Testing (UFT ) UFT PRO (LeanFT)—Essentially combines the best of both the vendor-based and open-source worlds by morphing Selenium with some key functionality currently found inUFT (QuickTest Professional [QTP]).
• Microsoft Coded UI—Uses Selenium to help test Chrome and Firefox browsers. But unlike Selenium, which is only for web-based testing, Coded UI is unique in that it allows you to automate a bunch of different technologies and is not limited to the browsers.
• SmartBearTestComplete—Allows you to automate web, desktop, and mobile applications. Best of all, you can choose from script-free, drag and drop functionality, or JavaScript, Python, VBScript, JScript, DelphiScript, C++Script, or C#Script as a scripting language.
• IBM Rational Functional Tester—Like most companies, IBM’s test portfolio has grown with the acquisition of tools like Rational and Green Hat. It appears that much of the strength of its functional test tools comes from its support of numerous technologies including Windows, Mac, and mobile platforms.
• Tricentis—Self-billed as “the continuous testing company,” which is in line with many independent tool reviews like Gartner’s finding that one of its strengths is its extensive efforts to support Agile testing and continuous improvement processes.
• Worksoft—Worksoft is well known for its enterprise resource planning business end-to-end solutions.
• Testplant—One of the few test automation tools listed that has strong support for Apple’s platform. In fact, because of its unique, image-based recognition approach, it has the ability to test hard-to-automate applications—especially those with object recognition issues. Unfortunately, anyone who has done image-based, functional test automation knows how difficult these types of tests can be to maintain, and some customers have noted that as an issue.
• Ranorex—Supports a ton of technologies across all kinds of platforms—all from one tool. Noteworthy, however, is that it lacks a full, end-to-end solution and focuses mainly on functional test automation.
• Progress—For those of you who may not be familiar with this company, Progress recently acquired Telerik, which is the home of the popular free debugging tool Fiddler. Also, I know a few test engineers who actually use Progress’s Test Studio as a front end for their Selenium test automation efforts. Strengths of Progress are its integration with Visual Studio and its supported languages.
• Automation Anywhere—Differentiates itself by being the only robotic “bots on demand” focused process automation platform. Whereas some of the other tools in this list are able to test a large set of technologies, Automation Anywhere does not support testing for packaged applications like SAP or support for native mobile apps testing.
• TestIm – Leverages machine learning to speed up the authoring, execution and–most importantly–the maintenance of automated tests. Their goal is to help you to start trusting your tests.
  
  

AlexQin

API Automation Test Tools
Open-source API tools

• Rest-Assured—Rest-Assured is an open-source Java domain-specific language that makes testing REST service simple. It simplifies things by eliminating the need to use boilerplate code to test and validate complex responses. It also supports XML and JSON Request/Responses.
• RestSharp—This is a simple REST and HTTP API Client for .NET.
• Postman—Postman is a REST client that started off as a Chrome browser plugin, but recently came out with native versions for both Mac and Windows.
• SoapUI—This is the world-leading open-source functional testing tool for API testing. It supports multiple protocols such as SOAP, REST, HTTP, JMS, and AMF.
• Fiddler—Fiddler is a tool that allows you to monitor, manipulate, and reuse HTTP requests. Fiddler does many things that allow you to debug website issues, and with one of its many extensions, you can accomplish even more. Check out my article on how to get started with Fiddler.
• Karate—Since Karate is built on top of Cucumber-JVM, you can run tests and generate reports like any standard Java project. But instead of Java, you write tests in a language designed to make dealing with HTTP, JSON, or XML simple.
• Citrus Framework—Use this tool to create automated integration tests for message protocols and data formats for technologies like HTTP, REST, JMS, TCP/IP, SOAP, FTP, SSH, XML, JSON, and more.
  
  

AlexQin

Vendor API Tools

• SoapUI Pro—Since the free version is open-source, you can actually gain access to the full source code and modify as needed. The pro version is more user-friendly, and has additional functionality, including a form editor, an assertion wizard for XPath, and a SQL query builder.
• UFT API—In previous releases, HP had separate products for functional testing. QTP was used for testing GUI applications, and Service Test was for testing non-GUI technologies. HP’s latest test tool release—UFT—combines both products and features a front end that merges the separate tools into one common UI.