Test Automation Framework

This post is outcome of my reading of one case study about frameworks in book "Experiences of test automation" by Dorothy Graham and Mark Fewster.  The case study is written by Lars Wahlberg, who has more than 10 years of experience in building test automation frameworks.

What is Automation Framework?
According to Wikipedia, the framework provides the basis of test automation and simplifies the automation effort.It contains APIs (Abstraction layer), Test Engine, some coding guidelines.

Abstraction Layer:
Abstraction layer is  very important factor for a good framework. This layer should be very optimal, i.e. automation engineer should be able to automate testcases without knowing technical details of system under test(SUT). And at the same time it should not be so thick that it itself becomes complex and maintenance cost is increased.
API names should be self explanatory.

Test Engine:

Test engine does following things:
1. Selects tests to be executed
2. Summarizes test results
3. provides methods for setup, cleanup, reporting and assertions.

Example can be taken as Junit or Xunit. An example can be seen, when I'll compare these definitions with my project framework.

Coding Guidelines:
If coding guidelines are followed then framework can be extremely beneficial.

 

 My learning from the case study: 
1. Abstraction level should be optimal. It should not be too thick or too thin.
2. Independent testers are required although developers can also right the test cases and execute them.
3. If configuration is also version controlled with test scripts, it can be beneficial to test system with different configurations, which can lead to explore more defects.
4. We should have separate testers and automators.
5.  Domain Knowledge is necessary for test design.
6. Test script implementation language plays a great role in automation. It should be chosen wisely keeping the design of system under test in mind.
7. There are different roles in software test automation:

       Roles:

       Test Engineer:
 1. A test engineer should have technical skills to understand the system at API level.
2. He should know a programming language.
3. It is good to have test design training.
4. Knowledge of test methodology is necessary.

     Reviewer:
1. He is senior test engineer.
2. He usually has more than 5 years experienced in domain. Good reviews can also find defects.

     Test Automation Architect:
1. A test automation architect is responsible for developing and maintaining of test framework.
2. He defines best practices and coding standards for automation.
3. Trains test engineers to do minor changes in abstraction layer, so that he does not become a bottleneck.


    Daily build Engineer:
1. Executes test batch
2. Does first analysis of results
3.  we can use tools for daily builds, and continuous builds. e.g. Jenkins, Hudson

 8. use object oriented programming, coding standards, and some good examples.
9. Always check for the configuration, if it is not present then skip the test.
10. Test automation ROI is measured on worth of rapid deliveries and worth of each passed test.
11. test automation is cost is total of cost maintaining:
       Test tool + Framework + Test Scripts
12. Each test suite maps to some functionality.


Comparison with My Current Project:

Framework : Our framework is developed using Perl and Selenium Remote Driver module written in Perl.It is for web-based application. OS interactions are done using AutoIT scripting.

Abstraction Level : Framework APIs use Object Oriented Perl. Thus it provides a fair abstraction level. One API handles many arguments, and we can give arguments or parameters as needed.

Development life cycle: Following Agile

Team Size: 17. We have automation team of 13 people. 4 people maintain framework. QA team is different, which include manual testers.

Application Domain: SSL VPN solution, web-based

Test Engine: Xunit. We are using Xunit(Junit type engine for perl). We have modules written in perl for this: Test::Class and Test::More. These follow TAP (Test Anywhere Protocol). We can select test to be executed. It also summarizes the results in following way:

ok 1 - ....

not ok 2 - ......

We have methods for setup, cleanup etc in this test engine. 

System Configuration :  system config is created in form of XML. For now it is not version controlled. It can be imported by the application.

Manual Test Procedures: These are written by QA team. Test suites and test cases are managed by Zephyr. We can also use HP ALM for that, which was being used in my previous project. We save these manual procedures in Perldoc. And then start automating them. Thus we do not need to document tests.

Technical skills needed: Product is SSL VPN. So domain knowledge includes networking, TCP/IP layers, and VPN tunneling protocols, SSL VPN etc.
Programming skills are required and object oriented programming is must. Should have some knowledge about selenium. 
Test design skills are good to have.

Daily build execution: On Jenkins. Batches are run.

Simple Example of Object Oriented programme in Perl

This is the example from the book "Intermediate Perl" and it is the easiest one I could find.

First of all, the final script we want to write is:

use strict;

use horse;

my $tv_horse = horse->named('Mr. Ed');

$tv_horse->set_name("Mister Ed");
$tv_horse->set_color("grey");

print $tv_horse->name, ' is ', $tv_horse->color, "\n";


We can see method invocation, as we have used arrow operator(->). Therefore we can conclude that this script is using APIs from an OO module (A Perl module is a reusable package defined in a library file whose name is the same as the name of the package (with a .pm on the end).). That module is horse.

we can think of above program as written below in non-OO Perl:


    Horse::named(Horse, 'Mr. Ed');
    $tv_horse::set_name($tv_horse, 'Mister Ed');
    $tv_horse::set_color($tv_horse, 'grey');
    $tv_horse::name($tv_horse);

Following is explanation of above program line by line :

use strict;
Good for debugging, it is recommended to use.

 use horse;
This line is to load the module horse, which has APIs(or Methods) defined or inheriting APIs from another module.


my $tv_horse = horse->named('Mr. Ed');
This line creates a new object instance of horse with name 'Mr. Ed'. named can be thought of as an API, although in Perl we call it as a constructor. It'll be explained when we take a look at horse module.

$tv_horse->set_name("Mister Ed");
This line uses API set_name, which, as name says, sets the name for horse object. It is a good practice to use self explanatory names.

$tv_horse->set_color("grey");
This line also explains itself the functionality.

print $tv_horse->name, ' is ', $tv_horse->color, "\n";
This line is using 2 APIs- name and color. But these are somehow different, we'll see that later.


Now the file horse.pm (Module which was loaded in previous program):

package horse;

use animal;

use strict;
use vars qw(@ISA);

our @ISA = qw(animal);

1;

There are no APIs defined in this package. However it is inheriting APIs from another package 'animal':

@ISA = qw(animal);
This line will give error with use strict . So we have to use line:

use vars qw(@ISA);

In a package we should always return 1. 

Now file animal.pm, which contains all APIs:

package animal;

use strict;

sub named
{
    my $class = shift;
    my $name  = shift;
    my $self = {
                 name  => $name,
                 color => $class->default_color
    };
    bless $self, $class;
}

sub default_color { "brown" }

sub set_name
{
    my $self           = shift;
    $self->{name} = shift;
}

sub set_color
{
    my $self            = shift;
    $self->{color} = shift;

}

sub name
{
    my $self = shift;
    ref $self ? $self->{name} : $self;
}

sub color
{
    my $self = shift;
    ref $self ? $self->{color} : $self;
}

1;

if we think of our final script as follows, then it is very simple to write animal.pm:

Horse::named(Horse, 'Mr. Ed');
    $tv_horse::set_name($tv_horse, 'Mister Ed');
    $tv_horse::set_color($tv_horse, 'grey');
    $tv_horse::name($tv_horse); 

Choosing the right tool

I was reading a case study about choosing the automation tool for Google webmasters site in book "Experiences in Automation Testing". The team consisted only 2 members, out of which one was 90% involved in manual testing. They initially chose eggplant to automate their tests. Eggplant is a tool, which compares the captured images of GUI, and compares bit by bit. This was time consuming as well as posed some problems like:

1. Whenever some text is changed in font, test fails, although there is no fault in product
2. If browser windows size is different at the time of testing, test will fail, even if everything is correct on product side

At the last they had to throw away all automation to change to new automation tool. So they chose web-driver (A tool developed in Google itself).



Web driver is a part of Selenium 2.0.

Learning:

1. Choosing automation tool greatly depends on System under test (SUT)
2. Unit test tests a small fragment of code, and does not access any external resources such as database.
3. A good review of code can also also reduce bugs.
4. A good automation strategy focuses on the most important objectives for the automated tests.
5. Tool language plays great role in good automation.
6. tests that give immediate feedback to developers are the most useful to them, because that time they remember the most of they have developed.
7. Web-driver(part of selenium 2.0) works by examining HTML source returned from web server and by manipulating JavaScript. Most of testing involves finding HTML elements and then doing string comparisons looking for expected text.

How browser works

Where I learned:

Tali Gabriel, an Israeli web developer, researched about browsers and went through many open source web browser's source code, and came up with the document about how browser works.

She also studied about Web-Kit and Gecko, the browser engines.

Browser is user interface for getting resources from web.
How browser interprets and displays HTML file.


Mostly used Terms:

Resources
URI
HTML5 and CSS specifications

Common things in all browsers:
Address bar
back and forward button
favorites bar
bookmarking options
refresh and stop buttons
home button



Browser high level architecture:
User Interface
browser engine
rendering engine
HTML parser
HTTP
JavaScript
Back end  UI
networking
Data persistence


networking - to get the HTML document from server
HTTP - protocol for transfer
HTML - language browser understands



DOM tree

HTML is very forgiving language. It is not a context free grammar, so it needs a different parser.
Other languages like JavaScript, XML etc. have context free grammar, that's why their parsers can be made from parser generators.

There is a formal format for defining HTML, that is DTD (Document Type Defifnition)

DOM tree is made by parsing of HTML. It is a object model representation of HTML. We can see this object representation in FireBug, a plugin for firefox.

In my project, we do not use DOM tree much. It is used mainly in Web programming.

In selenium, XPATH is of much importance. With almost all APIs we have to give Xpath to locate the object on web page.

A good tutorial is on w3c site:

http://www.w3schools.com/xsl/xpath_intro.asp

I'll highlight main points of Xpath in other post.

Object Oriented Perl

All about OO in Perl:

• Perl's built-in OO system is very minimal. 
• Object oriented Perl is all about "method invocation (using '->' operator)". 
• Perl's OO system is class-based, which is fairly common.
• In Perl, any package can be a class. The difference between a package which is a class and one which isn't is based on how the package is used.
• An object represents a single discrete thing.
• A class defines the behavior of a category of objects.
• All objects belong to a specific class.
• When we want to create a specific object, we start with its class, and construct or instantiate an object.  

Blessing:

• most Perl objects are hashes, but an object can be an instance of any Perl data type (scalar, array, etc.).
• Turning a plain data structure into an object is done by blessing that data structure using Perl's bless function.
• we sometimes say that an object has been "blessed into a class".  

Constructor:

In Perl, constructor is just another method. new is used mostly as constructor name, but any other name can also be used.

Class-based OO system terminology:

Object (Instance of a class)        = Data + Subroutine
Attribute    = Data
Method      = Subroutine

Suppose there is a package "File". And there is subroutine in this package called "getname".

If we are not using OO feature, then we have to call subroutine as:

use File;
File::getname();


If we are using OO feature, then File->getname(), invokes subroutine method in subroutine "File". Thus "getname "is object oriented version of  subroutine:

use File;
File->getname();
And internally it is always invoked in the form of:

Class::method('Class');

And if there are some arguments, then:

 Class::method('Class', \%args);

That's why, we write the methods as follows;

Constructor example:

Suppose there is a class or package Browser.pm and the constructor of this class is:

sub new
{
    my $class = shift;
    my $self = {
            sel                       => undef,
            JT                        => undef,
            DSID_CHECK   => 1,
            proxy                  => undef,
            capabilities         => undef,
    };
    bless( $self, $class );
    return $self;
}


Now we create an object of Browser as follows:

use Browser;
my $objBrowser = Browser->new();

Now suppose there is one method login() in browser.pm and in other file we call it as:

$objBrowser->login(\%args);

Now internally, it will be called as:

Browser::login($objBrowser, \%args);  #object is passed as first argument

So we have to write the login method in Browser.pm as follows:

sub login
{
      my $self = shift;
      my $args = shift;
      ........
      ........
}



So we can define OO in perl as following simplae points:

1. A call like $obj->method(\%args) is internally called as: 
       $obj::method($obj, \%args)
2. bless $self, $class.
    "bless", blesses $self into object of class $class, and returns $self.
    $self contains reference data of object.

The above 2 points will make OO understanding easy.

Selenium and web driver API Introduction

First Learning of Selenium as IDE:
==========================
I got first glimpse of Selenium in one video tutorial, when I wanted to change my profile from manual testing to automation testing. This tutorial was about Selenium IDE (A plugin for mozilla firefox browser to automate web testing, by recording the steps and rerunning them). That time I thought that Selenium was about "record and play" only.

Selenium Webdriver
===============
But in my second project I came to know about selenium web-driver APIs.We use Selenium webdriver with perl.

Webdriver APIs use native support for automation provided by browsers. I have to get the knowledge about this native support.

A good resource can be the following:

How Browsers Work

My first testing project

My first project was CPE-QA for NetApp client. I was working in Tata Consultancy Services. My responsibility was to execute automated and manual test cases. It was functional testing and regression testing(black box testing), which we used to perform, therefore most of the test cases were automated and only few were to be performed manually. I worked on many features related to admin interfaces like RSH, SSH, Telnet, SNMP; licensing for storage product. But later I was also included in load and stress testing, debugging automated scripts written in Perl.

From that point I got interest in learning Perl to get fully into Perl automation. I started reading scripts related to my area. These scripts were written in core perl.

But, that time I was unable to understand the syntax, regular expressions, Perl idioms(small phrase like code in perl for doing something specific). So I started learning perl from some video tutorials on youtube, on google. Then I came to know about the reference book "Programming Perl"written by creator of perl, Larry Wall. It was of great help. from initial 4 chapters, I got to know most of the syntax of perl.

Then I started doing some practice of perl from "Learning Perl" written by Randal Schwartz. Then I started fixing scripts in my project.

But to practice creating new scripts, I wanted a project only for perl automation. So I left the project and started practicing perl at home. I did many exercises on internet, contributed in CPAN pull request challenge.

Then I got a new project for client ""Pulse secure" (A company spin-off from Juniper). The product was SSL VPN gateway device and software. There, I automated many testsuites, related to SSL VPN client, SSO, Kerberos and licensing.

I used following things in this project:

Perl
Selenium
Eclipse
Git
AutoIT scripting

We mainly used Framework APIs. Framework was created using selenium web driver APIs, using Perl module Selenium::Remote::Driver.