This post is outcome of my reading of one case study about frameworks in book "Experiences of test automation" by Dorothy Graham and Mark Fewster. The case study is written by Lars Wahlberg, who has more than 10 years of experience in building test automation frameworks.
What is Automation Framework?
According to Wikipedia, the framework provides the basis of test automation and simplifies the automation effort.It contains APIs (Abstraction layer), Test Engine, some coding guidelines.
Abstraction Layer:
Abstraction layer is very important factor for a good framework. This layer should be very optimal, i.e. automation engineer should be able to automate testcases without knowing technical details of system under test(SUT). And at the same time it should not be so thick that it itself becomes complex and maintenance cost is increased.
API names should be self explanatory.
Test Engine:
Test engine does following things:
1. Selects tests to be executed
2. Summarizes test results
3. provides methods for setup, cleanup, reporting and assertions.
Example can be taken as Junit or Xunit. An example can be seen, when I'll compare these definitions with my project framework.
Coding Guidelines:
If coding guidelines are followed then framework can be extremely beneficial.
My learning from the case study:
1. Abstraction level should be optimal. It should not be too thick or too thin.
2. Independent testers are required although developers can also right the test cases and execute them.
3. If configuration is also version controlled with test scripts, it can be beneficial to test system with different configurations, which can lead to explore more defects.
4. We should have separate testers and automators.
5. Domain Knowledge is necessary for test design.
6. Test script implementation language plays a great role in automation. It should be chosen wisely keeping the design of system under test in mind.
7. There are different roles in software test automation:
9. Always check for the configuration, if it is not present then skip the test.
10. Test automation ROI is measured on worth of rapid deliveries and worth of each passed test.
11. test automation is cost is total of cost maintaining:
Test tool + Framework + Test Scripts
12. Each test suite maps to some functionality.
Comparison with My Current Project:
Framework : Our framework is developed using Perl and Selenium Remote Driver module written in Perl.It is for web-based application. OS interactions are done using AutoIT scripting.
Abstraction Level : Framework APIs use Object Oriented Perl. Thus it provides a fair abstraction level. One API handles many arguments, and we can give arguments or parameters as needed.
Development life cycle: Following Agile
Team Size: 17. We have automation team of 13 people. 4 people maintain framework. QA team is different, which include manual testers.
Application Domain: SSL VPN solution, web-based
Test Engine: Xunit. We are using Xunit(Junit type engine for perl). We have modules written in perl for this: Test::Class and Test::More. These follow TAP (Test Anywhere Protocol). We can select test to be executed. It also summarizes the results in following way:
System Configuration : system config is created in form of XML. For now it is not version controlled. It can be imported by the application.
Manual Test Procedures: These are written by QA team. Test suites and test cases are managed by Zephyr. We can also use HP ALM for that, which was being used in my previous project. We save these manual procedures in Perldoc. And then start automating them. Thus we do not need to document tests.
Technical skills needed: Product is SSL VPN. So domain knowledge includes networking, TCP/IP layers, and VPN tunneling protocols, SSL VPN etc.
Programming skills are required and object oriented programming is must. Should have some knowledge about selenium.
Test design skills are good to have.
Daily build execution: On Jenkins. Batches are run.
What is Automation Framework?
According to Wikipedia, the framework provides the basis of test automation and simplifies the automation effort.It contains APIs (Abstraction layer), Test Engine, some coding guidelines.
Abstraction Layer:
Abstraction layer is very important factor for a good framework. This layer should be very optimal, i.e. automation engineer should be able to automate testcases without knowing technical details of system under test(SUT). And at the same time it should not be so thick that it itself becomes complex and maintenance cost is increased.
API names should be self explanatory.
Test Engine:
Test engine does following things:
1. Selects tests to be executed
2. Summarizes test results
3. provides methods for setup, cleanup, reporting and assertions.
Example can be taken as Junit or Xunit. An example can be seen, when I'll compare these definitions with my project framework.
Coding Guidelines:
If coding guidelines are followed then framework can be extremely beneficial.
My learning from the case study:
1. Abstraction level should be optimal. It should not be too thick or too thin.
2. Independent testers are required although developers can also right the test cases and execute them.
3. If configuration is also version controlled with test scripts, it can be beneficial to test system with different configurations, which can lead to explore more defects.
4. We should have separate testers and automators.
5. Domain Knowledge is necessary for test design.
6. Test script implementation language plays a great role in automation. It should be chosen wisely keeping the design of system under test in mind.
7. There are different roles in software test automation:
8. use object oriented programming, coding standards, and some good examples.Roles:
Test Engineer:
1. A test engineer should have technical skills to understand the system at API level.
2. He should know a programming language.
3. It is good to have test design training.
4. Knowledge of test methodology is necessary.
Reviewer:
1. He is senior test engineer.
2. He usually has more than 5 years experienced in domain. Good reviews can also find defects.
Test Automation Architect:
1. A test automation architect is responsible for developing and maintaining of test framework.
2. He defines best practices and coding standards for automation.
3. Trains test engineers to do minor changes in abstraction layer, so that he does not become a bottleneck.
Daily build Engineer:
1. Executes test batch
2. Does first analysis of results
3. we can use tools for daily builds, and continuous builds. e.g. Jenkins, Hudson
9. Always check for the configuration, if it is not present then skip the test.
10. Test automation ROI is measured on worth of rapid deliveries and worth of each passed test.
11. test automation is cost is total of cost maintaining:
Test tool + Framework + Test Scripts
12. Each test suite maps to some functionality.
Comparison with My Current Project:
Framework : Our framework is developed using Perl and Selenium Remote Driver module written in Perl.It is for web-based application. OS interactions are done using AutoIT scripting.
Abstraction Level : Framework APIs use Object Oriented Perl. Thus it provides a fair abstraction level. One API handles many arguments, and we can give arguments or parameters as needed.
Development life cycle: Following Agile
Team Size: 17. We have automation team of 13 people. 4 people maintain framework. QA team is different, which include manual testers.
Application Domain: SSL VPN solution, web-based
Test Engine: Xunit. We are using Xunit(Junit type engine for perl). We have modules written in perl for this: Test::Class and Test::More. These follow TAP (Test Anywhere Protocol). We can select test to be executed. It also summarizes the results in following way:
ok 1 - ....
not ok 2 - ......
We have methods for setup, cleanup etc in this test engine.
System Configuration : system config is created in form of XML. For now it is not version controlled. It can be imported by the application.
Manual Test Procedures: These are written by QA team. Test suites and test cases are managed by Zephyr. We can also use HP ALM for that, which was being used in my previous project. We save these manual procedures in Perldoc. And then start automating them. Thus we do not need to document tests.
Technical skills needed: Product is SSL VPN. So domain knowledge includes networking, TCP/IP layers, and VPN tunneling protocols, SSL VPN etc.
Programming skills are required and object oriented programming is must. Should have some knowledge about selenium.
Test design skills are good to have.
Daily build execution: On Jenkins. Batches are run.
