Continuous testing: A paradigm change in testing

We have been working with global customers across regions setting up outcome based test center of excellence. Today agile development methodology is being used or will be used by most of the customers. Independent testing teams have to work in an embedded and integrated manner within scrum teams. Also organizations are moving towards continuous delivery. This requires a different approach towards testing. Off late i am spending lot of time with my team on developing a test framework that can be used for continuous delivery for a test organization which will result in an independent, integrated and embedded test entity. We are calling the testing approach as continuous testing. This is a mixture of both manual functional testing and test automation at different levels. To enable continuous delivery, testing should be done at different levels with most of it being automated.

A paradigm shift should be brought to enable continuous delivery in an organization. This requires a mindset change across the IT organization and view testing as an activity that enhances the overall productivity and quality of IT delivery. Testing should be looked as a strategic activity to ensure that the right investments are made. Few challenges that we identified to enable continuous testing on which we invested are listed here

1.  The knowledge gap of the tester: In agile development, stress is more put on creating documentation that would drive a discussion. This would mean that testers cannot expect a detailed requirement specification document or functional design document which they can use to prepare test cases. Also the turnaround time for tester is limited and he needs to turn around test cases, test results and automation all within the sprint. To make this happen we had to ensure that this knowledge gap that is normally observed in test teams is overcome

2.       Who owns the quality of systems in production? : This has been hotly debated. I might get into trouble for bringing this up. In an organization, this question should be answered and the response should be unambiguous. There might be multiple teams or functions responsible for quality or contributing towards quality but accountability should rest with just one team. That team should take the ownership and drive other teams towards improvement. This has been our personal experience and fixing this will solve many problems. This team should be empowered to take the decisions that will improve quality. This one decision will take the organization into virtuous orbit resulting in a domino effect across all other functions

3.       The perpetual automation backlog: Test Automation is indisputably our biggest challenge. Traditionally automation has been an afterthought. Once an end to end business process is ready it would be flipped over to the automation team who will start with understanding the business process. Also lot of stress is put on identifying the business case for automation. Once there is a business case approved, test automation is taken up as a separate project. Later once automation is completed it is added as part of the regression suite and executed there after forever with no updates (in most cases). But for continuous delivery using agile development, test automation cannot be a one off project that is done on a cost benefit analysis decision.Automation has always been lagging when compared with development or manual testing. For success with continuous delivery this gap should be minimized or completely eliminated. At one level, it does need change in the way we look at automation and the framework but the biggest challenge that we had to overcome was how do we does this within the sprint. How do we enable our teams at sprint level to automate the user stories across multiple teams under the constraints? We are in the process of changing the mindset of both functional tester (FT) and automation tester (AT) and motivating them to become ‘FAT’ testers (functional automation testers)

4.       Flexible automation framework: Automation frameworks are traditionally built to manage end to end business processes that run during regression testing. The supporting automation for test automation like test data automation, exception handling, start stop scripts will change when we look at automating user stories and use that as the basis for building end to end automation for cross functional business processes. It gets further complicated when the automation scripts needs to cut across multiple package applications and also automation scripts written in different technologies need to be integrated to form an end to end chain. This framework is called as the ‘CONTest framework’ (continuous testing framework)

5.       Develop a balanced view on nonfunctional testing: Nonfunctional testing is becoming extremely important in the digital era. But nonfunctional testing can really the exchequer if we go overboard on it mainly due to the tools used in this area. Most of the open source tools are not very effective and licensed tools are exorbitantly expensive and the volumes do not justify the investment. So it is important to do a profiling of the application for various parameters like performance, security, load etc and determine the usage patterns. This should be done at the starting of the project and this requires a mindset change in the development organization. So change for the test team to establish the right nonfunctional requirements and test them at regular intervals

6.       Look beyond functional quality: All said and done testing as an execution activity comes only after development. Our challenge was to come up with ideas that will intercept the development activities much earlier. But at the same time over doing this activity will increase the cost tremendously. So essentially we had to overcome 3 challenges. First challenge was to identify areas where we as independent testing team can intercept before we start test execution and provide improvements which are preventive in nature. Second challenged was to find optimal balance between velocity and cost to bring in these preventive activities. Third challenge was combine these findings with functional findings that we would find during our regular functional testing.

7.       Integrated test tooling: To enable continuous delivery, clarity on end to end process value chain is required. Once this end to end value chain is clear, the process should be automated using tools. Customers traditionally have invested in point tools which do a certain activity well. The tools usage also has led to silo teams. BAs use a requirements management or modeling tool, developers use a IDE and configuration management tool and testers use a test management and test automation tool. All these tools are standalone and don’t talk to each other. So it is important and also a challenge to create this end to end to value chain and tag the change to test so it assure business about quality.

These are challenges that large organizations face at enterprise level. Continuous testing cannot be just achieved by test automation. Other areas like environment provisioning, deployment, build management should be automated to take benefit from the high levels of test automation. May be these are topics for future blogs.

Any critique and comments are sincerely appreciated.

PS: We were able to overcome most of the challenges mentioned above and are moving forward to reach next levels of maturity which is enabling us to help the customer implement a virtuous improvement cycle across the enterprise.