We have been working with global
customers across regions setting up outcome based test center of excellence.
Today agile development methodology is being used or will be used by most of
the customers. Independent testing teams have to work in an embedded and
integrated manner within scrum teams. Also organizations are moving towards
continuous delivery. This requires a different approach towards testing. Off late
i am spending lot of time with my team on developing a test framework that can be
used for continuous delivery for a test organization which will result in an independent,
integrated and embedded test entity. We are calling the testing approach as continuous testing. This is a mixture
of both manual functional testing and test automation at different levels. To
enable continuous delivery, testing should be done at different levels with
most of it being automated.
A paradigm shift should be
brought to enable continuous delivery in an organization. This requires a
mindset change across the IT organization and view testing as an activity that
enhances the overall productivity and quality of IT delivery. Testing should be
looked as a strategic activity to ensure that the right investments are made.
Few challenges that we identified to enable continuous testing on which we
invested are listed here
1. The knowledge gap of the tester: In
agile development, stress is more put on creating documentation that would
drive a discussion. This would mean that testers cannot expect a detailed
requirement specification document or functional design document which they can
use to prepare test cases. Also the turnaround time for tester is limited and
he needs to turn around test cases, test results and automation all within the
sprint. To make this happen we had to ensure that this knowledge gap that is
normally observed in test teams is overcome
2. Who owns the quality of systems in
production? : This has been hotly debated. I might get into trouble for
bringing this up. In an organization, this question should be answered and the
response should be unambiguous. There might be multiple teams or functions
responsible for quality or contributing towards quality but accountability
should rest with just one team. That team should take the ownership and drive
other teams towards improvement. This has been our personal experience and
fixing this will solve many problems. This team should be empowered to take the
decisions that will improve quality. This one decision will take the
organization into virtuous orbit resulting in a domino effect across all other
functions
3. The perpetual automation backlog: Test Automation
is indisputably our biggest challenge. Traditionally automation has been an afterthought.
Once an end to end business process is ready it would be flipped over to the
automation team who will start with understanding the business process. Also
lot of stress is put on identifying the business case for automation. Once
there is a business case approved, test automation is taken up as a separate
project. Later once automation is completed it is added as part of the
regression suite and executed there after forever with no updates (in most
cases). But for continuous delivery using agile development, test automation
cannot be a one off project that is done on a cost benefit analysis decision.Automation has
always been lagging when compared with development or manual testing. For
success with continuous delivery this gap should be minimized or completely
eliminated. At one level, it does need change in the way we look at automation
and the framework but the biggest challenge that we had to overcome was how do
we does this within the sprint. How do we enable our teams at sprint level to
automate the user stories across multiple teams under the constraints? We are
in the process of changing the mindset of both functional tester (FT) and automation
tester (AT) and motivating them to become ‘FAT’
testers (functional automation testers)
4. Flexible automation framework:
Automation frameworks are traditionally built to manage end to end business
processes that run during regression testing. The supporting automation for
test automation like test data automation, exception handling, start stop
scripts will change when we look at automating user stories and use that as the
basis for building end to end automation for cross functional business processes.
It gets further complicated when the automation scripts needs to cut across
multiple package applications and also automation scripts written in different
technologies need to be integrated to form an end to end chain. This framework
is called as the ‘CONTest framework’
(continuous testing framework)
5. Develop a balanced view on nonfunctional
testing: Nonfunctional testing is becoming extremely important in the
digital era. But nonfunctional testing can really the exchequer if we go
overboard on it mainly due to the tools used in this area. Most of the open
source tools are not very effective and licensed tools are exorbitantly
expensive and the volumes do not justify the investment. So it is important to
do a profiling of the application for various parameters like performance,
security, load etc and determine the usage patterns. This should be done at the
starting of the project and this requires a mindset change in the development
organization. So change for the test team to establish the right nonfunctional
requirements and test them at regular intervals
6. Look beyond functional quality: All
said and done testing as an execution activity comes only after development.
Our challenge was to come up with ideas that will intercept the development
activities much earlier. But at the same time over doing this activity will
increase the cost tremendously. So essentially we had to overcome 3 challenges.
First challenge was to identify areas where we as independent testing team can
intercept before we start test execution and provide improvements which are
preventive in nature. Second challenged was to find optimal balance between
velocity and cost to bring in these preventive activities. Third challenge was
combine these findings with functional findings that we would find during our regular
functional testing.
7. Integrated test tooling: To enable
continuous delivery, clarity on end to end process value chain is required.
Once this end to end value chain is clear, the process should be automated
using tools. Customers traditionally have invested in point tools which do a
certain activity well. The tools usage also has led to silo teams. BAs use a
requirements management or modeling tool, developers use a IDE and
configuration management tool and testers use a test management and test
automation tool. All these tools are standalone and don’t talk to each other.
So it is important and also a challenge to create this end to end to value
chain and tag the change to test so it assure business about quality.
These are challenges that large organizations face at enterprise
level. Continuous testing cannot be just achieved by test automation. Other
areas like environment provisioning, deployment, build management should be
automated to take benefit from the high levels of test automation. May be these
are topics for future blogs.
Any critique and
comments are sincerely appreciated.
PS: We were able to overcome most of the challenges
mentioned above and are moving forward to reach next levels of maturity which
is enabling us to help the customer implement a virtuous improvement cycle
across the enterprise.