Introduction
Our approach towards implementing agile testing method was very different from the
traditional testing practices. We recently successfully completed testing a product using
this approach. It was a rich learning experience and helped us in gaining the knowledge
of testing products in an Agile environment. The product, that we were supposed to work
on, was completely a new product for us and we had no prior knowledge of it. We were
responsible for the first release of this product. The following sections of this paper
identify our methodology in successfully releasing it.
Challenges Faced
Initial Hiccups
• There was no formal documentation provided by the client: Requirements were in
form of user stories. It was a very high-level definition of any requirement containing
just enough information, which made it quite difficult to understand the scope and
hence draft the test cases.
• There was no prior knowledge of the product: The product was an entirely new
one and we were unsure of the workflows, which made it difficult to prioritize the
business and technical requirements.
Roadblocks during the course of the Project
• QA build was short-lived: Due to the tight project deadlines, and frequent changes
in the requirements, the testing time was getting continuously shortened. Initially,
biweekly QA builds were delivered, but it later got reduced to daily nightly builds,
which posed a great difficulty in managing our QA efforts.
• Time to update the test case documents was not available: Requirements were
never frozen throughout the life cycle of the project. So, the documents (test cases,
regression test cases) required frequent updates. With QA build cycle getting
continuously shortened as the release date approached, the update task was getting
quite difficult to accomplish.
• Managing the regression task: As per the client’s instructions to perform the
regression as frequently as possible to ensure that no fixed bugs get reopened, or the
changes in one module doesn’t affect the other modules, it became essential to
perform regression testing with wide coverage. Additionally, keeping track of the
bugs and updating the same in the tracking tool was equally important.
Are we ready for release?
It was challenging for the testing team to provide a definitive to answer to this question
because all the QA efforts were employed in a very short time frame. With the kind of
build cycles being followed and the corresponding testing, this was a question that was
difficult to answer.
Our Approach to overcome challenges in Agile
Efficient Documentation – Capture the essence, not the details!
To minimize the risk of over-documentation, thereby wasting time on test cases which
might get irrelevant with the changing requirements in the successive builds, we ensured
that our documentation was highly effective and required minimal changes in case the
requirements changed.
We ensured this by using the following techniques:
• Maintained test scenario sheets: A test scenario sheet is a matrix which includes all
the possible flows which could be performed by a user in the application. In this
sheet, the steps to execute these scenarios were not included. It was assumed that
while execution the tester would be well aware of the steps to follow for these
scenarios.
Payment Mode/Order Type Pickup Carry Out On Hold Backorder
Cash × × × ×
Check √ √ √ √
Gift Card √ √ √ √
Credit Card × × √ √
×-Scenario Passed √-Scenario Failed
Table 1: Test Scenario Sheet
• Maintained review comments sheet: The documents we prepared based on the
communication with the client, were sent for review and the review comments on
these documents were maintained in a sheet known as the review comments sheet.
So, instead of incorporating the changes in our documents we referred to the review
comments sheet for any validation. So, it helped us to save some time in investing our
efforts in documenting the changes.
• Maintained activity flow diagrams through mind mapping techniques, where-in
we mapped the user’s possible flows down the application.
Figure 1. Activity Flow Diagram
Exploring the product – Understanding the business
As the product was new to us and we had no prior knowledge, we initially started with
exploratory testing and used software from the end-user’s perspective within its overall
environment. This rough test helped us to uncover all the critical functionalities, defect
prone areas and few important failure scenarios. Also, it helped us in prioritizing these
features and areas in our future testing course.
Active Communication – Let’s talk!
We adopted an approach where-in we interacted with the client and the onsite
Development team, not only through the formal ways of communication like client calls
and emails, but also through informal chats like IM and calling directly on their cell
phones in case of any issues.
We believed that it would be a good approach to reach the client to clarify our issues on
that day itself, rather than to wait for their response through any formal means of
communication.
Also, we helped the onsite Development Team in reproducing the logged bugs in case
they had any issues surrounding the bugs. We kept this communication as informal as
possible, which established a comfort zone between the Development and the QA team.
Constant Feedbacks
We ensured to have frequent bug triages, where-in we discussed on the priority of the
bugs which need to be fixed and the bugs which need to be deferred. Also, we provided
feedback on the design features from the customer’s perspective.
As the QA builds were short-lived, we discussed regarding the prioritization of our
testing efforts and requested the client and the onsite Development team a list of the
impacted areas and new modules, modified or added in the coming build which helped us
in further focusing on the impacted areas accordingly.
Identification of different sets of test cases – Categorization: It always helps!
We identified and continuously prioritized executable and regression test cases with each
new build. We felt the need to identify two sets of test cases:
• Executable Test Cases: Test cases which could be executed for the features/modules
which have been built.
• Regression Test Cases: Test cases which needed to be run on daily basis to ensure
that the newly added features do not affect the flow of the previously working
modules.
Increasing the scope of the bug tracking tool
We used JIRA as the bug tracking tool in our project. Apart from logging and tracking
bugs we requested the client to raise any kind of requirement changes as a change request
in JIRA. This helped us in keeping the teams in sync and helped the testing team to
incorporate the changes accordingly in the next test cycle.
Yes, we are ready for release – Let’s go live!
When the time of the release approached, we sat with the client and judged the readiness
for the build release to production. Instead of looking at the number of open bugs, we
analyzed the consequence of these bugs on the overall application.
In couple of meetings with the client, we were able to identify the bugs which can be left
open and does not pose a serious threat for going live. The status of these open bugs was
changed to deferred and it was decided that these bugs would be resolved in the next
version.
Benefits of our Approach
Adaptive Testing Approach – Plan as we Progress
Though the testing time was less but still we were able to execute our testing efficiently
which might not have been possible had we followed any of the traditional testing
approach.
Encouraged Futuristic Thinking
The brainstorming sessions between the client, the development team and the testing
team helped encourage the team members to provide some valuable feedback back to the
client about the product. The client highly appreciated such lateral thinking and decided
to implement a few of those suggestions as enhancements in the future release.
Identification of show stoppers at early stages
Running the prioritized regression tests with each new build helped us to identify the
critical bugs affecting the previously built features, when incorporated with the new
modules.
Timely Delivery
With this approach, we were able to manage testing activities efficiently and hence give a
green signal for the product to go live despite the tight deadlines for testing.
Avoiding waste in documentation
With our new technique to manage documents efficiently, we were able to avoid a lot of
waste in terms of time and resources required for the documentation process.
Improvement Areas
Adding Automation for Test Management
To speed up the entire effort, we could have automated the regression tests which were
being executed in each new build.
Increasing client awareness regarding our approach for future products
As this approach was planned as we progressed with the project, it was not well
structured. But we were able to initiate the process at our project level to make the client
aware of the process we followed throughout the product delivery cycle. This made them
well-informed about our working method of testing under Agile environment.
Conclusion
The successful delivery and our experience with this approach made us believe that
testing can be well justified even in Agile environment.
