Since a last decade or two, there
have been tremendous changes in the software development methodologies. Variety
of tools besides agile development, extreme programming, and test driven
development have changed the philosophy and are still modifying how the
software development takes place. Needless to say, there is a constant need for
change in developing and owning a software development. During the course of
software development project, however, there is a need of continuous
maintenance and growth of the software product as it can be hit by any kind of
unexpected and unplanned changes.
One of the biggest challenges
that is faced during software development phase is to handle traceability; a
bi-directional trace between the requirements and components implementations as
well as between requirements and acceptance tests. In such, manual maintaining
of the traceability can be cumbersome and so for providing better and
sustainable results, the need for an automated build acceptance test surfaces.
Build Verification Test or the
Build Acceptance Test is a set of tests that helps the testing team to go a
step back for a step ahead. These tests are run on Builds- each one, and newly
created one to ensure that a given module is ready for testing. These tests are
the core functionality test cases that ensure that the application is stable
and can be tested thoroughly. Typically, the entire process is automated. The
test helps subsequent stages and team side-step half baked work and builds that
are too early or un-ready for testing. When these tests give a green signal,
they lead the build to verification stages and bug-fixing tables.
In simpler words, BVTs are run on
daily builds and if the BVT fails, the build is rejected and a new build is
released after the bugs are fixed. The build tests address core functionality
test cases, application stability and are thorough in their focus. This can
entail main functions of the module, data-and-format alignment, and addition of
files or improvements, new and improved files and culminate into overall
project efficiency by making room for discovery and correction of defects
Keep in mind that the success of
BVT depends on which test cases are included in BVT. For that to happen, one
has to consider the following points:
Include only critical test cases in BVT
Write effective test cases scripts
Include right file formats
Include useful information for yes/no diagnosis
Peek into and flags around failure causes,
choice and degree of automation, insertion of break-build penalties, presence
of all new modified files etc.
All the test cases that are included should have
known expected results
Ensure that all included critical functionality
test cases are sufficient for application test coverage
ONLY CRITICAL TEST CASES
The most important point to
consider is the choice of test cases, given its regression-test flavor. There is
a lot of thrust on inclusion and exclusion hence, picking only critical test
cases and including test coverage focus is a good approach to start with. This can
facilitate new critical case with consistency of different configuration check.
This helps to avoid build failure that is prompted from modules and test cases
are unstable or new.
It is worth knowing that tests
take time, resources and server muscles and so it is pertinent to run those
tests first where propensity of spotting failures is more or the improved time
to fault detection comes in. this is helpful to shave away necessary time,
cost, resources and completeness of a build along with ease for testing teams. Criticality
and application test coverage should be balanced well in the selection of
tests. Also, important functions of the module, clarity on expected results,
verification-strength or implementable-readiness for application functions is
vital to be considered.
TESTS SHOULD BE STABLE
These tests should devote
adequate time on making cases and going for stable test cases. It is less probable
to weed out critical bugs or software-breakers if test cases are not formulated
well. That is the reason why a build has to be stable for strong and gainful
execution of test cases in the ensuing testing process. Likewise, careful
design along with the coverage of basic functionality is immensely important.
Presently, Agile is one of the
most commonly used methodology in software development life cycle, with its
pre-defined short iteration cycles for 2-3 weeks that delivers quality product.
QA teams work with developers to achieve the goal of BVT. Under BVT, the
developers set up or create the VMs/Machines for testing, set up the
environment build deployment, verify build and inform the QA team to further
If this entire process is
automated, the BVT process will easily save around 3-4 man days on each
iteration besides being reliable and without manual errors. To ensure the
success, following steps will be followed:
Set up Bamboo or Jenkins in a way that the
nightly test builds would start at the end of every sprint or manually, which
pulls the latest code from the GIT/SVN, executes unit test cases, generates
build and uploads the build on Artifactory. Once the build is uploaded; it will
start the automated BVT plan.
The VMs or machines can be generated
automatically using Amazon Web Service, Dockers or VMWare ESXi Server which
have their own required software already installed. APIs can be called using
Python, Java or even shell script, whichever is applicable.
After the VMs are available, ensure that the web
application server and database software has been installed successfully. To
automate this option, Ansible can be considered a good option.
With the latest build from Artifactory, deploy
it on Web Application server with the help of Ansible. Once it is deployed
successfully, check if it can access a web page.
After the build is deployed, start executing the
pre-defined test cases which encapsulate the core functionality of the product.
Automate those test cases as per the requirement using any language or tool
such as Java, Python, shell script and selenium etc.
Add the test step results into Jira directly
from the Test script to avoid manual dependencies.
Send the final status email to the manual QA
team that the build is stable/not stable for further testing.
This is one of the examples to
automate build verification test process in Agile.
This is quite understandable that
the automated acceptance tests play the key role in software development
process. Automated BVT is very effective for daily as well as long-term builds
and helps saves time, cost, and resources of the company. Given the nature of
the build acceptance tests, their size and scope, assortment of crucial
functionality areas would be apt that would refine the coverage of the these
tests for the overall goal of empowering the actual testing that follows.