STAREAST Software Testing Conference – May 1-6, 2016 – Orlando, FL USA

Date : May 1–6, 2016

Location : Orlando, FL

Venue : Renaissance Orlando at SeaWorld®

The Premier Event for Software Testing and QA Professionals

STAREAST is one of the longest-running, and most respected conferences on software testing and quality assurance. The event week features over 100 learning and networking opportunities and covers a wide variety of some of the most in-demand topics.

• Test Management
• Test Techniques
• Test Automation
• Agile Testing
• DevOps & Testing
• Testing the Internet of Things
• Mobile Testing
• Testing Metrics
• Cloud Testing
• Performance Testing

[Know more about the Conference]

Is Production Environment Really Sacred?Testing in Production

I was once tasked with testing a service that was integrated with ATM. We approached a 3rd party ATM Integrator to set up an ATM in our staging environment to test this scenario. The cost involved was very high and our testing budget did not permit us to use a third part ATM integrator. So what was the alternative? It was an unanimous decision in the program steering group to test in production, since the feature was tested properly in the previous version. A couple of internal users were tasked with testing this feature immediately after deploying the newer version to production.

Recently, I have been noticing that testing in production is becoming a popular practice, as part of the defect and incident analysis, or as a concluding test before going live, aiming to eliminate uncertainty, and give confidence to management and operations teams.

There are many factors why organizations are forced to test particular scenarios in production. First, with shrinking IT budgets, organizations are having difficulty in creating test environments that represent the full functionality the production environment contains, some of these include, load balancers, ATMs, SMS Gateways, billing, etc. Most of these aspects are tested using simulators in test environments, but ideally organizations would like actual integration with all these hardwares/softwares in the staging environment, to perform UAT before moving to production. At the same time, business process are becoming more and more integrated, which in turn demands test environments that are connected end-to-end. It is always a challenge to create a test environment that is equal to production.

With advancement of technology, the nature of services that are rolled out is complex. One of the projects I was worked on involved a mobile payment service for financial inclusion, that spanned across multiple organizations, 3rd party integrators, geographically distributed teams, and many stakeholders in the service. For example, we had to work with different stakeholders such as banks, merchants, SMS Integrators, ATMs, payment terminal vendors, billing payment aggregators and mobile payment platform provider. Add to this, all hardwares in data centers like servers, load balancers. Testing particular scenarios in such an integrated environment is possible only in the production environment (or spend huge sums of money to set up and maintain a staging environment with such a complicated integration).

Of course there are many risks associated with this practice such as creating and maintaining test accounts and test data in production, security controls and accountability, testing in production can cause production incidents, postponing real testing until deployment. (a separate blog is required to address these risks). but Testing in Production (TiP) when performed in a controlled manner within the organization’s IT policy, is a better way to ensure elimination of any remaining risks or uncertainity before GTM.

Adopting Agile Testing

The term agile is sometimes used to represent anything that is dynamic or an unstructured way of working with others. However, as per the document “Agile Manifesto” conceived by a group of software engineers, it is a focused and yet rapidly iterative software process adhering to a set of principles, and aims to promote a more efficient, humane and collaborative way of developing computer programs and IT systems.  Agile testing is the practice of software testing for bugs or performance issues that are within the context of the agile workflow. It focuses on the concept of  Whole Team, getting as much feedback as possible on code, and building the quality as early as possible. Agile testing has shorter feedback loop between product owner and team members. Agile testers are an integral part of the cross-functional team, and have their say in all the phases of SDLC (Software Development Life Cycle). The flow of information is uninterrupted due to collocated teams. Let us take a closer look at some of the key features of agile testing methodology, and why more and more companies are adopting it.

Why Become Agile?

Agile and traditional waterfall methodologies differ in terms of mentality of teams, role of testers in the team, and at what stage the testing begins.

  • In conventional testing, test execution does not start until after the complete functional development.
  • In traditional testing methods, there is usually significant delay between when the software is written and developers get the feedback. Defects and bugs are found quite late in the process which can cause serious issues if any major changes are required.
  • If the business requirement changes, it affects the already developed test cases.
  • As the communication and testing is siloed, the chances are that different groups will have different final product expectations.
  • Also, as testing is the last activity before the decided release date, the quality of testing suffers.

Traditional QA engineers rely heavily on documents, however, generating test cases may not be as cut-and-dried in agile, as agile testing QA engineers do not get big requirements documents as a base for test cases, and do not get months to write the test cases. They directly become the part of the communication streams which can be written, verbal or virtual, and collect the information they require. In agile, it is accepted that the change is healthy.  Some of the key features of agile testing are:

  1. In the Agile approach, testers and developers are seen as two sides of the same production coin that meet regularly and compare notes daily. Testers, developers and quality-assurance personnel work closely together replacing siloed functions. The split between software testers and software developers in the traditional waterfall process positions them as separate entities at different points along the production cycle, and this is the most fundamental problem that agile helps resolve.
  2. Agile testing is testing when possible, even as early as requirements gathering phase and there is Test Driven Development (TDD). Agile testing drives development by questioning stories and refining acceptance criteria during the iteration planning. It starts by working on stories, and continues with the TDD.
  3. Agile requires continuous and elaborate collaboration between stakeholders throughout the company, throughout the production process. In each and every phase of the development, testing is an essential component, and quality is built in at every stage through constant feedback from all who are responsible for the final product. Hence, the working versions with essential features of the final system are frequently produced. Each of these working systems are fully integrated and carefully tested, just like final delivery.
  4. It is a team effort where anyone can pick-up and execute a testing task. Developers and design experts also think about the testability of the product and testers can give their inputs in the early stages of software architecture or designing interface.
  5. Testing starts when the Sprint starts and entire team is responsible for the result. Testing includes test planning, build acceptance testing, functional testing, regression testing, demo testing and test automation.
  6. Testers are an integral part of the team, and their experience is fully utilized where they participate in planning and requirement analysis.
  7. The feedback loop is short and testers actively participate in the Retrospective and Planning meetings.
  8. Pair testing can be used together with the developer or other tester. Testers can also help developers create automated tests or design tests which are beneficial for both.
  9. There is continuous integration for every change committed to the source code repository.
  10. Teams switch to agile due to simplicity of the principles, transparent communication with customers and accurate planning.
  11. Agile testing may seem complex, however if implemented successfully, it leads to greater simplicity as it is more risk and priority focused, faster, adaptable, flexible and efficient.

Critics of Agile

The experience of Agile can vary, and for some, it just results in lower quality, chaotic processes, miscommunication, and a bundle of several other issues. A research was conducted by Voke Media with 200 software companies on their attempt to embrace Agile, and their report stated:

Out of over 200 participants, 64 percent said that switching to Agile Development was harder than it initially seemed.  Forty percent of respondents did not identify an obvious benefit to the practice.  Out of those who did, 14 percent thought it resulted in faster releases, and 13 percent—that it created more feedback.  Seven percent of participants noted that Agile developers were happier due to reduced future planning and documentation.

There is no doubt that there are certain challenges involved in implementing agile testing methodology. As testing happens on the fly, there is minimum documentation, which can cause problem if the team member is unfamiliar with the project. In such case, handing over the project to someone completely new can be disastrous as only senior executives are capable of taking the decisions required in agile testing. If the customer is not clear about the final outcome they want, the project can easily get taken off the track.

Additionally, agile testing principles can be quite demanding on the developers’ time, and require their commitment for collaboration throughout the project. Agile testing also requires more effort from the entire team as testing continuously gets modified or interrupted to fit the need of the situation. Agile can be quite time consuming as developers and testers spend a lot of time through the iterations to ensure the best quality throughout the project.

In the agile environment, there are frequent changes in the system, and to test each change, regression testing is required to ensure that there are no new defects in the previously developed features. Planning regression testing is difficult as agile testers are busy testing new stories for current sprint. Hence, it is necessary to have automated tests for previous sprints checked into the source repository.

Inadaptability to Agile could be due to difficulty in leaving the traditional understanding of the roles, and resistance to change. Agile is not a panacea that will solve all testing problems, however, its principles are great tools to reveal several problems, and it is up to people how effectively they are able to resolve them. Agile development and agile testing go hand in hand, and together they complete the idea of agile methodology. Frequent testing along the lines of iterative guidelines are the benchmarks of the agile testing. For the best chances of success, the testing engineers must become embedded agile team members, and embrace agile team dynamics.