What the Asian Financial Sector can Teach us About Continuous Testing
DevOps is a hot cake in the Asian financial sector these days. From talent to tooling, there is a hunt for acquiring the best, and many Asian banks have already built or are building their capabilities in this space.
As a consultant, I’ve spent a lot of time talking to senior leadership from many of these organisations and discussing their strategy around software quality and what they are doing to achieve it. After all, DevOps is as much about quality as it is about speed.
Interestingly, most of the time I get feedback that they are doing well with testing as they’ve automated 70-85 percent of regression tests. But when I ask the dreaded question regarding defect ratio and re-working in the backend, often the answer is: "Everything is going well except all that automation has not helped much with catching defects early."
This is the moment I put forward the concept of continuous testing (CT) and its various aspects. By definition, continuous testing is an extension of your continuous integration and continuous deployment (CI/CD)—it’s much more than just automation. It’s about testing becoming everyone's responsibility in the software development lifecycle (SDLC), from the time user stories are captured to the time you release your features into production.
For most banks that are practising agile development, testing has been made part of the development team’s role, which is the right step to test early and test often. But software testing by nature largely depends on several factors, including surrounding applications (for SIT), realistic data (for UAT), interface responses (for unit/API tests), and much more.
The result? In most cases, testing is not completed as part of the sprint cycle, delaying release and often compromising on quality to gain speed.
At the heart of continuous testing is a shared ownership from business analysts to developers to operations teams for achieving shift left, and there are vendors offering tools helping in different aspects here. This is also an area where I spend considerable time educating my customers.
Based on my experience, there are six strategies I’ve found to be successful with continuous testing:
- Build tests for requirements. Some Asian banks have already started doing this for green field projects by adopting test-driven development (TDD) and behaviour-driven development (BDD). For BAU projects, it requires a two-pronged approach: 1) refining and baselining your current test bench and 2) building the discipline for new feature development. These can considerably improve feature quality and customer experience.
- Optimise with automation. In many financial institutions, regression suites pile up with each release. Instead of running more tests (even when they are automated), IT leaders and practitioners must aim to optimise those suites using the right tools. I know of a bank that is running close to 3,500 test scenarios in their regression bench every time a feature is added to their core banking application. The amount of testing at each phase becomes a key question to address, and being self-aware of what was not tested is equally as important.
- Test in a production-like setup. Today with cloud computing, configuration management, containers, and data virtualisation, software teams can build a production-like copy of their environment within minutes and get feedback on defects much earlier in the test lifecycle. The key here is new technologies reduce the total cost of ownership (TCO) of these environments as they can be spun up and torn down on-demand. In most Asian banks, test environments are often limited or shared to keep the running cost low, but on-demand test environments can address this problem.
- Eliminate constrained dependencies. In most Asian banks, the constrained dependency in the core banking system reside in the mainframe or AS/400. Almost all transactional tests eventually have to hit these mammoths. Payment gateways and third-party dependencies create another set of challenges. All of these can be addressed with intelligent virtualisation of these systems. This can range from mocks, stubs, service virtualisation, and most recently data virtualisation of mainframes, leading to significant improvements in the quality of testing while keeping costs lower.
- Leverage cloud for performance tests. With central banks across Asia easing down on cloud adoption, more and more banks are exploring the idea of testing for scale, reach, and performance tests in the cloud. Performance test labs are one of the most expensive setups in most banks today. Hence, leveraging the elasticity of a multi-cloud environment gives the best results and helps keep down costs.
- Address data security automatically and consistently. When it comes to data security in lower (development and test) environments, Asian banks are in a more advantageous place than their global counterparts in Europe and North America, as the laws around data breaches are less punitive in Asia. But most of the Asian banks still practise data security measures like masking, hashing, and encryption to some extent. The challenge here is that these are often done manually or by scripts (maintained manually with release) and are not consistent across upstream and downstream applications. A single strategy of consistent data security can address both compliance and quality significantly.
IT leadership in most banks have now started grappling with the idea of continuous testing because frankly, many companies are just starting to understand and build modern CI/CD pipelines, precisely because software testing has a big scope to improve all aspects of culture, process, and technology.
In sum, continuous testing is critical to DevOps success. Most Asian banks have a separate team handling test automation and keep them separate from their central CI/CD teams. But as noted previously, test automation is not continuous testing, and capabilities like on-demand test environment provisioning, effortless data refreshes from production, and automated but consistent masking can be achieved only through a central team driving adoption and governing policies.