Commonwealth Bank has doubled the number of software changes being delivered to production, while decreasing the volume and duration of incidents, a result it credits to a DevSecOps and engineering platform transformation.
Speaking at AWS re:Invent 2024, chief technology officer Rodrigo Castillo said DevSecOps had also delivered cultural changes inside of the bank, with end-to-end ownership over service delivery.
“In just a year we have doubled the number of changes delivered to production and reduced the number of incidents that are impacting our customers in a higher proportion, and the duration of those incidents as well,” Castillo said.
He spoke to a slide that showed more or less a complete reversal since FY21, when the bank was impacted by a higher number of incidents relative to the amount of software changes being made.
For incidents that still occur, Castillo said the bank has adopted a “blameless culture”, coupled with weekly operational reviews - mirroring a post-incident activity AWS itself that gets all teams to come prepared to deep dive into problems, and randomly selects which teams present their findings.
In its technical documentation, AWS notes that this “pushes teams to maintain high-quality operational dashboards that reflect the real-time health and performance of their services.”
“More problems are being solved from the root, and our time to resolve incidents has reduced to half,” Castillo said.
Castillo said that engineers pushing to production are supported by “highly automated” capabilities and tools that enable more security and quality checks in the development lifecycle.
He said the bank had seen a “4x increase in the velocity of the cyber reviews” and similar improvement levels in “the way we monitor compliance with our controls.”
“We used to perform assurance of around 2500 controls attributes per year,” he said.
“With this model, we are doing more than 12,000 per month, so it’s a huge increase, and we still have a lot more control assurance processes to be automated, so we are just starting.”
Castillo said that security, resilience and reliability would always be top priorities for the bank.
Underpinning all of this is a “12-capability model” that teams are measured against.
However, as teams vary in their maturity with the various capabilities, they have the “flexibility to work where they are in most need of help.”
“Some things can be more mature - testing, for example - and we don’t want them to focus on that if they are already mature,” Castillo said.
“They might choose [instead] automated security or automated control assurance to work on [because] it’s where they need the most help.”
Security academy
Hundreds of engineers have been put through a security academy to help them take more responsibility for the security of their output.
“Today, engineering teams are doing the majority of their security designs,” Castillo said.
“They are taking end-to-end ownership of their services, security included. They don’t see that security is something that another team will do for them - it’s being done by them within their team.
“The second version of our security academy has been launched, providing new modules to continue developing our engineering teams and training them on security.”
Aside from making more changes more often, with fewer incidents, Castillo said that engineers were happier post-transformation.
“We have seen our engineering NPS [net promoter score] double in the past four quarters,” he said.
“They feel that they can contribute more without creating security vulnerabilities or technical debt, and they feel more valued.”
An accompanying slide noted that "67 percent of engineers feel they can work at a pace that does not contribute to incurring technical debt or security vulnerabilities", while "82 percent of engineers feel valued for their engineering skills in [the] organisation."
Ry Crozier attended AWS re:Invent 2024 in Las Vegas as a guest of AWS.
