We knew that the most recent findings from our 2014 CRASH Report would cause a stir among the software development community -- especially Agile advocates -- but we were pleasantly surprised by the overwhelmingly positive reception the news received.
Much of the feedback mirrored opinions we’ve heard from industry insiders, but were largely ignored by the development community who accepted ‘pure’ Agile as the epitome of development methods. Many who have worked on large business critical systems felt attention to architecture up front was necessary to avoid serious constraints or painful re-architecting later in development.
In fact many of the agile thought leaders such as Scott Ambler and Alistair Cockburn have advocated adjusting agile methods to better handle the architectural challenges of complex applications. Now we have the hard data to support their recommendations. On the large, typically business-critical applications analyzed in the 2014 CRASH Report, a mix of Agile and Waterfall methods produced code with fewer violations of good architectural and coding practice than either Agile or Waterfall methods used alone. Although many in the Agile community have rejected CMMI as process overkill, we found that CMMI Level 1 organizations produced much worse code that organizations at higher CMMI Levels.
We reached out to a few industry experts for their reactions to the report. Here’s what they had to say:
"The CAST report showing some advantages to hybrid methods combining features of Agile and Waterfall is a good study with valuable information. The industry needs more research on hybrid methods and this is hopefully a study that will lead to additional studies.
As it happens my own company reached similar conclusions from independent studies with different clients. We both agree that hybrid methods have some advantages over "pure" methods if the combination is done thoughtfully.
Agile and Waterfall, Agile and RUP, and Agile and TSP, are all useful combinations with good results for both quality and costs."
-Capers Jones, Namcook Analytics
“CAST Software’s 2014 CRASH report’s findings that Agile and hybrid methods being most effective were consistent with Galorath’s observations that Agile itself is not so much a methodology as a mind-set. Iterative or incremental development with constant feedback based on frequent builds allow course corrections when costs are low and keep a project agile (with a small a). And while SCRUM type Agile approaches can be very effective, agile benefits can often achieved using hybrid approaches…
“CAST Health factor scores for the mix of Agile and Waterfall are higher than for Agile or Waterfall approaches used separately.”
-Dan Galorath, Galorath Incorporated
“The CRASH report provides some excellent findings on software quality as it relates to development methods, CMMI levels and processes, and software size. The relationship between CMMI level and software quality certainly backs up years of research into development processes. As modular development methods (Agile, Scrum) are constantly evolving, and many development organizations are professing to be using some form of Agile, it is not surprising that ‘pure Agile’ development , with likely very loose processes, would have some issues with software quality. My opinion is a mix of Agile and Waterfall methodologies may not always create better software than pure Agile or pure Waterfall, but organizations can strive to gain benefits of both methodologies. Software development analysts will need to continue to monitor the impact that evolving development methodologies have on the quality and cost of software (both in terms of development and O&M).
As a software estimator, I look forward to using CAST’s findings to gain better insight into quality weaknesses based up development methodologies.”
-Kevin McKeel, Logapps
Industry research also shows that architecturally complex violations -- structural flaws involving interactions among components that reside in different application layers -- absorb 52 percent of defect repair effort, even though they only account for 8 percent of the total vulnerabilities in an application. That’s why an up-front emphasis on architectural quality and design will result in a more dependable and less costly application than most ‘pure’ methods.
If you’d like a summary of key findings from the 2014 CRASH Report, you can download a copy.