We welcome guest blogger Bill Dickenson, an independent consultant and former VP of Application Management Services for IBM, who brings decades of experience in application development and DevOps. Dickenson’s post below discusses how using CAST’s automated software analysis and measurement solutions helps achieve the benefits of DevOps, while eliminating the risks.
The recent move to cloud based development/operations (DevOps) is changing the testing and development lifecycle by accelerating the speed that code can migrate from development, through testing, and into production. Cloud based testing environments can be instantiated and refreshed at an unprecedented speed.
One of the key advantages to DevOps is the ability to create a parallel testing environment. And it is widely assumed that the ability to create environments quickly should improve the testing process. While that is clearly true for the functional testing and validation of requirements in code, code quality tends to degrade. As the speed increases and the focus on user acceptance and functional validation has increased, the number of defects moving into production is also on the rise. The nature of these defects is significant, including security violations, missing or incomplete code blocks, and unmaintainable code. It almost appears as if the tradeoff in functional testing is with code quality.
Avoiding this pitfall requires an adjustment that is well within the capabilities of most IT shops. DevOps requires that many of the supporting processes change to support the notion of continuous delivery of service. Unlike the traditional processes with “end” states, DevOps features a continuous process.
Specification –With releases being more frequent, design documents should be produced more frequently and with emphasis on business value capture, rather than code. Typically, design documents should be aimed at a one month cycle. Within one month, the tasks can be accomplished predictably. Outside of one month, the accuracy of the estimates begins to degrade and more importantly, the ability to deliver in continuous releases. Metrics for specification include:
- Business Value for change: This is becomes key to the priority. It is also a feedback look to the business line that requested the change.
- Estimated Function Points or Story Points: Having a basis for estimating will generally improve the accuracy and reliability of the estimates. In later sections, there will be a feedback loop based on automated metrics.
- Risk factors and Value blockers: In any change, there are events or connections that would stop the business value from being realized. Many times knowing these can change the design.
- Estimates based on function points: Estimating tools tend to produce consistent results through their range while estimates based on “experience” or “gut feel” tend to become less accurate as the estimates grow. Incorporating a feedback loop will create a self-correcting estimate process. As DevOps accelerates the rate of change, having the right changes in the queue will deliver more business value.
Baseline: An easily overlooked step is to baseline the metrics for the current code base. Running an analysis on the existing state of applications produces a baseline set of values which will be used to guide the process. One of the goals in DevOps is to continue to improve the code base and not allow defective code to creep in and destroy the value of more frequent releases. As support costs are also directly related to code quality, every attempt should be taken to reduce this burden. CAST produces an overall structural quality dashboard that gives an excellent view of the current state of architectural and code quality and the improvement trends.
Coding: In a continuous cycle, there is no way to batch up requests. Instead, automation of processes becomes key. Software quality becomes essential as speed allows code to migrate into production without inspection. Random selection of tools will not achieve the goal. It is important to have a few key tools. CAST provides all of the required metrics and then some. Metrics for coding:
- Function Point counts based on each week’s development. This change is more important than it initial appears. Tracking function points from week 1 through week 4 will provide data for estimation and when coupled with code quality, will provide a key metric around function point by software quality which will become the basis for estimating new projects.
- Code Quality: CISQ/OMG have established standards for code quality that have a direct relationship to cost of maintenance. While business functionality is key, cost of maintenance can erode business value. Code produced must meet with the standards required. The average support budget is usually forecast as 20% of development. Actual studies have indicated that well written code comes in around 5%.
- Transferability/Changeability: In a DevOps environment, code is released more frequently and the emphasis is on multiple streams of development. The ability to integrate quickly becomes essential. The transferability index is an indication of code clarity. Code with high transferability can be changed/maintained quickly and in a DevOps environment is essential to success.
- Critical Violations/Security: Without question, Security and “open” code has become a critical topic. Surprisingly, there are few metrics around it to ensure poor code does not make it into production. While these vulnerabilities can be detected through inspection, the DevOps process occurs too fast for manual processes.
Testing – Testing in DevOps is and should be focused on functional requirements, and errors related to specifications. But testing for code quality and completeness cannot be neglected. Non-functionaly metrics to be examined during integration testing are:
- Efficiency: One of the inherent characteristics of DevOps is the focus on smaller, more continuous development which tends to have more calls to external resources. The efficiency metric examines such calls and identifies those that could affect system performance. Repair at this stage will save performance issues later.
- Green Index: CAST provides a collection of that examines performance and other machine consumption factors which have a correlation to operations issues.
- Risk Factors: There are a collection of risk factors which give indications of the chances of release failure. As DevOps becomes more routine, managing risk becomes essential. Using this index will allow organizations to decide how much risk they can tolerate per release.
Release Packaging: Release speed is one of the most exciting aspects of DevOps, but also one of the most misunderstood. While it is true that companies such as Amazon may achieve multiple releases per hour, speed is truly a product of business need. However, because releases can be done more frequently, they tend to be less risky and less complex.
Each application and in fact each business area has a different need for speed. Not every application benefits from more continuous release cycles. However the discipline required is the same, whether it be weekly, monthly or even hourly releases.
Production Monitoring: While the steps above will dramatically reduce the code anomalies, as release speed increases, additional bugs will get into the system. As a result, some of the metrics above focus on the ability to fix code rapidly. Transferability and changeability seemed like overkill in code development but their value becomes apparent when in repair. The goal should be to position fixes to be executed cleanly and correctly.
DevOps offers an opportunity to do great things for the businesses we serve but they also bring some built in risks. Using automated software analysis and measurement solutions such as CAST allows you to achieve that benefit without the risks degrading the hard won benefits. The right metrics and greater visibility can change everything.
For more information on Bill, please reach out to him on LinkedIn here.
