Structural Quality Metrics in Outsourcing SLAs


When I speak to customers and prospects trying to incorporate static code analysis into their software development processes, one of the most common questions that I get is “How do we incorporate the outputs of static analysis into SLAS?” Given the prevalence of outsourcing in Fortune 500 and Global 1000 companies, this question is not surprising. Companies have always struggled to measure the quality of products being delivered, beyond the typical defect densities measured after the fact.

To help organizations answer this and similar questions, I thought I would compile some frequently asked questions around introducing Structural Quality Metrics into SLAs.  Before I get into the details, I want to caution readers against using these metrics simply for monitoring, or as a tool to penalize vendors.  This approach invariably becomes counterproductive, and instead I recommend looking at these metrics as an opportunity to make the vendor-client relationship more transparent and fact-based—a win-win on both sides.

In addition to the FAQ’s below, for more on this topic you don’t want to miss our next webinar on May 16th.  We are pleased to have Stephanie Moore, Vice President and Principal Analyst with Forrester Research discussing how to “Ensure Application Quality with Vendor Management Vigilance.”  You can register here.

What kind of structural quality metrics can be included in SLAs?

  • Quality Indices: Code analysis solutions parse the source code and identify code patterns (rules) which could lead to potential defects. By categorizing these improper code patterns into application health factors such as Security, Performance, Robustness, Changeability and Transferability, you can aggregate and assign a specific value to each category, like the Quality Index in the CAST Application Intelligence Platform (AIP). You should set a baseline for each of these health factors and monitor the overall health of the over time.
  • Specific Rules: Quality indices provide a macro picture of the structural quality of the application, however there are often specific code patterns (rules) that you want to avoid.  For example, if the application is already suffering from performance issues, you want to make sure to avoid any rule that would further degrade the performance. These specific rules should be incorporated into SLAs as “Critical Rules” with Zero Tolerances.
  • Productivity: Amount charged per KLOC (kilo lines of code) or per Function Point. Static analysis solutions should provide the size of the code base that is added in a given release.  Along with KLOC, CAST AIP provides data on the number of Function Points that have been modified, added and deleted in a release.  This is a very good metric, specially in a multi-vendor scenario where you can see how different vendors are charging you and can set targets and monitor productivity for each vendor.

How do you set targets for Structural Quality Metrics?

The ideal way to set targets is to analyze your applications for a minimum of two to three releases and use the average scores as a baseline.

An alternative method is to use industry benchmark data.  CAST maintains data from hundreds of companies across different technologies and industries in a benchmarking repository called Appmarq, and it can be used to set targets based on industry averages or best-in-class performers.

When do you introduce Structural Quality Metrics into an SLA?

Of course, the best time to introduce Structural Quality Metrics into SLAs is at the beginning of the contract, when it is the easiest to set expectations on quality objectives based on the static analysis solution outputs.  However, if you are in the middle of a long-term contract with a vendor, you can try to make changes to the existing SLAs. A situation like this will require collaboration with the vendor to define common goals on why, how and when to use a static code analysis solution and what kind of metrics make the most sense in the context of those goals.

To hear an analyst perspective on achieving maturity in your outsourcing relationships, don’t forget to register for our webinar on May 16th with Forrester Analyst Stephanie Moore.

Tagged: Jay Sappidi
  This report describes the effects of different industrial factors on  structural quality. Structural quality differed across technologies with COBOL  applications generally having the lowest densities of critical weaknesses,  while JAVA-EE had the highest densities. While structural quality differed  slightly across industry segments, there was almost no effect from whether the  application was in- or outsourced, or whether it was produced on- or off-shore.  Large variations in the densities in critical weaknesses across applications  suggested the major factors in structural quality are more related to  conditions specific to each application. CRASH Report 2020: CAST Research on  the Structural Condition of Critical Applications Report
Get the Pulse Newsletter  Sign up for the latest Software Intelligence news Subscribe Now <>
Open source is part of almost every software capability we use today. At the  very least libraries, frameworks or databases that get used in mission critical  IT systems. In some cases entire systems being build on top of open source  foundations. Since we have been benchmarking IT software for years, we thought  we would set our sights on some of the most commonly used open source software  (OSS) projects. Software Intelligence Report <> Papers
Making sense of cloud transitions for financial and telecoms firms Cloud  migration 2.0: shifting priorities for application modernization in 2019  Research Report
Jay Sappidi
Jay Sappidi Founder and CEO at Plumsoft
Plumsoft is a leading Cloud ERP solution provider. PlumERP, built on Plumware Cloud Development Platform, is a cloud-based enterprise application that combines lower cost of ownership with innovative technology and approach for enterprises. PlumERP is one of the most comprehensive ERP solutions in the cloud and provides a unified suite that covers entire business cycles from Quote-to-Cash, Procure-to-Pay, Financial Management, HCM, SCM and Production Planning applications designed for today’s organizations and the way people work
Load more reviews
Thank you for the review! Your review must be approved first
New code

You've already submitted a review for this item