The Holy Grail: Objective risk level estimation


In my last post we discussed the complimentary nature of remediation cost and risk level assessment. As a follow up, I wanted to dwell on the objective risk level assessment. Is it even possible? If not, how close to it can we get? How valuable is an estimation of the risk level? Could it be the Holy Grail of software analysis and measurement? Or is it even worth the effort?

Risk level

By definition, risk level is about the severity of the consequences of a situation and the probability for this situation to happen. A high-probability, low-impact situation can then "compete" with a low-probability, high-impact one for the riskiest situation award. But when dealing with software, what could be a good indicator of risk severity and risk probability?

This is a good starting point to look at objective risk level assessment, especially since software analysis and measurement is all about measuring things anyway.

Hopefully, when looking for risky patterns to measure in the source code, application structure, and architecture, the reasons to look for the patterns is well-known. What you’re not going to be able to measure are the consequences of these risky patterns on the organization's activity. Would it cause revenue loss? Would it cause a downgraded customer satisfaction? Would it prevent the release of the "killer" Christmas offer?

This information can be collected, but not analyzed and measured from the software itself. However, this missing link shouldn't stop us in our tracks.

Then, when looking at the way an application is built, (at its various execution paths, etc.) one can get an idea of how frequently a faulty piece of code or pattern is used. However, what we can’t see is the frequency with which end-users will use any given application feature or cause the application to use any given specific execution path. Is it a pattern that is used by a popular transaction from the web-facing shopping site? Is it a component used throughout the application as it participates in application cache management?

Risk severity level estimation with SAM

Some risky patterns, but not the vast majority, can be “direct hits.” They prove that risk is real, and waiting to happen. This makes estimating their severity a lot easier. The business context will still have to be found outside of the software itself (the caveat stated above is acknowledged) but, other than that, the outcome is obvious. For instance, CWE patterns such as cross-site scripting will allow for phishing, which can be incredibly risky if the application is web-facing.

The majority of risky patterns are not direct hits. They are evidence to support the trend that risk is real and very much alive in the enterprise. In this case, the amount of evidence will be the next best thing to proving the risk is real.

For example, let's talk about complexity. I am not a big fan of the "complexity is bad" argument. I am more in favor of "unneeded complexity is bad." ("No Silver Bullet,” in which Fred Brooks differentiates essential complexity from accidental complexity). Of course, determining the "unneeded" part is the true challenge.

I wouldn’t spend much time focusing on cyclomatic complexity. I would look at many complexity-related indicators to pinpoint the components that concentrate most if not all of them. I can still be proven wrong, but the component’s bad smell is so strong that I am not often disappointed. Comparing components with high cyclomatic complexity alone and components with high cyclomatic complexity, high integration complexity, high essential complexity, high depth of code, etc. as far as complexity matters, the latter components are the real winners, as they smell stronger for some good reasons.

This kind of reasoning can be applied to areas other than complexity (I used the complexity area as it widely and better understood).

You can easily compute a "performance bad smell" when a component starts to concentrate many issues that contribute to possible negative impact on performance levels. Each issue taken individually is not a direct hit, but when they start to pile up they can become an issue.

Therefore, as a general approach, computing a "bad smell" by looking at everything that is not right with a component in a given area can help estimate, but not yet measure, the risk severity level.

Risk probability estimation with SAM

Risk probability can be addressed with or without contextual information. Contextual information is always a plus, but might not be available. Therefore, always have a plan B handy.

First, let's focus on plan A. I said earlier in this blog post that some information is missing from the source code, application structure, and architecture. However, SAM can help bridge the gap between the application innards and the business context and purpose. With a bridge, it can be a lot easier to know how frequent and important a feature using a faulty component will be in use, thus providing an estimate of the risk probability.

What kind of bridge am I talking about? I’m referring to the ability to map user-facing transactions (the application features in a sense) with the technical components within the applications that participate in the transaction execution. This bridge is objective as it relies on the actual application source code and structure. By the way, this bridge can also help estimate the risk severity because you’ll know how much damage would be caused had a transaction failed or performed too slowly.

What if I can get the information from the real world? Here is one possible plan B. With SAM, you can actually count "how many roads are there to Rome?" The roads are wired in the application source code and structure. Of course, it fall shorts of the Plan A. But it's always applicable and it's totally objective. The more roads, the higher the risk.

The Holy Grail

By nature, SAM-based risk level is incomplete, yet SAM-based risk level assessment can be automatically "objectivized." And I am pretty sure there are new ways to be devised to move forward. Do you think that objective risk level assessment is the Holy Grail for SAM, or does the future hold more in store for us? Share your ideas in a comment.


Philippe-Emmanuel Douziech
Philippe-Emmanuel Douziech Principal Research Scientist
Philippe Emmanuel Douziech is a Principal Research Scientist at CAST Research Labs and is the Head of European Science Directorate at CISQ. He has worked in the software industry for more than 20 years and is skilled at assessing software risk and quality.
Load more reviews
Thank you for the review! Your review must be approved first
You've already submitted a review for this item