Technological and practical items to consider when evaluating SAST Tools
As the prevalence of the DevSecOps paradigm continues to grow and organizations continue to realize that baking security into every stage of the software development life cycle is not only important, but critical, to the security stature of the organization as a whole, the necessity of creating informed decision-making processes around which tools and technologies are best suited for their needs also increases. One way that an organization can enhance their DevSecOps Security Model is to integrate Static Application Security Testing (SAST) tools into their CI/CD pipeline to facilitate automatic identification of security flaws in code written by developers on a per-commit basis; however, choosing the correct SAST tool can be a daunting task given that the number of products on the market continues to grow and each organization has unique processes and infrastructures that the tool needs to integrate with. In this article, we aim to provide guidance for organizations that have decided to integrate a SAST tool into their CI/CD pipeline and outline important things to consider before acquiring one.
Stakeholders
Before beginning the journey towards selecting and acquiring a SAST tool, stakeholders should be identified and included in the process. A few potential stakeholders in the selection process could include security specialists, members of the DevOps team, and members of development teams.
Security Engineers
Whether it’s an internal security team, or a third party brought in to assist in the decision-making process, it is critical to include dedicated security personnel in the SAST evaluation process. Security engineers can help identify the value of the results that the SAST tool produces for the languages and frameworks that the organization plans to scan as well as help determine what configurations can be made to the tool to help reduce false positives while also minimizing the chance that real issues are excluded from the results.
DevOps Teams
The organization’s DevOps team should be included in the selection process to determine how well a given tool would integrate into the existing infrastructure. While security personnel can evaluate the results of the scan, the DevOps team can evaluate the level of effort needed to implement the tool, any features that could assist in automating the scanning process, and determine if a given tool poses a time hindrance that is too disruptive to the rate at which code can be developed and deployed to production environments.
Development Teams
As development teams are ultimately responsible for patching security issues that are identified by a SAST tool, it is important to establish a healthy relationship between developers and the results of a SAST scan. For this reason, developers should have a voice in the selection process to help determine if a tool provides functionality that facilitates good development practices without inhibiting development speed and efficiency.
Technical Considerations
Once the stakeholders and their roles in the decision making process have been identified, the evaluation process can begin. Unfortunately, there is no single best SAST tool for every possible CI/CD pipeline configuration; therefore, several environmental and technical aspects of a SAST tool need to be evaluated before selecting the product that best fits the needs and requirements of a given organization.
Language Support
The most important aspect of a SAST tool is its support for languages that are being used in the applications that will be scanned. Open source solutions such as bandit for Python and gosec for Golang are effective at identifying vulnerabilities in their respective languages, but it is not likely that the entire application portfolio that an organization wishes to scan is written in a single language. It is possible to use several different SAST tools in a CI/CD pipeline to scan their respective languages then coalesce the reported issues from these tools into a vulnerability management tool such as Defect Dojo, but it is often more efficient and practical to use a single solution that supports all of the languages that exist in an application portfolio.
Accuracy
One of the main goals of Agile development frameworks is to deploy code to production early and often. Forcing developers to take time out of their day to triage potentially hundreds of false positives is inherently antagonistic to this goal. A SAST tool is only as valuable as the true positives that it identifies.  Selecting a tool that has a low false positive rate while maintaining a high true positive rate for the languages that will be scanned is imperative to the successful adoption of the tool into the development workflow.
Quantitative analysis of the accuracy of a SAST tool can be useful when comparing tools when it comes time to make the final decision on which tool to acquire. OWASP Benchmark quantifies the accuracy of security analysis tools using Youden’s Index.
The formula shown here will output a number between -1 and 1 where:
- 1 designates that the scan identified all vulnerabilities in the code correctly while reporting no false positives
- 0 designates that the scan reported an equal amount of true positives and false positives
- and -1 designates that the scan reported only false positives and no true issues
The closer the result is to a 1, the more accurate the results of the scan were. Additionally, a comparison can be made between scans that result in similar Youden’s Indexes by graphing the results.
Two scans can result in the same Youden’s Index and while falling in different places on the graph above. Results that are closer to the upper right corner of the chart report more total issues while ones that are closer to the bottom left report less total issues. These results allow an organization to do a qualitative analysis and decide if they would rather have a SAST tool that reports more false positives but also identifies more real vulnerabilities or a tool that reports less noise but is more likely to miss real issues.
Value of Results
A SAST tool’s accuracy is not the only important factor in the value of the results that it outputs. For a SAST tool to be valuable, it needs to be able to provide developers with actionable steps that can be taken to not only mitigate a single instance of a vulnerability, but mitigate the risk of an entire vulnerability class. This can mean providing a comprehensive description of the issue that includes source-to-sink analysis, impact and criticality of the issue, and suggestions on how to fix the instance identified and how to prevent new instances of similar issues in the future. The description should also be easily understood by a developer that may not be an expert in the field of security.  While SAST tools provide value by identifying issues in code before they are deployed to production, they can provide even more value if they serve as an education tool for developers.
Scan Time
No matter how efficient and effective a SAST tool’s results are, if a scan of a large repo or commit takes a long time to complete the effectiveness of the remediation process will degrade. Long scan times could potentially extend build times or return results to developers after a piece of code has already been moved into a production environment.  To honor the Agile workflow, the SAST tool should be able to operate quickly to minimize any additional time added to the development and deployment process.
Configuration
It can sometimes be the case that a SAST tool can be configured to increase the accuracy and value of results. Understanding the complexity and modularity of the configuration options available in a tool can help determine the level of effort required for the initial set up of the tool and the level of effort required to integrate new repositories as they are added. Does each scan have to have its own configuration, or is there a way to set a global configuration that can be used in each scan?  Does the tool provide capabilities to add scanning rules such as identifying custom sanitization libraries that prevent common injection vulnerabilities and, therefore, not report said vulnerabilities?
Integration
A SAST tool that is valuable to an organization is one that integrates well with the existing infrastructure out of the box. For example, say that an organization’s existing infrastructure uses Jenkins as a build and automation tool and Jira as a ticketing system. Does the SAST tool have a Jenkin’s plugin that provides fine grained control over scan configurations and how the tool interacts with the build process that also receives frequent updates? Does the tool have the capability to seamlessly integrate with Jira to automate ticket creation to get the issues into developers’ hands as they are identified?  Does the product have the capability to plugin to developers’ IDEs to allow them to run scans before they commit code to a repository? Is the SAST tool only offered as a hosted solution, or are there on-prem solutions available?  These are just a few examples of questions that should be asked by the DevOps team during the selection process.
Administration
Administration of the SAST tool goes hand-in-hand with the integration of the tool. Identifying how simple it is to administer updates to the tool, whether or not the tool supports authentication methods that are used by the organization (e.g. SAML or LDAP authentication), and evaluating security logging features provided by the tool are all important to determining the effort required to manage the tool.
Conclusion
There is no silver bullet SAST tool for every organization and creating a defined process for making informed decisions about which solution or solutions work best in a given infrastructure, application portfolio, and organizational structure is often an important and overlooked practice that can be invaluable during the transformation towards a mature DevSecOps model.
