Strategy & Metrics

Strategy & Metrics Level 1

The Strategy & Metrics practice encompasses planning, assigning roles and responsibilities, identifying software security goals, determining budgets, and identifying metrics and gates.  

[SM1.1: 81] Publish process and evolve as necessary.

The process for addressing software security is published and broadcast to all stakeholders so that everyone knows the plan. Goals, roles, responsibilities, and activities are explicitly defined. Most organizations pick an existing methodology, such as the Microsoft SDL or the Synopsys Touchpoints, and then tailor it to meet their needs. The secure SDLC process must be adapted to the specifics of the development process it governs (e.g., waterfall, agile, CI/CD, DevOps, etc.) because it will evolve with both the organization and the security landscape. In many cases, the process is controlled by the SSG and published only internally; it doesn’t need to be publicly promoted outside the firm to have the desired impact. In addition to publishing the process, some firms also encode it into an application lifecycle management (ALM) tool as workflow.

[SM1.2: 66] Create evangelism role and perform internal marketing.

To build support for software security throughout the organization, someone in the SSG must play an evangelism role. This internal marketing function helps keep executives and other stakeholders up to date on the magnitude of the software security problem and the elements of its solution. An agile coach familiar with security, for example, could help teams adopt better software security practices as they transform to an agile methodology. Evangelists can increase understanding and build credibility by giving talks to internal groups (including executives), extending invitations to well-known experts, authoring white papers for internal consumption, or creating a collection of papers, books, and other resources on an internal website and promote its use. An early example of such an evangelist was Michael Howard’s role at Microsoft just after Bill Gates’ 2002 security memo kicked off their new security strategy.

[SM1.3: 73 Educate executives.

Executives are periodically shown the consequences of inadequate software security and the negative business impact it can have on the organization. They’re also shown what other organizations are doing to mature software security, including how they deal with the risks of adopting “flavor of the day” engineering methodologies with no oversight. By understanding both the problems and their proper resolutions, executives can support the SSI as a risk management necessity. In its most dangerous form, security education arrives courtesy of malicious hackers or public data exposure incidents. Preferably, the SSG will demonstrate a worst-case scenario in a controlled environment with the permission of all involved (e.g., by actually showing working exploits and their business impact). In some cases, presentation to the Board can help garner resources for an ongoing SSI. Bringing in an outside guru is often helpful when seeking to bolster executive attention. Tying education to specific development areas, such as mobile or cloud services, or particular methodologies, such as CI/CD and DevOps, can likewise help convince leadership to accept SSG recommendations when they might otherwise be ignored in favor of faster release dates or other priorities.

[SM1.4: 107] Identify gate locations, gather necessary artifacts.

The software security process includes release gates (or checkpoints, guardrails, milestones, etc.) at one or more points in the SDLC or, more likely, multiple SDLCs. The first two steps toward establishing security-specific release gates are to identify gate locations that are compatible with existing development practices and to then begin gathering the input necessary to make a go/no-go decision. Importantly, the gates might not be enforced. For example, the SSG can collect security testing results for each project prior to release then provide their informed opinion on what constitutes sufficient testing or acceptable test results without trying to stop a project from moving forward. Shorter release cycles, as seen in organizations practicing CI/CD, often require creative approaches to collecting the right evidence and rely heavily on lightweight, super-fast automation. The idea of identifying gates first and enforcing them later is extremely helpful in moving development toward software security without major pain. Socializing the gates and then turning them on once most projects already know how to succeed is a gradual approach that can motivate good behavior without requiring it.

Strategy & Metrics Level 2

[SM2.1: 49] Publish data about software security internally.

To facilitate improvement, the SSG publishes data internally about the state of software security within the organization. This information might come in the form of a dashboard with metrics for executives and software development management. Sometimes, these published data won’t be shared with everyone in the firm but only with relevant executives who then drive change in the organization. In other cases, open book management and data published to all stakeholders helps everyone know what’s going on, the philosophy being that sunlight is the best disinfectant. If the organization’s culture promotes internal competition between groups, this information can add a security dimension. The time compression associated with CI/CD calls for measurements that can be taken quickly and accurately, and might initially focus less on historical trends (e.g., bugs per release) and more on speed (e.g., time to fix).

[SM2.2: 53] Enforce gates with measurements and track exceptions.

Software lifecycle security gates are enforced for every project, so to pass a gate, a project must either meet an established measure or obtain a waiver. Even recalcitrant project teams must now play along and the SSG tracks exceptions. In some cases, gates are directly associated with regulations, contractual agreements, and other obligations, with exceptions tracked as required by statutory or regulatory drivers. In other cases, gate measures yield key performance indicators that are used to govern the process. Allowing any projects to automatically pass or automatically granting waivers without due consideration defeats the purpose of enforcing a gate. Even seemingly innocuous software projects, such as a new mobile client for an existing back end or an application ported to a cloud environment from an internal data center, must successfully pass the prescribed security gates in order to progress or remain in production. Similarly, APIs, frameworks, libraries, bespoke code, microservices, container configurations, and so on are all software that must traverse the security gates. Remember, it’s possible, and often very useful, to have enforced gates both before and after the development process itself.

[SM2.3: 52] Create or grow a satellite.

Create a collection of people scattered across the organization who show an above-average level of security interest or skill—a satellite. Forming this group of advocates, sometimes referred to as champions, is a step toward creating a social network that speeds the adoption of security into software engineering. One way to build the initial group is to track the people who stand out during introductory training courses; see [T3.6 Identify new satellite members through training]. Another way is to ask for volunteers. In a more top-down approach, initial satellite membership is assigned to ensure complete coverage of all development/product groups, but ongoing membership is based on actual performance. A strong satellite is a good sign of a mature SSI. In new or fast-moving technology areas, satellite members can help combine software security skills with domain knowledge that might be underrepresented in the SSG. Agile coaches and DevOps engineers make particularly useful satellite members, especially for detecting and removing process friction.

[SM2.6: 51] Require security sign-off.

The organization has an initiative-wide process for accepting security risk and documenting accountability, with a risk acceptor signing off on the state of all software prior to release. The sign-off policy might require the head of a business unit to sign off on critical vulnerabilities that have not been mitigated or on SSDL steps that have been skipped, for example. The sign-off policy must apply both to outsourced projects, such as a boutique mobile application, and to projects that will be deployed in external environments, such as the cloud. Informal or uninformed risk acceptance alone isn’t a security sign-off because the act of accepting risk is more effective when it’s formalized (e.g., with a signature, a form submission, or something similar) and captured for future reference. Similarly, simply stating that certain projects don’t need sign-off at all won’t achieve the desired risk management results. In some cases, however, the risk acceptor can sign-off on a particular set of software project acceptance criteria, which are then implemented in automation to ensure the criteria are applied even in the fastest processes; however, there must be an ongoing verification that the criteria remain accurate and the automation is actually working.

Strategy & Metrics Level 3

[SM3.1: 21] Use an internal tracking application with portfolio view.

The SSG uses centralized tracking automation to chart the progress of every piece of software in its purview, regardless of development methodology. The automation records the security activities scheduled, in progress, and completed, incorporating results from activities such as architecture analysis, code review, and security testing even when they happen in a tight loop. A combined inventory and risk posture view is fundamental. The SSG uses the automation to generate portfolio reports for multiple metrics and, in many cases, publishes these data, at least among executives. Depending on the culture, this can cause interesting effects via internal competition. As an initiative matures and activities become more distributed, the SSG uses the centralized reporting system to keep track of all the moving parts.


[SM3.2: 6] Run an external marketing program.

To build external awareness, the SSG helps the firm market the SSI outside the internal teams. In this way, software security can grow beyond being a risk reduction exercise and instead become a competitive advantage or market differentiator. The SSG might publish papers or books about its software security capabilities, or have a public blog. It might provide details in external conferences or trade shows. In some cases, a complete SSDL methodology can be published and promoted outside the firm, with mobile, cloud, and new technology security projects making important software security case studies. Regardless of venue, the process of sharing details externally and inviting critique is used to bring new perspectives into the firm.

[SM3.3: 14] Identify metrics and use them to drive budgets.

The SSG and its management choose the metrics that define and measure SSI progress in quantitative terms. These metrics in turn drive the initiative’s budget and resource allocations, so simple counts and out-of-context measurements won’t suffice here. One such metric could be security defect density, a reduction in which could be used to show a decreasing cost of remediation over time. Recall that, in agile methodologies, metrics are best collected early and often in a lightweight manner. The key is to tie technical results to business objectives in a clear and obvious fashion in order to justify funding. Because the concept of security is already tenuous to many business people, making an explicit tie-in can be helpful.

[SM3.4: 0] Integrate software-defined lifecycle governance.

Organizations begin replacing traditional document, presentation, and spreadsheet-based lifecycle management with software-based delivery platforms. Humans, sometimes aided by tools, no longer drive progression from each lifecycle phase to the next. Instead, organizations rely on automation to drive the management and delivery process with ALM/ADLM software, such as Spinnaker, and humans participate asynchronously (and often optionally), like services. Automation often extends beyond the scope of CI/CD to include functional and nonfunctional aspects of delivery, including health checks, cut-over on failure, rollback to known good software, defect discovery and management, compliance verification, and a way to ensure adherence to policies and standards. Some organizations are also evolving their lifecycle management approach by integrating their compliance and defect discovery data to begin moving from a series of point-in-time go/no-go decisions (e.g., a security test at each gate) to a future state of continuous accumulation of assurance data (e.g., output from sensors embedded in development and production).