logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo logo

SSDL Touchpoints: Security Testing (ST)

The overall goal of the Security Testing practice is quality control performed during the development cycle. Those performing security testing must ensure the detection and correction of security bugs. The SSG must enforce adherence to standards and the reuse of approved security features.

Use of black box security tools in QA, risk driven white box testing, application of the attack model, code coverage analysis.
  Objective Activity Level
ST1.1 execute adversarial tests beyond functional ensure QA supports edge/boundary value condition testing 1
ST1.3 start security testing in familiar functional territory drive tests with security requirements and security features
ST2.1 use encapsulated attacker perspective integrate black box security tools into the QA process 2
ST2.4 facilitate security mindset share security results with QA
ST3.1 include security testing in regression include security tests in QA automation 3
ST3.2 teach tools about your code perform fuzz testing customized to application APIs
ST3.3 probe risk claims directly drive tests with risk analysis results
ST3.4 drive testing depth leverage coverage analysis
ST3.5 move beyond functional testing to attacker's perspective begin to build and apply adversarial security tests (abuse cases)

ST Level 1: Enhance QA beyond functional perspective. QA must progress to include functional edge and boundary condition testing in its test suites. Test suites must include functional security testing.


Ensure QA supports edge/boundary value condition testing. The QA team goes beyond functional testing to perform basic adversarial tests. They probe simple edge cases and boundary conditions. No attacker skills required. When QA understands the value of pushing past standard functional testing using acceptable input, they begin to move slowly toward "thinking like a bad guy." A discussion of boundary value testing leads naturally to the notion of an attacker probing the edges on purpose. What happens when you enter the wrong password over and over?


Drive tests with security requirements and security features. Testers target declarative security mechanisms derived from requirements and security features. For example, a tester could try to access administrative functionality as an unprivileged user or verify that a user account becomes locked after some number of failed authentication attempts. For the most part, security features can be tested in a similar fashion to other software features. Security mechanisms based on requirements such as account lockout, transaction limitations, entitlements, and so on are also tested. Of course, software security is not security software, but getting started with features is easy.


ST Level 2: Integrate the attacker perspective into test plans. QA must integrate black-box security testing tools into its process. QA must build test suites for functional security features. The SSG must share its security knowledge and testing results with QA.


Integrate black box security tools into the QA process. The organization uses one or more black box security testing tools as part of the quality assurance process. The tools are valuable because they encapsulate an attacker's perspective, albeit in a generic fashion. Tools such as IBM Security AppScan or HP WebInspect are relevant for Web applications, and fuzzing frameworks such as Codenomicon are applicable for most network protocols. In some situations, other groups might collaborate with the SSG to apply the tools. For example, a testing team could run the tool, but come to the SSG for help interpreting the results. Regardless of who runs the black box tool, the testing should be properly integrated into the QA cycle of the SSDL.


Share security results with QA. The SSG routinely shares results from security reviews with the QA department. Over time, QA engineers learn the security mindset. Using security results to inform and evolve particular testing patterns can be a powerful mechanism leading to better security testing. This activity benefits from an engineering-focused QA function that is highly technical.


ST Level 3: Deliver risk-based security testing. QA must include security testing in automated regression suites. The SSG must ensure this security testing and its depth is guided by knowledge about the codebase and its associated risks as well as adversarial testing that simulates the attacker's perspective.


Include security tests in QA automation. Security tests run alongside functional tests as part of automated regression testing. The same automation framework houses both. Security testing is part of the routine. Security tests can be driven from abuse cases identified earlier in the lifecycle or tests derived from creative tweaks of functional tests.


Perform fuzz testing customized to application APIs. Test automation engineers customize a fuzzing framework to the organization's APIs. They could begin from scratch or use an existing fuzzing toolkit, but customization goes beyond creating custom protocol descriptions or file format templates. The fuzzing framework has a built-in understanding of the application interfaces it calls into. Test harnesses developed explicitly for particular applications can make good places to integrate fuzz testing.


Drive tests with risk analysis results. Testers use architecture analysis results to direct their work. For example, if the architecture analysis concludes "the security of the system hinges on the transactions being atomic and not being interrupted partway through," then torn transactions will be become a primary target in adversarial testing. Adversarial tests like these can be developed according to risk profile—high-risk flaws first.


Leverage coverage analysis. Testers measure the code coverage of their security tests in order to identify code that isn't being exercised. Code coverage drives increased security testing depth. Standard-issue black box testing tools achieve exceptionally low coverage, leaving a majority of the software under test unexplored. Don't let this happen to your tests. Using standard measurements for coverage such as function coverage, line coverage or multiple condition coverage is fine.


Begin to build and apply adversarial security tests (abuse cases). Testing begins to incorporate test cases based on abuse cases provided by the SSG. Testers move beyond verifying functionality and take on the attacker's perspective. For example, testers might systematically attempt to replicate incidents from the organization's history. Abuse and misuse cases based on the attacker's perspective can also be driven from security policies, attack intelligence, and guidelines. This turns the corner from testing features to attempting to break the software under test.