Pass and Fail Criteria Before tests were performed, the following criteria were defined to ensure defect prevention and bug resolution: All test cases must pass with expected results as described in the individual test case spreadsheets. A test case is considered to have passed if the actual result matched the expected result that is documented for the case. If the actual result does not match the expected result, it was treated as a failed test case, a bug was created, and a severity score was assigned. If a test case failed, it was not assumed that the solution guidance was necessarily defective. For example, misinterpretation of product documentation, incomplete documentation, or inaccurate documentation could cause failures. Each failure was analyzed to discover its cause based on actual results and the results that were described in project documentation. Failures were also escalated to the appropriate owners of the respective Microsoft products. Release Criteria The primary release criterion for the Windows Server 2003 Security Guide was related to the severity of bugs that were still open. However, other issues that were not being tracked through bugs were also discussed. The criteria for release are: No bugs are open with severity levels 1 and 2. All open bugs are triaged by the leadership team, and their impacts are fully understood. Solution guides are free of comments and revision marks. The solution successfully passes all test cases in the test lab environment. Solution contents have no conflicting statements. Bug Classification The bug severity scale is described in the following table. The scale is from 1 to 4, with 1 as the highest severity and 4 as the lowest severity. Table D.1 Bug Severity Classification Severity Most common types Conditions required 1 Bug blocked build or further Solution did not work. – – testing. User could not begin to use – Bug caused unexpected user significant parts of the computer or – accessibility. network. Steps defined in the User had access privileges that – – documentation were not clear. should not be allowed. Results or behavior of a function or User access was blocked to certain – – process contradicts expected results server(s) that should be allowed. (as documented in functional Expected results were not – specification). achieved. Major mismatch between the Testing cannot proceed without – – security template files and the being addressed. functional specification.
Pass and Fail Criteria Before tests were performed, the following criteria were defined to ensure defect prevention and bug resolution: All test cases must pass with expected results as described in the individual test case spreadsheets. A test case is considered to have passed if the actual result matched the expected result that is documented for the case. If the actual result does not match the expected result, it was treated as a failed test case, a bug was created, and a severity score was assigned. If a test case failed, it was not assumed that the solution guidance was necessarily defective. For example, misinterpretation of product documentation, incomplete documentation, or inaccurate documentation could cause failures. Each failure was analyzed to discover its cause based on actual results and the results that were described in project documentation. Failures were also escalated to the appropriate owners of the respective Microsoft products. Release Criteria The primary release criterion for the Windows Server 2003 Security Guide was related to the severity of bugs that were still open. However, other issues that were not being tracked through bugs were also discussed. The criteria for release are: No bugs are open with severity levels 1 and 2. All open bugs are triaged by the leadership team, and their impacts are fully understood. Solution guides are free of comments and revision marks. The solution successfully passes all test cases in the test lab environment. Solution contents have no conflicting statements. Bug Classification The bug severity scale is described in the following table. The scale is from 1 to 4, with 1 as the highest severity and 4 as the lowest severity. Table D.1 Bug Severity Classification Severity Most common types Conditions required 1 – Bug blocked build or further testing. – Bug caused unexpected user accessibility. – Steps defined in the documentation were not clear. – Results or behavior of a function or process contradicts expected results (as documented in functional specification). – Major mismatch between the security template files and the functional specification. – Solution did not work. – User could not begin to use significant parts of the computer or network. – User had access privileges that should not be allowed. – User access was blocked to certain server(s) that should be allowed. – Expected results were not achieved. – Testing cannot proceed without being addressed.
WindowsServer2003SecurityGuide.docx