ACATS 4.1 User's Guide
Title Page
Contents   Index   References   Previous   Next 

5.6 Grading Test Results

Although a single test may examine multiple language issues, ACATS test results are graded "passed", "failed", or "not applicable" as a whole.
All customized, applicable tests must be processed by an implementation. Results must be evaluated against the expected results for each class of test. Results that do not conform to expectations constitute failures. The only exceptions allowed are discussed in 5.2.5; in such cases, processing the approved modified test(s) must produce the expected behavior. Any differences from the general discussion of expected results below for executable or non-executable tests are included as explicit test conditions in test prologues.
Warning or other informational messages do not affect the pass/fail status of tests.
Expected results for executable and non-executable tests are discussed in Sections 5.6.1, 5.6.2, and 5.6.3. Tests that are non-applicable for an implementation are discussed in 5.6.4. Withdrawn tests are discussed in 5.6.5.
The ACATS provides a tool that can be used to automate much of the test grading process; see 6. This tool enforces the expected results as described in the following sections, as well as checking for processing errors (compiling test files in the wrong order, for instance) and omissions (failing to compile required test files).

Contents   Index   References   Previous   Next