ACATS 3.1 User's Guide
5.6 Grading Test Results
Although a single test may examine multiple language
issues, ACATS test results are graded "passed", "failed",
or "not applicable" as a whole.
All customized, applicable tests must be processed
by an implementation. Results must be evaluated against the expected
results for each class of test. Results that do not conform to expectations
constitute failures. The only exceptions allowed are discussed in 5.2.5
in such cases, processing the approved modified test(s) must produce
the expected behavior. Any differences from the general discussion of
expected results below for executable or non-executable tests are included
as explicit test conditions in test prologues.
Warning or other informational messages do not affect
the pass/fail status of tests.
Expected results for executable and non-executable
tests are discussed in Sections 5.6.1
. Tests that are non-applicable for
an implementation are discussed in 5.6.4
tests are discussed in 5.6.5