ACATS 4.1 User's Guide
4.2.2 Class B
Class B tests check that illegal constructs are recognized
and treated as fatal errors. They are not expected to successfully compile,
bind, or execute. Lines that contain errors are marked -- ERROR:
and generally include a brief description of the illegality on the same
or following line. (The flag includes a final ":" so that search
programs can easily distinguish it from other occurrences of the word
"error" in the test code or documentation.) Some tests also
mark some lines as -- OK, indicating that the line must not
be flagged as an error.
An implementation passes a class B test if each indicated
error in the test is detected and reported, and no other errors are reported.
The test fails if one or more of the indicated errors are not reported,
or if an error is reported that cannot be associated with one of the
indicated errors. If the test structure is such that a compiler cannot
recover sufficiently to identify all errors, it may be permissible to
"split" the test program into separate units for re-processing
(see Section 5.2.5
for instructions on modifying
In some cases and for some constructs, compilers
may adopt various error handling and reporting strategies. In cases where
the test designers determined that an error might or might not be reported,
but that an error report would be appropriate, the line is marked with
-- OPTIONAL ERROR: or a similar phrase. In such cases, an implementation
is allowed to report an error or fail to report an error without affecting
the final grade of the test.
Similarly, in cases where the test designers determined
that an error might be reported at one of several source locations, all
such source locations are marked with -- POSSIBLE ERROR: and
an indication of which error (if the test contains several) is expected.
In such cases, an implementation is considered passing if it reports
an error at any of the possible places for the error to be reported.
It fails if no error is reported at any of the places.
All of these test markings can be followed by an
optional range indicator (see 6.3.2
describe the expected locations of an error message for the error, and
are primarily used by the automated Grading Tool (see 6