ACATS 4.1 User's Guide
Title Page
Contents   Index   References   Previous   Next 

3.2.3 Coverage Guidelines for Specific Rule Categories

The Ada standard is divided into various categories of rules. These rule categories have different requirements and thus differ in what is the appropriate testing philosophy. This translates into different types of test objectives for each rule category.
Uncategorized text:

This is generally introductory text, and does not require any testing. This text sometimes includes definitions; these are handled as described in 3.2.2, “General Coverage Guidelines”.
Testing that legal syntax is implemented is accomplished by all of the other kinds of tests, especially by the usage-oriented tests. Generally, tests for illegal syntax have a very low value. Most compilers use a syntax that is similar to that of the standard, and many use parsers generated by tools, and thus the likelihood of errors is low. Moreover, the number of possible bugs is essentially unlimited. There are three cases where testing is warranted: First, rules that are given in text form rather than BNF are treated as legality rules (see below). Second, the Ada syntax grammar is not directly usable by either of the common technologies for parsers (LL/recursive descent, and LR/LALR). In cases where the grammar used by a compiler must necessarily be more general than that given in the standard, the additional requirements are treated as legality rules. Finally, an irregularity in the syntax rules can suggest an obvious (but incorrecct) syntax extension. It can warrant a test that such an extension is not implemented. For instance, most places in Ada that declare an object allow the use of subtype_indications, but subprogram profiles only allow subtype_marks. It should be tested that subtype_indications are not allowed when declaring parameters, even though this is a syntax rule.
Name Resolution Rules:

Name resolution rules should usually have a B-Test that checks that too much information is not used for resolution. (For instance, whether a type is limited or non-limited should generally not be used. Another example is whether an access type is pool-specific or general access usually should not be used.) It also may have a C-Test to check that legal cases are allowed; but this objective is often handled by tests for other test objectives (if that's the case, a note should point out where they are tested). One common case where a C-Test is needed occurs when a name resolution rule uses some property for resolution. In this case, a C-Test that the property can be used to resolve an overloaded function call is needed. (For instance, an if statement condition must be of a Boolean type; it should be tested that a function returning some other type is ignored when resolving a condition.)
Legality Rules:

Legality rules should almost always have an associated B-Test that checks that illegal cases are detected. They may also have a C-Test to check that legal cases are allowed; again, this is often handled by tests for other test objectives. (They also can be covered in the B-Test with "OK" cases.)
Static Semantics:

Many Static Semantic rules are definitions, and as such can be handled as described in 3.2.2, “General Coverage Guidelines”.
Many other static semantic rules are miscategorized by the standard. For instance, most of the library packages are defined as static semantics, even though most of the rules are really dynamic. These should be tested as appropriate for the correct category.
Remaining rules should be tested as legality rules (arguably, these too are miscategorized).
Post-Compilation Rules:

A post-compilation rule should always have L-Tests to check that the check is made in illegal cases. These checks are especially likely to be omitted (because they're often complex to implement), so these tests should have a high priority. In some cases, a C-Test may be needed to ensure that legal cases work and the check does not go too far. Often that will be covered by other tests, so a separate test is not needed (a note should be provided to show where it is covered).
Dynamic Semantics:

Dynamic semantics rules can be divided into two sub-categories.
First, these rules can define run-time checks. These checks should have C-Tests that verify that the check is made and the appropriate exception is raised. Tests like this have to be carefully constructed to avoid running afoul of 11.6 permissions.
Otherwise, these rules can define the normal operation of a construct. These should have usage-oriented C-Tests. By making these tests usage-oriented, the ACATS tests the normal usage of these features, not unusual corner-cases.
One special case: rules that say the execution of something "has no effect" can't be usefully tested. Testing that nothing happens is not interesting. And testing for the absence of an effect requires guessing incorrect effects that might have occurred (which is unlikely to actually detect anything), and thus any such test would have a very low value.
Bounded (Run-Time) Errors:

These usually require C-Tests, but not always. If the bound includes "work normally", then the bounded error is not usefully testable, since the ACATS does not test implementation choices. Whether something works or raises Program_Error is not interesting. In other cases, however, since there is a bound on the behavior, it can be useful to ensure that one of the prescribed results happens rather than havoc.
Erroneous Execution:

This is not testable. Since the language allows anything to happen, there is no useful information to be gained from including this in a test. Indeed, it's important to avoid any such case in an executable test.
Implementation Requirements:

These usually are dynamic requirements that should be tested like the appropriate dynamic semantics rules. Otherwise, they should be tested like a legality rule.
Implementation Permissions:

Usually permissions are not testable. Since the ACATS is not testing the quality of implementations, the choices made by an implementation are not appropriate things to test. So a test should be created only if there is value in testing for implementations exceeding the permission.
Implementation Advice:

Advice is usually not testable. Again, since the ACATS is not testing quality of implementations, the choices made by an implementation are not appropriate things to test. Moreover, advice need not be followed, and other implementation choices can be made. Usually, if there are bounds to what is acceptable, they are covered by other rules, and thus the tests would be there.
Documentation Requirements and Metrics:

No tests are required for documentation requirements. Documentation existence could be part of the conformity assessment process, but in any case it is outside of the scope of the ACATS (which is testing the implementation, not the documentation).
Notes and Examples:

These are non-normative text, and do not require any testing.

Contents   Index   References   Previous   Next