ACATS 4.0 User's Guide
1.2 ACATS Coverage of Ada
The ACATS needs to test as many Ada language rules
as possible in order to meet the goal of enhancing Ada software portability.
After all, a rule that is not tested is far more likely to be incorrectly
implemented than one that is tested.
Therefore the ACATS strives for complete coverage
of the standard. Complete coverage
every rule in the Ada standard has one or more associated tests that
ensure that the rule is implemented properly.
Complete coverage is especially important for legality
rules and runtime checks. It is easy for implementers to miss these rules,
as their compiler may do something useful in the absence of the checks.
But allowing such incorrect code can be a major portability problem when
a program is moved to a compiler (including a later version of the same
compiler) that properly implements the rules and checks.
Of course, complete coverage does not mean that every
sentence in the Ada standard has an associated test. There are many lines
in the standard that are not rules at all, such as notes and examples.
There are also many lines in the standard that are not testable; documentation
requirements are but one example. There are also rules in the standard
which could be tested, but such tests would be outside of the purpose
of the ACATS. Rules which express options for Ada implementations (such
as permissions or advice) are in this category (as the ACATS does not
test implementation quality).
It should be obvious that the importance of testing
particular rules in the Ada standard vary widely. In part, that's because
the various Ada features themselves have widely varying importance. (For
example, looking at three features that ACATS 4.0 could have tested,
proper operation of if expressions is more important than correct functioning
of record equality, which in turn is more important than run-time accessibility
checks for coextensions.) But it's also because of the differing nature
of the rules. For instance, testing of syntax rules is not very important
in most cases, as correct syntax will be checked by tests for other rules,
and tests for incorrect syntax require guessing what error a compiler
might make in interpreting syntax. Unless an error seems particularly
likely (as might happen from some irregularity in the syntax), such guessing
is unlikely to find actual errors. This makes it impossible for there
to be a definitive set of rules that need to be tested to accomplish
complete coverage, as the decision as to whether a particular rule is
usefully testable can be a judgement call.
In any case, complete coverage is a goal for the
ACATS. However, it is not expected to be ever be achieved. As the ACATS
gets nearer to the goal, the value of additional tests will drop (since
higher value tests get created earlier), and at some point, the cost
of creating and processing new tests would outweigh their value.
Since exactly which rules need to be tested to accomplish
complete coverage will always be a judgement call, the test coverage
analysis of the ACATS is a set of living documents, which will be updated
with new information and tests with each new ACATS version.
Coverage testing will generally not test combinations
of features, so problems that only manifest themselves in such combinations
will not be detected. Tests designed primarily to cover language rules
are most useful to prevent gross errors in implementations (such as forgetting
to implement checks or features). As such, the ACATS also supplements
those tests with tests written to emulate the patterns of use of Ada
features. Such tests provide tests of common combinations of features,
and ensure that common idioms are implemented properly.
A detailed description of how coverage is determined
for the Ada standard can be found in Section 3.2