ACATS 3.0 User's Guide
1.2 ACATS Coverage of Ada
The ACATS needs to test as many rules as possible
in order to meet the goal of enhancing Ada software portability. After
all, a rule that is not tested is far more likely to be incorrectly implemented
than one that is tested.
Therefore the ACATS strives for complete coverage
of the standard. Complete coverage
every rule in the Ada standard has one or more associated tests that
ensure that the rule is implemented properly.
Complete coverage is especially important for legality
rules and runtime checks. It is easy for implementers to miss these rules,
as their compiler may do something useful in the absence of the checks.
But allowing such incorrect code can be a major portability problem when
a program is moved to a compiler (including a later version of the same
compiler) that properly implements the checks.
Of course, complete coverage does not mean that every
sentence in the Ada standard has an associated test. There are many lines
in the standard that are not rules at all, such as notes and examples.
There are also many lines in the standard that are not testable; documentation
requirements are but one example. Finally, there are rules in the standard
that could be tested, but such tests are unlikely to find errors; for
instance, most tests for illegal syntax are in this category as the test
would need to guess what error might be made.
Complete coverage is a goal for the ACATS; it is
not expected to be achieved in near future. Thus the test coverage analysis
of the ACATS is a set of living documents, which will be updated with
new information and tests with each new ACATS version.
Coverage testing will generally not test combinations
of features, so problems that only manifest themselves in such combinations
will not be detected. Tests designed primarily to cover language rules
are most useful to prevent gross errors in implementations (such as forgetting
to implement checks or features). As such, the ACATS also supplements
those tests with tests written to emulate the patterns of use of Ada
features. Such tests provide tests of common combinations of features,
and ensure that common idioms are implemented properly.
A detailed description of how coverage is determined
for the Ada standard can be found in Section 3.2