ACATS 4.1 User's Guide
Title Page
Contents   Index   References   Previous   Next 

1.2 ACATS Coverage of Ada

The ACATS needs to test as many Ada language rules as possible in order to meet the goal of enhancing Ada software portability. After all, a rule that is not tested is far more likely to be incorrectly implemented than one that is tested.
Therefore the ACATS strives for complete coverage of the standard. Complete coverage means that every rule in the Ada standard has one or more associated tests that ensure that the rule is implemented properly.
Complete coverage is especially important for legality rules and runtime checks. It is easy for implementers to miss these rules, as their compiler may do something useful in the absence of the checks. But allowing such incorrect code can be a major portability problem when a program is moved to a compiler (including a later version of the same compiler) that properly implements the rules and checks.
Of course, complete coverage does not mean that every sentence in the Ada standard has an associated test. There are many lines in the standard that are not rules at all, such as notes and examples. There are also many lines in the standard that are not testable; documentation requirements are but one example. There are also rules in the standard which could be tested, but such tests would be outside of the purpose of the ACATS. Rules which express options for Ada implementations (such as permissions or advice) are in this category (as the ACATS does not test implementation quality).
It should be obvious that the importance of testing particular rules in the Ada standard vary widely. In part, that's because the various Ada features themselves have widely varying importance. (For example, looking at three features that ACATS 4.1 could have tested, proper operation of if expressions is more important than correct functioning of record equality, which in turn is more important than run-time accessibility checks for coextensions.) But it's also because of the differing nature of the rules. For instance, testing of syntax rules is not very important in most cases, as correct syntax will be checked by tests for other rules, and tests for incorrect syntax require guessing what error a compiler might make in interpreting syntax. Unless an error seems particularly likely (as might happen from some irregularity in the syntax), such guessing is unlikely to find actual errors. This makes it impossible for there to be a definitive set of rules that need to be tested to accomplish complete coverage, as the decision as to whether a particular rule is usefully testable can be a judgement call.
In any case, complete coverage is a goal for the ACATS. However, it is not expected to be ever be achieved. As the ACATS gets nearer to the goal, the value of additional tests will drop (since higher value tests get created earlier), and at some point, the cost of creating and processing new tests would outweigh their value.
Since exactly which rules need to be tested to accomplish complete coverage will always be a judgement call, the test coverage analysis of the ACATS is a set of living documents, which will be updated with new information and tests with each new ACATS version.
Coverage testing will generally not test combinations of features, so problems that only manifest themselves in such combinations will not be detected. Tests designed primarily to cover language rules are most useful to prevent gross errors in implementations (such as forgetting to implement checks or features). As such, the ACATS also supplements those tests with tests written to emulate the patterns of use of Ada features. Such tests provide tests of common combinations of features, and ensure that common idioms are implemented properly.
A detailed description of how coverage is determined for the Ada standard can be found in Section 3.2.

Contents   Index   References   Previous   Next