ACATS 4.1 User's Guide
Title Page
Contents   Index   References   Previous   Next 

3.2.1 Coverage Documents

ACVC 2.0 originated coverage documentation for what is now the ACATS. That coverage document mapped tests to paragraph numbers in the [Ada95] Standard. Over time, a number of deficiencies in that approach have been identified: 
Where a paragraph contains multiple rules, it is not possible to tell if all of the rules are tested. Indeed, a number of instances where important rules were untested have been identified.
Since there is not an indication of the type of rule represented by a paragraph, it is not obvious whether the rule is appropriately tested. For instance, a Legality Rule tested only with C-Tests is not appropriately tested (see 3.2.3 for more on this topic).
Rules not modified in [Ada95] were not considered. However, sometimes additional test objectives are needed for existing rules because of other changes to the language. Moreover, there is no determination of whether the legacy tests adequately tested the original rule.
The plaintext coverage document is not easily processed by tools. In particular, paragraphs that are covered are indicated only by the test name(s) that cover the paragraph; there is no common string for a tool to count. As such, it is difficult to get metrics about coverage from the document, and in particular to quantify progress between versions.
For these reasons, the development of new and much more detailed coverage documentation was initiated for ACATS 3.0. The new documentation considers each sentence of the standard individually if necessary, listing the test objectives for each sentence, and documenting why no objectives are needed if none are provided.
It also lists tests for each objective (including Legacy tests), testing priorities for untested objectives, notes on each objective (including any untested cases), and any additional notes needed.
There is also a summary document that summarizes totals for important metrics for the coverage documents as a whole.
The documents are maintained as spreadsheets (in order to make calculation and extraction of metrics easy). Because spreadsheet programs may be unfamilar to many readers, and compatibility of spreadsheet programs is incomplete, these documents are provided in Adobe Portable Document Format (PDF). (The spreadsheets are available on request from the ACAA Technical Agent, agent@ada-auth.org.)
For ACATS 4.1, the following subclauses of the Ada Standard documents are included in the new coverage documents: 
3.2.4
3.9.3 through 3.10.1
4.1.3 through 4.4
4.5.7 through 4.5.8
5.4 through 5.5.2
6.1.1
6.5 through 6.8
7.3.2 through 7.6.1
8.3.1 through 8.5.3
Clause 10 (10 through 10.2.1)
13.1.1
B.3.3
These clauses were selected because of their importance in rule changes in either [Ada2012] or [Amend1].
The original coverage document is also included with the ACATS for those portions of the Ada Standard documents that have not yet had new coverage documents constructed. This document is of primary use for features of [Ada95], as it does not cover Legacy tests nor any features added or modified by either [Ada2012] or [Amend1]. As noted above, it is being replaced, so it will disappear completely in a future version of the ACATS.

Contents   Index   References   Previous   Next