ACATS 4.1 User's Guide
3.2.1 Coverage Documents
ACVC 2.0 originated coverage documentation for what
is now the ACATS. That coverage document mapped tests to paragraph numbers
in the
[Ada95] Standard. Over time, a number
of deficiencies in that approach have been identified:
Where a paragraph contains multiple rules, it is
not possible to tell if all of the rules are tested. Indeed, a number
of instances where important rules were untested have been identified.
Since there is not an indication of the type of
rule represented by a paragraph, it is not obvious whether the rule is
appropriately tested. For instance, a Legality Rule tested only with
C-Tests is not appropriately tested (see
3.2.3
for more on this topic).
Rules not modified in
[Ada95]
were not considered. However, sometimes additional test objectives are
needed for existing rules because of other changes to the language. Moreover,
there is no determination of whether the legacy tests adequately tested
the original rule.
The plaintext coverage document is not easily processed
by tools. In particular, paragraphs that are covered are indicated only
by the test name(s) that cover the paragraph; there is no common string
for a tool to count. As such, it is difficult to get metrics about coverage
from the document, and in particular to quantify progress between versions.
For these reasons, the development of new and much
more detailed coverage documentation was initiated for ACATS 3.0. The
new documentation considers each sentence of the standard individually
if necessary, listing the test objectives for each sentence, and documenting
why no objectives are needed if none are provided.
It also lists tests for each objective (including
Legacy tests), testing priorities for untested objectives, notes on each
objective (including any untested cases), and any additional notes needed.
There is also a summary document that summarizes
totals for important metrics for the coverage documents as a whole.
The documents are maintained as spreadsheets (in
order to make calculation and extraction of metrics easy). Because spreadsheet
programs may be unfamilar to many readers, and compatibility of spreadsheet
programs is incomplete, these documents are provided in Adobe Portable
Document Format (PDF). (The spreadsheets are available on request from
the ACAA Technical Agent,
agent@ada-auth.org.)
For ACATS 4.1, the
following subclauses of the Ada Standard documents are included in the
new coverage documents:
3.2.4
3.9.3 through 3.10.1
4.1.3 through 4.4
4.5.7 through 4.5.8
5.4 through 5.5.2
6.1.1
6.5 through 6.8
7.3.2 through 7.6.1
8.3.1 through 8.5.3
Clause 10 (10 through 10.2.1)
13.1.1
B.3.3
These clauses were selected because of their importance
in rule changes in either
[Ada2012] or
[Amend1].
The original coverage document is also included with
the ACATS for those portions of the Ada Standard documents that have
not yet had new coverage documents constructed. This document is of primary
use for features of
[Ada95], as it does
not cover Legacy tests nor any features added or modified by either
[Ada2012]
or
[Amend1]. As noted above, it is being
replaced, so it will disappear completely in a future version of the
ACATS.