Operating Procedures
for Ada Conformity Assessments
Version 3.1
December 10, 2013
Ada Resource Association, Inc.
P.O. Box 8685
New York, NY 10116
TABLE OF CONTENTS
APPENDICES
The International Organization for Standardization (ISO)
and the International Electrotechnical Commission (IEC), through Working
Group 9 (WG9) of Subcommittee 22 (SC22) of their Joint Technical
Committee 1 (JTC1), have established an International Standard titled
Ada: Conformity Assessment of a Language Processor (ISO/IEC
18009:1999), referenced in this document as [ISO 99]. This standard
specifies requirements for five aspects of a conformity assessment
program, as follows:
- Ada
Conformity Assessment Laboratories (ACAL), independent test
laboratories performing Ada Conformity Assessment in accordance with
[ISO 99] and the Ada Conformity Assessment Procedure (ACAP);
- Ada
Conformity Assessment Authority (ACAA), an organization that ensures
worldwide commonality of the Ada Conformity Assessment Process;
- Ada
Conformity Assessment Test Suite (ACATS), managed and administered
by the ACAA and used by the ACAL in performing conformity
assessments;
- Ada
Conformity Assessment Procedure (ACAP), governing the assessment
activities of the ACAL and the contents of the Ada Conformity
Assessment Test Report (ACATR); and
- Ada
Conformity Assessment Certificate (ACAC), issued by the ACAL and
ACAA in recognition that an Ada language processor has successfully
completed conformity assessment.
The
current document represents the Ada Resource Association (ARA)’s
establishment of an ACAA governed by the procedures (ACAP) given
herein. The primary goal of this document is to conform to the
International Standard [ISO 99] (providing an ACAP as defined in that
document).
[ISO 99]
requires a periodic review and update of the ACAP. This document
represents the results of the first such review and update. The major
change is to allow vendors to use newer versions of the test suite
for certifications by extension and derivation, and accordingly to
extend the expiration dates of these certifications.
The
following paragraphs summarize the conformity assessment system by
using the language of the International Standard [ISO 99].
Ada Conformity Assessment Test Suite
The
ACATS was originally derived from the Ada Compiler Validation
Capability (ACVC). The ACVC was a conformity test suite (with
supporting documents) developed under contract to the United States
Government and made available for public use; it was designed to
ensure that Ada processors achieve a high degree of conformity to the
Ada language standard. A processor is tested when operating in a
specific hardware and software configuration. The ACATS is customized
by a testing laboratory (ACAL) for each processor that is subjected
to conformity assessment; customization consists of adjusting the
ACATS appropriately for various implementation characteristics. The
ACATS is maintained by the ACAA; new releases of the ACATS are
defined by changes resulting from the discovery of deficiencies in
the test programs and by changes in the Ada standard (for instance,
the adoption of Technical Corrigendum 1 [TC1], Amendment 1 [Amd1],
and Ada 2012 [Ada2012]).
For an
Ada processor to successfully complete conformity assessment, it must
process each test program (for the “core” language) of a
customized ACATS so that the result is graded Passed, Inapplicable,
or Unsupported by ACATS grading criteria. A testing laboratory
customizes the test suite for a particular processor by appropriately
setting test parameters, by removing “withdrawn” tests
(tests ruled by the ACAA to be in error) and certain inapplicable
tests, by splitting as needed test files with multiple intended
errors so to enable complete error detection, by using any other
modified tests as directed by the ACAA, and by including each
optional set of tests (see below) as requested by the client.
In
addition to the specification of a “core” language,
[Ada2012] contains several Specialized Needs Annexes (SNAs); these
specify language requirements designed to meet the particular needs
of various general application domains, such as information-systems
programming. A processor for [Ada2012] need not include
implementation of any of these annexes, or it may implement only some
of the features of these annexes. Whereas all ACATS test programs for
the core language must be processed during conformity assessment,
those for the SNAs are processed only upon client request. A
conformity assessment is judged successful (leading to the issuance
of an ACAC) only if all tests for the core are correctly processed;
the certificate will additionally give credit for support of a SNA to
the extent that the relevant set of tests is correctly processed.
Conformity Assessment
Conformity
assessment involves interaction between the ACAL client and both the
ACAL and the ACAA. The assessment process consists of well-defined
actions which, when completed successfully, result in the award of an
ACAC for the tested processor. The key actions in the conformity
assessment of an Ada processor are:
- The
client and an ACAL reach a formal agreement for conformity
assessment, including the dates for the submittal of the results of
client-administered processing of the ACATS and for ACAL witness
testing.
- The
client petitions for deviation from the requirements of each ACATS
test program that is believed to be wrong for the candidate
implementation(s).
- The
ACAA rules on the client petitions.
- The
client processes the ACATS on the candidate processor(s) and submits
the results to the ACAL.
- The
ACAL analyzes the results of the client’s independent
processing of the ACATS. (If the results are not acceptable, the
previous action must be repeated, and new results analyzed.)
- The
ACAL conducts witness testing of the candidate processor(s),
documents this testing in an Ada Conformity Assessment Test Report
(ACATR), and submits the ACATR to the ACAA for review.
- The
client signs a Declaration of Conformity for each candidate
processor.
- The
ACAA reviews the ACATR, with comments to the ACAL. The ACAA
recommends to the ACAL that a conformity assessment certificate be
issued for the tested processor(s), if the testing is successful.
- The
ACAL issues a conformity assessment certificate for each tested
processor upon the successful completion of the preceding actions.
Hence,
successful testing of an Ada processor concludes with the ACAL's
awarding a conformity assessment certificate for that processor
(working in a specific configuration) to the client. This conformity
assessment certificate attests that the processor has been subjected
to an Ada conformity assessment and that no evidence of
non-conformity was found. The processor is said to be "certified
as conforming," as described in [ISO 99], clause 2.1. The
processor will be listed in the ARA's Certified Processors List
(CPL). The client may perform maintenance on the processor and may
claim conformity for such derived versions in accordance with the
ACAA procedures, so long as the client ensures that they produce the
same ACATS results as are documented in the ACATR. This maintenance
may even include adaptive maintenance that enables the processor to
run on entirely different host computers (i.e., re-hosting) or to
target closely related target computers. The ACAA provides a means
for listing derived processors in the CPL.
This
document provides operating procedures of the Ada Conformity
Assessment Authority (ACAA). This body is a part of an organization
that meets the requirements for assessing conformity of an Ada
language processor, as given in [ISO 99]. The other bodies making up
this organization are the Ada Conformity Assessment Laboratories
(ACAL), which perform the actual conformity assessments using the Ada
Conformity Assessment Test Suite (ACATS). The end product of a
successful conformity assessment is an Ada Conformity Assessment
Certificate (ACAC), indicating that a particular Ada language
processor is "certified as conforming," as defined in [ISO
99].
This
document forms an Ada Conformity Assessment Procedure (ACAP), as
defined in [ISO 99].
Detailed
procedures regarding the application of the ACATS are given in the
ACATS User's Guide [ACATS UG].
The
United States Department of Defense (DoD) sponsored the development
of the Ada programming language and established the Ada Joint Program
Office (AJPO) as part of an effort to support recognized principles
of software engineering for a wide range of applications. The AJPO
established a certification system to realize the benefits of
standardization, which include the ability to transfer software and
programming expertise between computer systems that use a conforming
Ada processor. When the AJPO ceased its operation of the
certification system, the Ada Validation Facilities agreed to act as
ACALs under the provisions of the emerging International Standard
(now standardized as [ISO 99]). The Ada Resource Association, in
cooperation with the Ada Joint Program Office, facilitated the
identification and U.S. Government funding for a candidate ACAA and
produced this document defining its operating procedures. The ACALs
then agreed to designate the identified organization as the ACAA and
to use these procedures as the ACAP.
It is
important to note the scope and intent of conformity assessment. The
purpose of conformity assessment is to ensure that Ada processors
achieve a high degree of conformity with the Ada standard
([Ada2012]). Characteristics such as performance and suitability for
a particular application are not specified by the standard, and thus
are outside the scope of Ada conformity assessment. Moreover, the
ACATS is a set of test programs intended to check broadly for correct
implementation; it is not possible to exhaustively test for
conformity. Thus, conformity is checked only to the extent of these
tests; processors that are certified as conforming may fail to
conform to the standard in ways peculiar to each, under particular
circumstances.
Witness
testing does not warrant that the product tested is free of
nonconformities, even if all tests are passed. The practical goal of
Ada conformity assessment is to identify processors that may be
procured and used to develop application programs that meet the
[Ada2012] goals of portability and interoperability.
The
ACATS (test suite) is not designed to replace the client’s
quality assurance testing or systematically to detect inconsistencies
or “bugs”, but to verify that the tested processor
correctly supports all required features. Rather than exhaustive
testing of permutations of features, the test suite contains a
carefully chosen set of test cases that cover the required syntax and
demonstrate the correct implementation of each of the applicable
general rules from the standard.
Neither
is conformity assessment intended as a means of performance
benchmarking. The Ada Conformity Assessment Test Report (ACATR) which
documents the witness testing does not contain information about the
speed, cost, or efficiency of executing the conformity assessment
tests.
Ada:
The programming language defined by [Ada2012].
Ada
Conformity Assessment: The process of checking the conformity of a
processor to the Ada programming language standard [Ada2012] and of
issuing a conformity assessment certificate for the implementation.
Ada
Conformity Assessment Authority (ACAA): The organization that
provides the technical and administrative oversight of Ada
Conformance assessment activities.
Ada
Conformity Assessment Certificate (ACAC): A certificate issued by
authority of the ACAA and an ACAL for a successfully tested Ada
processor (see Section 5.4).
Ada
Conformity Assessment Laboratory (ACAL): An independent testing
laboratory performing Ada conformity assessments.
Ada
Conformity Assessment Test Report (ACATR): A report produced by an
ACAL that documents the witness testing of an Ada processor (see
Section 5.4).
Ada
Conformity Assessment Test Suite (ACATS): The means for testing
conformity of Ada processors, consisting of the test suite, the
support programs, and the User's Guide [ACATS UG]. The ACATS was
based on the Ada Compiler Validation Capability (ACVC).
Ada
Conformity Assessment Test Suite Modification List (ACATS
Modification List): A listing of modifications to the ACATS
correcting flaws found in the suite. This list is periodically issued
by the ACAA (see Section 4.6.1).
Ada
Conformity Assessment Test Suite Version Control System (ACATS VCS):
A publicly accessible program that stores the current version of the
ACATS, as well as any modified tests used for conformity assessments
(see Section 4.5.1).
Ada
processor: A processor for the Ada programming language as defined in
[Ada2012].
Ada
Rapporteur Group (ARG): A subgroup of ISO/IEC/JTC1/SC22/WG9, the
International Organization for Standardization Working Group for Ada.
Members of the ARG are appointed by the convener of the ISO working
group for the purpose of resolving issues with respect to the
interpretation of the Ada programming language.
Adaptive
Maintenance: Maintenance performed to adapt a product to a changed
environment.
Base
implementation class: An implementation class in which the processor
has been awarded certified status through testing by an ACAL (see
Section 7.1.1).
Certified
Processors List (CPL): A publicly available list of processors to
which the ACAA has granted certified status. The CPL is maintained by
the ACAA.
Certified
status: (Also "certified as conforming") The status granted
to an Ada processor by the award of an ACAC (see Section 5.4).
Client:
An individual or corporate entity that has an agreement with an ACAL
that specifies the terms and conditions for ACAL services (of any
kind) to be performed. (Also used to refer to an organization that
intends to make such an agreement.)
Computer
system: A system containing one or more computers and associated
software. [ANSI/IEEE 90] In this document, a computer system
comprises the hardware and software (operating systems, kernels) that
are essential to the operation of the processor or the compiled code;
e.g., input/output devices are generally not included in this
definition.
Configuration:
A specific host and target computer system. “Configuration”
is usually used along with “processor” to completely
specify a conformity assessment.
Configuration
management: A discipline applying technical and administrative
direction and surveillance to: identify and document the functional
and physical characteristics of a configuration item, control changes
to those characteristics, record and report change processing and
implementation status, and verify compliance with specific
requirements. [ANSI/IEEE 90]
Conformity:
Fulfillment by a product, process or service of all requirements
specified. [ISO/IEC 86] See also Subclause 1.1.3 of [Ada2012].
Core
language: The Sections 1-13 and Annexes A, B and J of [Ada2012].
Corrective
maintenance: Maintenance performed to correct faults.
Customized
test suite: The ACATS tests, adjusted as necessary, that must be used
for witness testing of a given Ada processor (see Section 4.3).
Declaration
of Conformity: A formal statement from a client declaring that
conformity is realized on the Ada processor for which conformity
assessment status is requested (see Section 5.2).
Equivalence
(of ACATS results): A set R of ACATS results is equivalent to a
previously evaluated set of results, P, provided that (1) the ACATS
used in producing R is either the same as the ACATS used in producing
P, with the possible exception of using different
implementation-specific substitution values, or is the current ACATS
with the same code modifications applied as the ACATS used in
producing P; and (2) application of the ACATS grading rules results
satisfies all of the following: (a) every test graded as Passed in P
is also graded as Passed in R; (b) every test graded as Inapplicable
in P is graded as Passed or Inapplicable in R; (c) every test graded
as Unsupported in P is graded as Passed, Inapplicable, or Unsupported
in R; and (d) every test not present in P is graded as Passed,
Inapplicable, or Unsupported in R (see Section 7.2).
Host
computer system: The computer system on which a processor is
installed and executes.
Implementation:
A processor running on a particular configuration.
Implementation
class: A collection of implementations that are sufficiently closely
related that the certified status of one member of the collection may
be extended to the other members, provided that certain conditions
are met (see Section 7.1).
Maintained
implementation class: An implementation class in which the processor
is derived (by applying corrective and perfective maintenance
changes) directly from the processor that has been awarded certified
status by ACAL testing, and in which the processor has a mode of
operation in which it can produce ACATS results equivalent to those
of the processor named in the ACAC (see Section 7.1.2).
Operating
system: A collection of software, firmware, and hardware elements
that controls the execution of computer programs and provides such
services as computer resource allocation, job control, input/output
control, and file management in a computer system. [ANSI/IEEE 90]
Perfective
maintenance: Maintenance performed to improve performance or
maintainability. [ANSI/IEEE 90]
Processor:
A compiler, translator, or interpreter. The processor includes all
tools used in creating programs. For instance, many systems will
include a linker in the processor. A processor works in conjunction
with, but does not include, a configuration. In this document,
processor typically means an Ada processor.
Rehosted
implementation class: An implementation class in which the processor
is derived directly (by applying corrective, perfective, and adaptive
maintenance changes) from the processor that was awarded certified
status by ACAL testing; in which the common target computer system is
the same as that of the certified processor; and in which the
processor has a mode of operation in which it can produce ACATS
results that are equivalent to those of the certified processor (see
Section 7.1.3).
Self-testing:
The process of producing the results of processing an appropriately
customized test suite by the client (see Section 5.2).
Software
maintenance: Modification of a software product after delivery to
correct faults, to improve performance, or to adapt the product to a
changed environment. [ANSI/IEEE 90]
Specialized
Needs Annexes: Annexes C through H of [Ada2012]. These Annexes define
standards for additional functionality required by specific
application areas. An Ada processor may support some or none of these
annexes.
Target
computer system: The computer system on which the executable code
generated by a processor is loaded and executes.
Test
issue: (Also "dispute") Any problem arising during
conformity assessment (see Section 6).
Validated:
Equivalent to “certified status”. The status granted to
an Ada processor by the award of an ACAC (see Section 5.4). We define
this equivalent term to correspond to common usage in the Ada
community.
Validation
Certificate: Equivalent to “Ada Conformity Assessment
Certificate (ACAC)”. A certificate issued by authority of the
ACAA and an ACAL for a successfully tested Ada processor (see Section
5.4). We define this equivalent term to correspond to common usage in
the Ada community.
This
section specifies the roles of the bodies that are responsible for
Ada conformity assessment of clients who receive service from them.
An ACAL
is an independent testing laboratory that performs Ada conformity
assessment activities. [ISO 99] includes a list of requirements that
a testing laboratory must meet in order to be considered an ACAL.
These requirements will not be repeated here. The ACAL operates under
an ACAP consisting of its own operating procedures and the procedures
defined in this document. An ACAL performs the following principal
functions:
- in
cooperation with other ACALs, recognizing an ACAA;
- entering
into a contract for conformity assessment services with a client;
- evaluating
the results of a client's self-testing of a candidate Ada processor,
using the ACATS;
- forwarding
unresolved test issues to the ACAA for review and resolution;
- conducting witness testing of Ada processors;
- documenting
witness testing in an ACATR, forwarding the ACATR to the ACAA for
review and comment, and distributing the ACATR; and
- with the concurrence of the ACAA, issuing an ACAC
(certificate) for each processor the ACAL observed to successfully complete
witness testing.
The ACAA ensures worldwide
commonality of the Ada Conformity Assessment Process. The technical and
administrative functions of the ACAA are carried out by a technical
agent. It is established by a sponsor and is advised by an Advisory
Board.
The Ada
Resource Association, a trade association of Ada product suppliers,
sponsors the ACAA. The sponsor is responsible for the following:
- establishing the initial Operating Procedures for the ACAA;
- identifying the ACAA technical agent;
- arranging for funding of the ACAA technical agent;
- naming the members of the ACAA Advisory Board; and
- providing an Internet site for hosting the ACATS, the
on-line version of the Certified Processors List, and other conformity assessment
information.
ACAA
technical agent supports and coordinates the activities of the ACALs
by:
- providing
the members of the Advisory Board with sufficient information to
keep them aware of the status of the Ada Conformity Assessment
program;
- advising
the ACALs and the ACAA sponsor with regard to the conformity
assessment procedures;
- reviewing
each ACATR produced by each ACAL, assessing the fulfillment of
conformity assessment requirements by a client, and recommending to
the ACAL that a conformity assessment certificate be issued on the
successful completion of witness testing;
- receiving
and ruling on challenges to ACATS test programs, including
withdrawing or requiring modifications to such test programs;
- accepting,
reviewing, and acting upon requests for registration of processors
derived from processors that are certified as conforming;
- providing
configuration management of the ACATS;
- maintaining
a publicly available Conforming Processors List; and
- maintaining
a liaison with the ARG.
The ACAA
Advisory Board represents the interests of the wider Ada community in
the Ada Conformity Assessment process. Issues of policy and
procedures are brought to the attention of the Board, which may make
recommendations as to their resolution. Board members are appointed
by the ACAA sponsor, and include (but are not limited to) the
following:
- the
ACAA technical agent;
- the
technical director of each ACAL;
- the chair of the ARG;
- the convener of WG9;
- representatives of the sponsor;
- representatives
of the community of Ada users; and
- other
representatives of the wider Ada community.
A client
is an individual or organization that contracts with an ACAL for
conformity assessment services. Clients are required to provide
accurate and complete information as specified in these procedures
and the procedures of the ACAL.
The
designated ACATS is the suite of conformity tests, support software,
and documentation formerly known as the Ada Compiler Validation
Capability (ACVC). The ACVC was developed under various contracts
with the United States Department of Defense. It is designed to
demonstrate the conformity of an Ada processor with the standard
[Ada2012]. The use of the ACATS is documented in the
ACATS User’s
Guide[ACATS UG], which explains the criteria for evaluating the
results of the individual tests. While the ACVC was produced under
contract to the United States Government, it is available to any
individual or organization. The ACAA controls the content of the
ACATS as it is used in conformity assessments, including the
modification and addition of tests. Questions concerning Ada
conformity assessment or comments on ACATS test programs should be
submitted to the ACAA (see
Appendix A, Points of Contact).
Each
ACATS test program has one or more test objectives that are described
in a comment in the test program. Some test objectives might address
language features that are not required to be supported by every Ada
processor (for example, “check that the proper exception is
raised when Float’Machine_Overflows is True”). These test
programs generally contain an explicit indication of their
applicability and the expected behavior of processors for which they
do not apply. The determination of applicability is made according to
the grading criteria in the ACATS User's Guide or in the
internal test documentation, or as a ruling by the ACAA. For a
processor to be certified as conforming, all applicable test programs
for the core language (as defined in [Ada2012]) must be processed and
passed according to the specified grading criteria.
Reference
[Ada2012] includes certain sections designated as Specialized Needs
Annexes (SNA). The set of ACATS test programs for any of the SNAs
will be processed only upon client request (to demonstrate full or
partial support of the Annex). As permitted by [Ada2012], test
programs for the SNAs may be rejected at compile time or may exhibit
run-time behavior that indicates a lack of support for requirements
that only apply to SNAs. The ACAA may rule that tests producing such
behavior are graded as "Unsupported". If the ACAA finds
that the behavior is not in accordance with the permission granted by
[Ada2012], then the tests are graded as "Failed". Tests
graded as Unsupported are reported in the ACATR and the ACAC, but
these results do not affect the designation of the processor as being
certified as conforming. On the other hand, tests graded as Failed
are evidence of non-conformity, precluding the issuance of an ACAC
for the candidate processor.
The
various ACALs and the ACAA strive to apply the ACATS as uniformly as
is practical to all Ada processors. In order to apply common test
objectives that depend on implementation-dependent characteristics
(e.g., line lengths and numeric types), some test programs must be
adjusted to a given implementation following the procedures in [ACATS
UG]. These adjustments consist of inserting implementation-dependent
values in prescribed places in certain test programs.
In
addition to the anticipated test modifications, other changes may be
required in order to remove conflicts between a test program and
implementation-dependent characteristics (for example, the algorithm
for recovering from syntax errors). The allowable changes for each
Ada processor are determined by the [ACATS UG] and the ACAA, and may
require ACAL assistance — especially in the case of processor
error-recovery problems.
In order
to meet a test objective, it may be required to modify the code, the
processing method, or the grading of a test program. Only the ACAA
shall make the decision to use any of these modifications, as
described below:
- Code
Modification: The source code of the test is changed. Examples of
code modifications are the insertion of a pragma, the insertion of a
representation clause, or the splitting of a B-test into several
individual tests, if the processor does not detect all intended
errors in the original test. (This last example is the only
exception to the rule that only the ACAA shall make the decision to
apply a Code Modification.) Note that code modifications may be
required for all subsequent conformity assessments. These will
result in the identification of a new release of the current ACATS
version and an updated ACATS Modified Tests List
- Processing
Modification: The processing of the test by the Ada processor for
conformity assessment is changed. Examples of processing
modification are the change of the compilation order for a test that
consists of multiple compilations and the additional compilation of
a specific support unit in the library.
- Grading
Modification: The grading of a test result is changed. An example of
a grading modification is the grading of a test other than that
indicated by the output from the test program. This may be required
if the test makes assumptions about implementation features that are
not supported by the implementation (for example, the implementation
of a file system on a bare target machine).
The ACAL
customizes the ACATS for each processor that is subject to witness
testing. This customization always includes making all
requiredimplementation-dependent substitutions. It may also include
making code modifications that the ACAA directs for that specific
conformity assessment as well as removing some inapplicable test
programs as allowed by the ACATS User's Guide.
The
result of processing an ACATS test program can be given only one of
four possible grades: Passed, Inapplicable, Unsupported, and Failed.
The first three grades are considered to constitute acceptable
results. ACATS test programs that contain illegalities (which an
implementation must detect) generate diagnostic output that must be
inspected manually or by pattern-matching algorithms, matching system
diagnostics to the intended errors. Executable ACATS test programs
generate output using report procedures, which can be graded
automatically. The ACATS report package, Report, contains specific
output procedures for the two grades Failed and Inapplicable. If
neither of these is invoked, the Report.Result procedure will report
Passed or Tentatively Passed (indicating that the test has passed if
manual inspection reveals that specific additional requirements are
met). These results are the only ones that are generated by the test
code (if no result is reported, that is, if the test completes
abnormally, the result is graded Failed). The grade Unsupported is
established as a means of grading tests that apply to the Specialized
Needs Annexes (SNAs), as explained below.
The
ACATS test programs for the SNAs pose two problems for using the
three conventional grades of Passed, Inapplicable, and Failed. The
broad problem is that full support of any such Annex is not required
for conformity to [Ada2012] — there may be no support, or
merely partial support. Unfortunately, there is no way to
discriminate between full and partial support if only those three
grades are used, since the grades Inapplicable and Failed are not
appropriate for this (an implementation is not allowed to provide
deviant semantics for an unsupported Annex feature — that would
be a conformity assessment failure). The second problem is that there
are some test programs for Core features that are applicable also to
a SNA, in particular, the test programs for representation items.
These programs constitute tests for features that are defined in the
Core as optional, but are mandatory for full support of the Systems
Programming Annex (which itself is mandatory for full support of the
Real-Time Systems Annex).
Therefore,
the ACAL grades the result of processing such an ACATS test program
(i.e., one that uses a feature required by, or defined in, an Annex)
as Unsupported, if the prima facie result is failure but the
implementation's processing of the test program is an acceptable form
of non-support. For example, if a processor does not support a
particular form of a representation clause, it must reject any test
program that uses it — such rejection of an executable test is
usually graded Failed, but is graded Unsupported if the
implementation does not claim support of the relevant SNA. A
processor that accepts the representation clause and reports Failed,
on the other hand, is deemed to have failed the test regardless of
any claim (or lack thereof) of support for the SNA.
The
current baseline version of the ACATS is available to the general
public from an ACAL or from an Internet site supported by the ARA.
The current and any previous versions of the ACATS Modification List
are available from the same sources. ACALs may assist the client in
format conversion when providing the ACATS in a particular
distribution medium. If a client has a need for a superseded version
of the ACATS, it may be available from the ACAA or from an ACAL. See
APPENDIX A for points of contact
The ACAA
maintains an official ACATS web site, accessible via the Internet.
The web site includes a web-accessible version control system, which
contains the official version of the ACATS. Both old and new versions
of tests are accessible given the test name and version label
desired.
Instructions
for using the ACATS version control system are available on the web
page.
Test
challenges and ARG interpretations may reveal flaws in the ACATS. The
ACAA may remove, repair, and insert tests in response to those needs.
These test suite changes are listed in the Ada Conformity Assessment
Test Suite Modification List (ACATS Modification List, or AML for
short). The ACAA issues this list as needed. The AML contains
information about affected tests and instructions for acquiring new
and modified tests. The official versions of the tests (including
modified and new tests) are available via the official ARA web site
and other sources (see Section
4.5> ACATS Availability)
New
versions of the ACATS Modification List will be issued when test
suite changes are needed. The list contains at least the following
information:
- The
list's date, version identifier, the baseline version, and a list of
changes since the last list;
- For
each affected test, test name, category, new version label, and a
short description of the reason for the inclusion on the list. For
Allowable Modified tests, the date that the modification becomes
required, and the original test's version label also are included
(since it still can be used for conformity assessment); and
- A summary of how to access new and modified tests.
The
official version of a test is always available from the ACATS VCS.
Information in the list is advisory only; in the case of a conflict
between the ACATS VCS, and the list, the ACATS VCS is assumed
correct.
The ACAA
may issue various kinds of test suite modifications. Each affected
test has one of the following categories:
Withdrawn:The test is seriously flawed. It
may have errors that cannot be corrected, or may require extensive
corrections. It is removed from the test suite. Conformity
assessments do not process such tests.
If a correction of a
Withdrawn test is prepared, it will be treated as a new test.
Allowed Modification: The test has minor
flaws. A modified version has been posted on the ACATS VCS. The test
has an effective date that specifies when the test will be moved to
the Modified Category.
The
effective date will always be at least three months after the posting
date, and will be at the beginning of a quarter (that is, January
1st, April 1st, July 1st, or October 1st). A conformity assessment
may choose to process the original test or the new, modified test.
Test choices can be made on an individual test basis. That is, a
conformity assessment may choose to process some new modified tests
while using the original tests for others.
Modified:The test has minor flaws. A
modified version has been posted on the ACATS VCS. Conformity
assessments must process the modified test.
Pending
New: The test is newly created,
or is a correction of a test that was previously withdrawn, or has
added test cases. It is posted on the ACATS VCS. The test has an
effective date that specifies when the test will be moved to the New
category. The effective date will always be at least six months after
the posting date, and will be at the beginning of a quarter (that is,
January 1st, April 1st, July 1st, or October 1st). Pending New tests
are not used for conformity assessment until the effective date is
reached. Users of the test suite are encouraged to run the tests as
soon as possible.
If it becomes necessary
to modify a Pending New test, the effective date is adjusted as if
the test was newly created.
New: The test is newly created, or is
a correction of a test that was previously withdrawn, or has added
test cases. It has been posted on the ACATS VCS for at least six
months. Conformity assessments must process the test.
The ACAA
baselines the ACATS periodically. When the ACATS is baselined, the
original official set of files is updated with all of the changes
specified in the ACATS Modification List. Allowed Modification and
Pending New tests are not included (since they have not been
available long enough to include). Additionally, the documentation
associated with the test suite is updated. Future versions of the
ACATS Modification List are then based on the new test version. Any
Allowed Modification and Pending New tests will be listed in the
initial version of the ACATS Modification List for the new baseline
version.
The
effective date of a baseline version will be announced at least three
months prior to its being effective, and should be at the beginning
of a quarter (that is, January 1st, April 1st,
July 1st, or October 1st). The documentation
will be available not less than 30 days prior to it being effective.
Conformity assessments started after the effective date must use the
new baseline test suite. (Note that the tests that make up the
baseline version are known on the date that the effective date is
announced, and are accessible on the web site, even before the final
version of the baseline documentation is available.)
The
tests used for a particular conformity assessment are defined as the
tests from a particular baseline version of the ACATS, modified as
follows:
- removing
the tests categorized as Withdrawn in a specific version of the
ACATS Modification List
- replacing
the tests categorized as Modified in the same specific version of
the ACATS Modification List with the modified versions from the
ACATS VCS;
- adding
the tests categorized as New in the same specific version of the
ACATS Modification List;
- optionally
replacing the other tests categorized as Allowed Modification in the
same specific version of the ACATS Modification List with the
modified versions from the ACATS VCS;
- making any modifications resulting from the ACAA's
resolution of issues arising from the specific conformity assessment; and
- applying the customizations described in Section
4.3 of
this document.
Each
ACATR identifies the baseline version of the ACATS, and documents all
modifications made to that baseline version of the ACATS.
Conformity
assessments must use the most recent version of the ACATS
Modification List at the start of witness testing. (Note that the set
of required tests is always known at least three months in advance.)
Most tests and support files modified for an individual conformity
assessment are included in the ACATS Version Control System. Tests
modified only by making implementation-dependent substitutions
(typically by using a tool) will not be included. B-Test splits
allowed by section 4.2 and tests modified only by splitting between
compilation units will also not be included.
The
files will be posted on the ACATS Version Control System before the
ACATR is issued. Each conformity assessment has a unique version
label, which can be used on the web site to access all of the files
that differ from the baseline versions. The version label is included
in the ACATR. It is intended that the information in the ACATR and
the files available on the official ACATS Version Control System will
allow users to reproduce the conformity assessment on their own.
In order
for a client to obtain a conformity assessment certificate and an
ACATR, the client, the ACAL, and the ACAA must complete number of
steps. The same ACATS version, including the application of the
requirements of the ACATS Modification List, must be used to complete
the steps described in this section. Anyone intending to obtain a
conformity assessment certificate should contact an ACAL without
delay for advice on the handling of the ACATS, on interpretation of
the test grading criteria, and on the operational procedures of that
ACAL.
The
required steps follow:
- Establishment of Agreement
- Self-Test Evaluation
- Witness Testing
- Documentation
In order
to obtain conformity assessment services, an interested party must
become a client of an ACAL by reaching a formal agreement. This
agreement addresses the following topics:
- identification
of the Ada processor(s) and configuration(s) to be tested and the
ACATS version to be used;
- a statement of work, including self-test evaluation,
witness testing, and documentation;
- a
schedule of events and the arrangements for witness testing;
- financial arrangements;
- retention of records;
- ACAL liability; and
- confidentiality of conformity assessment information.
The
schedule for events, deliverables, and payments should take into
account the fact that certain steps in the conformity assessment
process require interaction with the ACAA. The ACAA and ACAL will
keep confidential a client’s intent to obtain a conformity
assessment certificate and the projected schedule for conformity
assessment. If the client requests more restrictive confidentiality
conditions for reasons of national security or procurement
sensitivity, the client will provide to the ACAL an official, written
statement describing the request and the reason(s) for the request;
the ACAL will also obtain further guidance from the ACAA.
Self-test
evaluation entails a series of actions and is usually where the bulk
of the conformity assessment effort is expended. These actions are
described in the following subsections.
After
entering into a formal agreement, the client obtains a customized
test suite from the ACAL. (At the client’s risk, the client may
prepare this customized test suite according to instructions in the
ACATS User's Guide, rather than obtaining it from an ACAL).
The client then processes all the tests in this customized test suite
using the candidate processor on the candidate configuration or on
another configuration that produces the same result. If the
implementation provides for options in the way programs are
processed, then the same set of options must be chosen for all test
programs, with the possible exception of options controlling the
production of information output. (For example, options that control
the format of listings, the format of error messages, and the
generation of listings may vary.) Any other exception constitutes a
test issue that must be resolved with the ACAL (see Section
5.2.3).
Test issues should be sent to the ACAL for analysis as soon as
possible.
Self-test
activities include as a minimum the processing of an appropriately
customized test suite by the client, preparation of a client supplied
Declaration of Conformity, and submission of any test issues.
Upon
completion of self-testing, the client delivers the complete set of
results in the agreed format to the ACAL. (See Section
5.2.4 for an
alternative to submission of complete results.)
Results
are accompanied by the following information:
- a
signed Declaration of Conformity (see below) giving the complete
identification of the components of the processor and configuration;
- a
list of test programs for which the implementation does not produce
an acceptable result (see Section 4.4), together with a rationale
for the implementation’s behavior;
- a
list of test programs for which the implementation produces an
Inapplicable result (whether self-reporting or otherwise), together
with a rationale for the implementation's behavior;
- the
necessary information for the ACAL to prepare the customized test
suite, including tests that must be split, the
implementation-dependent substitutions, and the modifications to
those support units specified in the [ACATS UG];
- the
set of Specialized Needs Annexes (SNAs) that the client requests
testing of;
- the
complete set of available option settings for each tool used in
executing the Ada processor;
- the
complete set of processor option settings actually used for
processing the customized test suite, including the default
settings; and
- a
sample script of the processing of a single, representative test.
The
Declaration of Conformity states that the organization responsible
for the production, maintenance or distribution of the Ada processor
is offering a product that is in conformity with [Ada2012]. The
client must ensure that the information contained in the Declaration
of Conformity does not infringe on the rights of a third party, and
may be required to provide a written statement of consent from any
third party involved. The Declaration of Conformity becomes part of
the ACAL records and is copied into the ACATR. The ACAL will not
issue a certificate until the ACAA has reviewed a signed Declaration
of Conformity. (See
Appendix C for an example of the Declaration of
Conformity.)
The ACAL
analyzes the client's submitted results of self-testing, checking
that all test programs have produced acceptable results according to
the ACATS evaluation criteria. During this analysis period, the
client and the ACAL resolve any test issues.
A test
issue is defined to be any of the following:
- a missing or incomplete result to a test program;
- a result presented in an inadequate form;
- a result that is not graded Passed or Inapplicable by
the documented evaluation criteria (note that Unsupported grades require submission
of a test issue);
- a disagreement between the client and the ACAL as to the
interpretation of a result;
- a change in the choice of options to be used during
testing; and
- any implementation characteristic that might affect the
conformity of the implementation during testing.
A client
may challenge an ACATS test program's correctness or applicability to
a particular implementation. Such challenges should be presented to
the ACAL in the petition format given in Appendix B. The ACAL will
forward any petitions to the ACAA for resolution; the ACAA will
strive to rule on the petition within two weeks of receiving it. The
ACAA reports all challenges and rulings to each ACAL. However, an
ACAL may not apply an ACAA ruling for one conformity assessment to
another conformity assessment without the ACAA so directing. (See
Section
6 for a description of the Challenge and Resolution Process.)
In some
cases, it may be agreed to leave a test issue until witness testing.
For example, it might be impossible to check the processing of
control characters by inspecting printed results. The ACAL will note
any unresolved issues and describe the results that are expected
during witness testing. It is also possible that the client
information for the production of the customized test suite (see
Section
5.2.2) was insufficient, so that corrections to the
customized test suite must be made, requiring additional processing.
The ACAL
and the client may agree that, at the client’s risk, parts of
the customized test suite need not be processed during self-testing.
There are two typical situations, as follows:
- the
client submits full test results from one processor and
configuration and either certifies that another processor and/or
configuration has identical results or submits automated difference
reports; or
- the
client certifies that the processor and configuration under test
produces results that are identical to those from a previously
submitted complete set of results evaluated by the ACAL (and still
in the possession of the ACAL) or submits automated difference
reports.
The
normal practice is to submit complete self-testing results for at
least one of the implementations under test. The ACAL may require the
submission of complete self-testing results.
Self-testing
is successful if the analysis of results and the resolution of test
issues show that all results have been provided and are acceptable.
Self-testing is successful with caveats if the results are
satisfactory except that they were incomplete or if resolution of
some test issues is deferred until witness testing by agreement
between the ACAL and the client.
Upon
successful completion of self-testing, with or without caveats, the
ACAL witnesses testing of the Ada processor in accordance with the
formal agreement between the ACAL and client. Witness testing takes
place in the presence of qualified ACAL ersonnel.
“Presence”
means either physical presence or telepresence as agreed between the
ACAL and client. Telepresence must be sufficient for the ACAL to
carry out the observations and monitoring required below. The
ACAL supplies a customized test suite that it has prepared based upon
client information and any information collected during the
resolution of test issues. The customized test suite will include the
set of test programs for the core language and each set, as requested
by the client, any (or none) of the Specialized Needs Annexes (SNAs).
The ACAL verifies that the processor identification, including
identification of the processor and configuration (hardware systems
and operating systems), matches that given in the Declaration of
Conformity. (If it does not agree, then the client must provide a new
Declaration of Conformity.)
The ACAL
observes the installation of the customized ACATS on the host
computer system, monitors the processing of the customized ACATS on
the host and target computer systems, and evaluates the results.
The entire customized test suite should be run on a single copy of
the Ada processor on a single configuration using a unique set of
option settings of the processor. (Differences in options controlling
the production of information output and differences from accepted
test issues are allowed. See section
5.2.1.) If the ACAL
determines that the results agree with those obtained from
self-testing and are satisfactory with respect to any caveats, the
witness testing has been successful; otherwise, the test is
unsuccessful. If any result of testing with a set of test programs
for a Specialized Needs Annex is unacceptable, the test report and
certificate of the conformity assessment will not recognize that the
set was processed
.
Each
conformity assessment effort is documented by an ACATR, and each
successful effort is further documented by an ACAC.
An ACATR
is produced for each processor and configuration subjected to witness
testing. Each ACATR contains, at a minimum, the following
information:
- Name and address of the ACAL;
- Location and date of witness testing;
- Identification of the test report by serial number,
date, or other appropriate means;
- A copy of any certificate awarded as a result of the
conformity assessment;
- Name and address of the client (and certificate awardee,
if different);
- Date, system, version, and release of the processor
being tested, and identification of any processing options used;
- Name and address of the manufacturer of the processor,
if different from the client;
- Complete description of the configuration including name,
model, and version of the hardware and software used during the test;
- Identification of the Specialized Needs annexes tested;
- Identification of the specific ACATS version used in
witness testing, including the conformity assessment’s ACATS VCS version label;
- All modifications to the test suite as used in witness testing,
including rationales;
- Identification of the manner in which witness testing
was conducted (such as physical presense or telepresence). If telepresence was
used, a description of the telepresence facilities used;
- Any deviations from, additions to, or exclusions from
the test procedures, and any other information relevant to a specific test,
including inapplicable and unsupported tests and the rationale for
inapplicability or non-support;
- A statement to the effect that the test report relates
only to the specific processor and configuration tested;
- A statement to the effect that the report shall not be
reproduced except in full, without the written approval of the ACAL; and
- Detailed descriptions of any test results revealing
non-conformities; and
- Signature and title of an authorized representative of
the ACAL responsible for the test report; and
- Signature of an authorized representative of the ACAA.
5.4.1.1 ACATR Production
The
ACATR is prepared by the ACAL but includes material that is produced
by the client, such as the documented processor options used during
witness testing. A draft version of the ACATR, based on results and
circumstances implied by the evaluation of self-testing results, is
sent to the ACAA for review. The draft version is also submitted to
the client for review during witness testing, and is updated to
account for client comments and observations made during witness
testing. For a successful conformity assessment, the final version of
the ACATR is signed by the ACAL and the ACAA. For an unsuccessful
conformity assessment, the final ACATR is provided to the client
only.
Final
test reports will never be modified. If it becomes necessary to
correct a final ACATR, the ACAL will prepare a separate document
titled “Supplement to Ada Conformity Assessment Test Report
<unique report identifier>”. Such a supplement will meet
the applicable requirements of section 5.4.1.
5.4.1.2 ACATR Availability
The
final version of the ACATR for a successful conformity assessment is
available to the general public from the client, from the ACAL that
produced it, and from the ACAA in electronic form. The ACAL may
require payment of a fee for ACATR reproduction and delivery. (See
Appendix A for points of contact.) By including an appropriate
request on the Declaration of Conformity (
Appendix C), the client may
disallow public availability of the ACATR and the ACAC.
With the
concurrence of the ACAA, the ACAL issues an Ada Conformity Assessment
Certificate (ACAC) for each processor and configuration that was
subject to successful witness testing. The information on the
certificate is derived from the client's Declaration of Conformity
and the ACATR. The ACAC conveys to the processor and configuration
the status of certified as conforming, as defined in [ISO 99]. An
entry is made in the CPL for each ACAC, unless the client has
requested confidentiality on the Declaration of Conformity (see
Appendix C).
The ACAC
contains the following information:
- Name and address of the ACAL;
- Name and address of the organization receiving
the certificate, and the
name and address of the manufacturer of the processor, if different;
- Unique certificate number;
- Date of issuance;
- Date of expiration (two years after the date of issuance);
- Unambiguous identification of the tested processor and
configuration;
- Identification of the ACATR on which the certificate is
based.
- Identification of the language standard against which
conformity was tested;
- Identificationof the test suite used in the assessment;
- The number of test programs graded Passed, Inapplicable,
and Unsupported for each Specialized Needs Annex that was tested and
identification of those annexes not tested
- A statement of restriction of applicability to the
specific processor nd configuration tested; and
- Signatures and titles of the ACAL and ACAA representatives
authorized to sign certificates.
Note
that an ACAC attests that testing was performed on a specific
processor using a specific test suite running on a specific
configuration, following the Ada Conformity Assessment Procedure, and
that no evidence of non-conformity was detected. It does not certify
that the processor is free of defects, nor does it certify that the
processor is usable for any particular purpose.
ACACs
expire two years after issuance. When an ACAC expires, the
corresponding entry in the CPL is clearly identified as expired.
(Entries for derived processors may also expire at the same time, see
section
7.5). Certificates expire in order to encourage periodic
retesting of processors, which ensures that they continue to meet the
requirements of conformity assessment.
For some
special procurement requirements, a client might wish to have witness
testing done with an obsolete version of the ACATS. The ACAP does not
include any procedures for recognizing testing with obsolete test
suite versions, but the ACALs may provide such a service outside the
system. Ada Conformity Assessment Certificates will not be issued for
testing with obsolete test suites, nor will CPL entries be created
based on such testing.
The ACAA
retains a copy of each ACATR (which includes a copy of the
Declaration of Conformity and the ACAC), records pertaining to issues
and their resolution, and a copy of each registration request. The
ACAL retains a copy of each ACATR, a copy of the customized ACATS
used in witness testing, and a copy of the witness testing results.
The ACAA retains its records until at least five years following
expiration of the ACAC. Each ACAL's procedures specify the length of
time that its records are retained, but records must be retained at
least seven years after the completion of witness testing.
The
client must agree not to advertise or make public claims that the Ada
processor is certified as conforming until after receiving the ACAC
or receiving formal notification from the ACAL that it has issued an
ACAC. A client who intends to advertise the completion of events that
indicates progress toward completion of conformity assessment must
sign a waiver of confidentiality. If a waiver of confidentiality has
been signed with the ACAL, the ACAL will respond to inquiries about
the client’s advertisements or public claims by acknowledging
receipt of conformity assessment materials (i.e., a formal agreement,
self-testing results, or witness testing results) without judgment
concerning the success of the witness testing.
This
section presents the process whereby tests may be challenged,
possibly resulting in their modification or withdrawal.
A
“deviation” is defined by the
ACATS User's Guide
as any result from processing an ACATS test program that is not a
Passed or Inapplicable result according to the established grading
criteria. This intentionally broad definition of a “deviation”
is intended to ensure that processor implementers bring all deviant
test results to the attention of the ACAA or ACAL, without assuming
that such results are acceptable. In petitioning for acceptance of a
deviation, the petitioner provides a rationale for each challenge
made against a test program. Petitions are sent to the ACAA, usually
electronically, by the petitioner or by an ACAL on behalf of its
client. For each deviation that is accepted (that is, when the ACAA
rules in favor of the petition), generally some correction is
indicated for the cited tests. The ACAA may withdraw a test program
or require that a modified version of the test be processed (see
section
6.3). Withdrawal of a test program or the provision of a
modified version of a test results in the release of a new version of
the ACATS Modification List.
The ACAA
typically resolves challenges by any of three methods:
- a
resolution that was made previously is applied to the current
petition (e.g., the same petition might be submitted at different
times by different petitioners);
- the
resolution can be determined unequivocally based on the Ada standard
or Ada Commentaries; or
- the
resolution is based on the deliberations of a body of Ada experts.
Although
these procedures do not set a time limit for reaching a resolution,
the ACAA attempts to rule on petitions within two weeks. Clients
should submit challenges well in advance of a scheduled witness
testing date (see Section
5.1).
On
receipt of a petition, the ACAA checks whether the issue matches any
that have been previously resolved. If the challenge is new, it is
given an initial ACAA analysis that involves research using the Ada
Commentaries in conjunction with the Ada standard and references to
previous deliberations. Often the ACAA consults Ada experts in order
to resolve a petition. The identity of the petitioner is not
disclosed when consulting outside experts. Resolution of a petition
is made by the ACAA, and all ACALs are informed of the resolution.
The
resolution of a petition is either an acceptance or rejection of the
petitioner’s arguments. Acceptance can result in withdrawal of
the test program from the ACATS, or a modification for conformity
assessment. A test issue may lead to the withdrawal of a test program
if the test is shown to be incorrect to a degree that wrongly
influences implementation. If the challenge shows the affected test
program(s) to be incorrect in only a minor, limited degree, generally
the ACAA will direct that the test(s) be processed with a test
modification.
There
are three types of test modification: Code, Processing, and Grading
modifications.
- A
Code Modification is an actual change to the code of the test (e.g.,
adding a choice to an exception handler).
- A
Processing Modification is a change to the way in which the test is
processed (e.g., re-ordering the compilation of component files of a
multiple compilation test).
- A
Grading Modification is simply the grading of the observed results
by other than the established grading criteria (e.g., interpreting
particular intermediate output and a final Failed result as Passed,
Inapplicable, or Unsupported, according to an understanding of the
test issue).
All test
modifications are documented in the ACATR.
A
petitioner may resubmit a rejected petition, clearly stating
additional information and reasoning as to why the original
petition resolution is incorrect. The
ACAA will resolve the resubmitted petition based on the deliberations
of a body of Ada experts. A resolution of the resubmitted petition
will be provided in no more than three weeks after submission.
A
petitioner may resubmit a petition twice. A petitioner who has
resubmitted a petition at least once may also request an extended
resolution. In an extended resolution, the ACAA forwards the
challenge to the ARG for resolution. (Extended resolution is not
available for issues that have an interpretation approved in the last
two years.) It is not anticipated that the ARG will resolve the issue
in time for the conformity assessment that gave rise to it.
Therefore, the tests involved in an extended resolution will be
graded as Unsupported; they will not be graded as failures for the
purpose of issuing a certificate of conformity.
However,
the expiration date of the certificate shall be marked "pending
issue resolution by ISO/WG9". The certificate shall expire on
the day on which WG9 approves an interpretation of the Standard
contradicting the petition and the processor will be removed from the
Certified Processors List, or on its normal expiration date,
whichever is sooner. The mark shall be removed from the Certified
Processors List if WG9 approves an interpretation of the Standard
confirming the petition.
There is
no limit on the number of test programs that can be challenged by a
petitioner. Although there is a risk that a petition will not be
decided in a conformity assessment client’s favor, early
submission of petitions can reduce the risk that a conformity
assessment will not be successfully completed on schedule. Any
interested party may challenge an ACATS test program.
As
permitted by [ISO 99], the ACAA provides mechanisms for extending the
certified status of a tested processor to an implementation class (a
set of closely related processors operating on a range of compatible
configurations). This section describes these certification extension
mechanisms.
An Ada
processor is typically designed to be used on any member of a set of
host and target computer-system pairs; furthermore, a processor is
usually provided with different modes of operation (also known as
“options” or “switch settings"). In witness
testing, a processor is tested under one mode of operation on a
particular configuration (host-target pair). The particular processor
that is tested may be viewed as representing an implementation class,
consisting of a particular (binary) processor and any configuration
(host-target pair) on which it operates and produces equivalent ACATS
results. Related implementation classes may include processors that
are maintained versions of the test processor, and processors for
which the host system is different. The ACAA may extend the
"certified conforming" status to entire implementation
classes.
The
tested processor may be viewed as representative of several related
implementation classes. These classes are categorized and defined in
the following subsections.
A base
implementation class includes a single (binary) processor that has
achieved certified status through a complete conformity assessment
(including witness testing). The processor may operate on multiple
(closely related) configurations. The target instruction set
architecture and target operating system of the additional
configurations must be the same as or a superset of those of the
witness tested processor. The host system must be able to execute the
witness tested processor. The processor must have a mode in which it
can produce ACATS results that are equivalent (see section
7.2) to
those of the tested processor for each configuration in the class.
A
maintained implementation class is a class that includes a single
(binary) processor that satisfies the following conditions:
- by
applying corrective and perfective maintenance changes the processor
is derived from a processor that is named in an ACAC; and
- the
processor has a mode of operation in which it can produce ACATS
results that are equivalent to those of the processor named in the
ACAC.
The
restriction of maintenance changes to corrective and perfective
maintenance implies that the processor must have the same
configuration(s) as the base processor class for the processor named
in the ACAC.
A
rehosted implementation class is a class that includes a single
(binary) processor that satisfies the following conditions:
- by applying corrective, perfective and adaptive
maintenance changes the processor is derived from a processor that is named
in an ACAC;
- the
processor's target computer system is the same as that of the
processor named in the ACAC; and
- the
processor has a mode of operation in which it can produce ACATS
results that are equivalent to those of the processor named in the
ACAC.
Adaptive
maintenance may include limited changes to enable the processor to
operate on a different host system from that of the processor named
in the ACAC.
A
rehosted implementation class may include closely related target
systems. As with a base implementation class, the
target instruction set architecture and target operating system of
the additional target systems must be the same as or a
superset of those of the processor named in the ACAC.
The
conditions for extending the certified status of a processor require
that the candidate processor be capable of producing ACATS results
that are equivalent to those produced by the certified processor and
configuration. In this context, equivalent ACATS results are those
satisfying the following conditions:
- The
customized ACATS used in testing the candidate is the same as or a
modified version of that used in witness testing the certified
processor. The modification may consist of providing different
implementation-specific substitution values. Alternatively, the
customized ACATS used in testing the candidate is the current ACATS
along with the most recent version of the ACATS Modification List,
customized as described in section 4.3. The customization includes
substituting implementation-specific substitution values, and must
include all Code Modifications listed in the ACATR for the certified
processor. In both cases, any other modifications must be approved
by the ACAA.
- For
every test that was graded as Passed by the certified processor, the
candidate processor's result shall be graded as Passed according to
the ACATS grading rules.
- For
every test that was graded as Inapplicable for the certified
processor, the candidate processor's result shall be graded as
either Passed or Inapplicable according to the ACATS grading rules.
- For
every test that was graded as Unsupported for the certified
processor, the candidate processor's result shall be graded as
either Passed, Inapplicable or Unsupported according to the ACATS
grading rules.
- For
every test not present for the certified processor, the candidate
processor's result shall be graded as either Passed, Inapplicable or
Unsupported according to the ACATS grading rules. (This can only
happen for tests added to the ACATS since the certified processor's
witness testing.)
The ACAA
must approve any deviation from the above requirements.
The
purpose of the ACATR Supplement is to document the extension of
certified status to an implementation class. See
Appendix D for a
sample ACATR. The Supplement contains the following information:
- identification of the client;
- identification
of the ACAC and ACATR that document the conformity assessment of the
reference processor and configuration from which the processor
classes are derived;
- signature of the ACAA (upon approval);
- the following class-specific information for each
implementation class submitted for approval:
- identification
of the (common) processor;
- the
category of the implementation class (base, maintained, or
rehosted);
- complete
description of the host and target computer systems (configurations)
that, together with the common processor, make up the implementation
of the class;
- identification
of the specific ACATS version (including ACATS VCS label) used to
test the representative processor;
- declaration
that a specifically identified representative processor and
configuration of the class was tested using the customized ACATS
using the same Code Modifications as used in testing the reference
processor;
- declaration
that the representative processor and configuration meets the
definition of a base, maintained, or rehosted implementation;
- declaration
that the representative processor conforms to [Ada2012];
- any
differences between the customized ACATS used in testing the
representative processor as compared to that used in testing the
reference processor, with an explanation of each;
- any
differences in the results of testing as compared to the results
produced by the reference processor, with an explanation of each;
and
- a
brief description of the maintenance changes.
A client
may request certification by extension for a base implementation
class by submitting an ACATR Supplement (see section
7.3) to an ACAL.
Similarly, a client may request certification by derivation for one
or more implementation classes by submitting an ACATR Supplement to
an ACAL.
The Ada
Conformity Assessment Certificate (ACAC) referenced in an ACATR
Supplement must have been issued within the five years previous to
the date of submission of the supplement.
When
submitting an ACATR Supplement (except as noted below), the client
must certify that a representative processor and configuration was
tested using a customized ACATS as described above, and that the
results were equivalent as defined by section 7.2. The client should
be prepared to substantiate this claim as requested by the ACAA or
ACAL.
The
receiving ACAL will check all test result differences indicated in
the ACATR Supplement, checking that all such test programs have
produced acceptable results according to the ACATS evaluation
criteria. Any test issues it identifies shall be resolved as
described for Self-Testing, see section
5.2.3.
Once any
test issues identified have been resolved (possibly by modification
of the supplement), the ACAL will append a summary of the test
results differences to the supplement and then will submit the
supplement to the ACAA for approval. On approval, the ACAA will
create CPL entries identifying the implementation class as certified
by extension or derivation. The ACAA will automatically reject a
supplement with any unresolved test issues.
The
ACATR Supplement for a certification by extension can be submitted at
the same time as the Declaration of Conformity for a conformity
assessment. In this case, the supplement does not need to include a
certification of equivalent results since this is tested by the ACAL
for the conformity assessment.
The ACAA
will reject obviously unreasonable claims of compatible
configurations, but will not do any in-depth analysis of such claims.
Users should regard the claims as vendor claims of compatibility.
Certification
by extension or derivation using the same test suite and
modifications as the original certificate (ACAC) expires at the same
time as the original ACAC. In contrast, certification by extension or
derivation using the current test suite and modification expires two
years after issuance. As with an ACAC, when certification by
extension or derivation expires, the corresponding entry in the CPL
is clearly identified as expired. Certificates expire in order to
encourage periodic retesting of processors, which ensures that they
continue to meet the requirements of conformity assessment.
Any
interested party may challenge any approved certification by
extension or derivation. Such a challenge must include non-conforming
output on a member configuration of the implementation class. If,
after analysis by the ACAA and rebuttal by the client, the processor
is found to violate the requirements of certification by extension or
derivation, the certification will be removed or corrected.
Anyone
wishing to challenge an approved certification by extension or
derivation, must provide the following information to the ACAA:
- Name and address of the challenger;
- Complete
identification of the processor and configuration on which the
non-conformity was noted (this should include exact version numbers
for all hardware and software involved);
- Options
used when running the processor;
- Identification
of the ACATS test(s) that are non-conforming;
- Listings
and test output which supports the non-conformity;
- Any
other supporting information that will help the ACAA make a
decision.
The ACAA
will acknowledge receipt of the challenge. Note that deviations from
the options or customized test suite used for the representative
testing on which the certificate is based will greatly increase the
chances of the challenge being rejected.
The ACAA
will analyze all received challenges, drawing on the test reports for
the original conformity assessments, ACATR Supplements, and other
relevant materials. If the analysis shows that the challenge has
merit, it will be forwarded to the original testing ACAL and to the
client for rebuttal. The client will be allowed thirty (30) days to
prepare a rebuttal to the challenge. Such a rebuttal should show why
the behavior of the processor is conforming, or demonstrate that the
processor does in fact conform when the tests are processed. The ACAA
will rule on the challenge after either receiving the rebuttal or the
expiration of the designated time. The ruling will be distributed to
the ACALs, the client, and the challenger.
If the
final ruling is that a challenge is upheld, the certificate by
extension or derivation will be removed from the CPL or, by agreement
between the client and the ACAA, modified to remove the offending
configuration. If an upheld challenge demonstrates that the client
fraudulently certified the testing of the representative processor
and configuration in the ACATR Supplement, the client’s right
to submit ACATR Supplements will be suspended for a period not less
than six months.
POINTS OF CONTACT
Ada Resource Association
Ben
Brosgol, President
Ada Resource Association, Inc.
P.O. Box 8685
New York NY 10116
Email: brosgol@adacore.com
Ada Conformity Assessment Laboratories
Jean-Pierre Rosen
AdaLog
2 rue du Docteur Lombard
92441 Issy-les-Moulineaux Cedex
FRANCE
Tel: +33 1 45 29 21 52
FAX: 33 1 45 29 25 00
Email: info@adalog.fr
Ada Conformity Assessment Authority
Randall Brukardt
ACAA
P.O. Box 1512
Madison, WI 53701
Tel: 608-245-0375
FAX: 608-245-0379
Email: Agent@ada-auth.org
Ada Rapporteur Group (ISO/IEC JTC1/SC22 WG9/ARG)
Jeffrey Cousins
BAE Systems Integrated System Technologies Limited
Warwick House, PO Box 87, Farnborough Aerospace Centre,
Farnborough, Hants, GU14 6YU, UK
Tel: +44 (0)20 8329 5430
Fax: +44 (0)20 8329 5001
Email: Jeff.Cousins@baesystems.com
Ada Conformity Assessment Test Suite (ACATS)
The
ACATS is available to the general public from an ACAL; it is also
available from the ACAA Internet site.
The
site includes downloadable versions of complete ACATS, the ACATS VCS
for access to individual files and modifications, recent versions of
the ACATS Modification List, and packaged versions of new and
modified tests.
Questions
concerning Ada conformity assessment or comments on ACATS test
programs should be sent to the ACAA (see above).
TEST ISSUE FORMAT
[Part A]
Petitioner: <client name>
Configuration: <host / target hardware and operating systems>
ACATS Version: <ACATS version number>
Self-Test Submittal Date: <due date for self-testing results>
Part
A will be completed once by each client; part B will be completed for
each test issue. It is not necessary for a self-testing submittal
date to have been established. Part A information is treated as
confidential.
[Part B]
Reference: <test name (,test name)>
Summary: <brief description of the test issue>
Discussion: <detailed description of the test issue>
In
this Discussion, arguments should be specified using test line
numbers and references to pertinent sections of the Ada standard,
Technical Corrigendum, or Commentaries (AI-xxxx). The petitioner must
describe the behavior of the implementation for the test or tests
that are challenged, stating the particular test messages produced.
The detailed description can be limited to the particular segment of
test code that is challenged. Relevant source code with processor
messages should be included. (For a group of tests that cause
essentially the same behavior, it is sufficient for a detailed
description to be given for one of them, with the relevant line
numbers given for the like problems in the related tests.)
If
the argument depends upon implementation constraints of hardware or
software (e.g., characteristics of the operating system), then these
should be specified; the particular computer and operating system
should be identified in the Discussion. It is especially important
that implementations that fail to pass some test due to capacity
limitations be described in enough detail for the ACAA to assess the
reasonableness of these limitations.
Failure
to fully specify the points pertinent to a test issue might result in
an adverse decision, with the petitioner having to argue the case
further with a second submittal to the ACAA. It is also possible that
the Summary will suffice to adequately present a test issue.
DECLARATION OF CONFORMITY
Declaration of Conformity
Identification
Client: <client organization name>
Certificate Awardee: <if different from client>
ACAL: <name of Ada Conformity Assessment Laboratory>
ACATS Version: <version number of ACATS>
Ada Processor and Configuration
Processor: <name and version number of Ada processor>
Host Computer System: <host hardware and operating system>
Target Computer System: <target hardware and
operating system>
Declaration:
I, the
undersigned, representing the Client, declare that the Client knows
of no deliberate deviations from the Ada language standard
(ANSI/ISO/IEC 8652:2012) in the Ada processor above. <The next
sentence should normally be deleted.> The Conformity
Assessment Test Report and Ada Conformity Assessment Certificate
associated with this effort are not to be made public without the
Client's permission.
________________________________________ ________________
<Name> Date
<Title>
<Client
Organization>
<Omit
the remainder if the certificate awardee is the client>
Declaration:
I, the
undersigned, representing the Certificate Awardee, declare that the
Certificate Awardee knows of no deliberate deviations from the Ada
language standard (ANSI/ISO/IEC 8652:2012) in the Ada processor
above.
_______________________________________ ________________
<Name> Date
<Title>
<Organization>
ACATR SUPPLEMENT FORMAT
Supplement to ACATR <Reference ACATR number>
<Submittal Date>
name of client organization>
(the client) hereby requests that the Ada Conformity Assessment
Authority (ACAA) extend the certified status documented in the
above-referenced Ada Conformity Assessment Test Report (ACATR) and in
Ada Conformity Assessment Certificate (ACAC) <Reference
ACAC number> to the
implementation class(es) described in the following pages.
Technical Contact:
<Name>
<Address>
<E-Mail>
Approval: ________________________________ ________________
{name} Date
Ada Conformity Assessment Authority
Implementation Class Information
<Complete the remaining pages for each processor class>
Implementation
Class Category: <Base, Maintenance, or Rehosted>
Processor Identification: <Name,
Version, and Release identification>
Host Systems:
<Description
of host computer models and operating systems. Ranges may be used.>
Target Systems:
<Describe
as above or use "Same as host" or "Any host">
Representative
Processor and Configuration Tested:
Host System:
<Identification
of specific host system (hardware and operating system) of tested
processor>
Target System:
<Identification
of specific target system (hardware and operating system) of tested
processor. "Same as host" is acceptable.>
ACATS Version Used for Testing Representative Processor:
<Version
and ACATS VCS label. This version and label must either be the
version and label used for witness testing the processor tested in
the original ACATR, or the current version and label as of the date
of submission. >
Client
Certification of Testing and Processor Derivation:
I, the
undersigned, representing the Client, certify that the above
identified representative processor was tested on the described
configuration with the customized ACATS version described above,
including the code modifications and implementation-defined
substitution values that were used in the conformity assessment
leading to the certificate named in this Supplement, with
modifications described in this Supplement, and that the testing
results were the same as those obtained in that conformity
assessment, with exceptions as described in this Supplement. I
further declare that the Client knows of no deliberate deviations
from the Ada language standard (ANSI/ISO/IEC 8652:2012)
in the identified representative processor above. I further
certify that the above-identified representative processor and
configuration meets the definition of base, maintained, or rehosted
implementation (as described in the Operating Procedures for Ada
Conformity Assessment).
________________________________ ________________
<Name> - <Client> Date
<For
maintenance and rehosted implementation classes>
Maintenance Changes:
<Include
a brief description of the significant changes in the compiler in
deriving it from the compiler named in the ACAC. Include the type of
maintenance (Corrective, Perfective, or Adaptive) for each change
noted.>
ACATS Modifications:
<Describe
each change in the ACATS used in testing the reference processor, as
compared to the customized ACATS used in testing the processor named
in the certificate. Justification is required for any change other
than the use of different implementation-dependent substitution
values and the use of the current ACATS version. For tests different
solely because of the use of the current ACATS version, a list of the
test names is sufficient.>
Test Results
Differences:
<Describe
and justify each difference between test results produced in testing
the reference processor as compared to those produced in testing the
processor named in the certificate. For tests whose results differ
solely because of the use of the current ACATS version, a list of the
test names is sufficient. The actual results for those tests must be
submitted to the ACAL along with this supplement in a format
acceptable to the ACAL. The ACAL will attach a summary report of
those results here.>
ACRONYMS
This document contains a number of acronyms, whose meanings
are given in the following list:
ACAA
|
Ada
Conformity Assessment Authority
|
ACAC
|
Ada Conformity Assessment Certificate
|
ACAL
|
Ada Conformity Assessment Laboratory
|
ACATR
|
Ada Conformity Assessment Test Report
|
ACATS
|
Ada
Conformity Assessment Test Suite
|
AJPO
|
Ada
Joint Program Office
|
ANSI
|
American
National Standards Institute
|
ARA
|
Ada Resource Association
|
ARG
|
Ada Rapporteur Group
|
CPL
|
Certified
Processor List
|
DoC
|
Declaration
of Conformity
|
DoD
|
Department
of Defense
|
IEEE
|
Institute
of Electrical and Electronic Engineers
|
ISO
|
International
Organization for Standardization
|
SNA
|
Specialized
Needs Annex
|
WG9
|
Working
Group 9 (of ISO/IEC JTC1/SC22)
|
REFERENCES
This
document references the following publications:
- [ACATS UG]
- ACATS User's
Guide, version 3.0, December 2007. This document is updated
periodically; references in this document refer to the most recent
version unless otherwise noted.
- [Ada2012]
- ANSI/ISO/IEC
8652:2012 Ada 2012 Reference Manual, December
2012 (supersedes [Ada95]).
- [Ada95]
- ANSI/ISO/IEC
8652:1995 Ada 95 Reference Manual, January 1995 (supersedes
[Ada83]).
- [Ada83]
- American National
Standards Institute and United States Department of Defense:
ANSI/MIL-STD-1815A Reference Manual for The Ada Programming
Language, 1983 Note: This standard is identical with
ISO/8652:1987.
- [Amd1]
- ISO/IEC
8652:1995/AMD 1:2007 Programming languages -- Ada Amendment 1
(usually called Ada 2005)
- [ANSI/IEEE 90]
- American
National Standards Institute / Institute of Electrical and Electronic
Engineers, Inc., Standard 610.12-1990; “ANSI/IEEE Standard
Glossary of Software Engineering Terminology”.
- [ISO 74]
- International
Standards Organization: ISO 2382/I-1974 Data Processing -
Vocabulary - Section 01: Fundamental Terms.
- [ISO/IEC 91]
- International
Standards Organization: ISO/IEC, Guide 2, 6th
edition 1991 - General Terms and Their Definitions Concerning
Standardization and Related Activities.
- [ISO 99]
- ISO/IEC
18009:1999, Information Technology -- Programming Languages -- Ada:
Conformity Assessment of a Language Processor
- [TC1]
- ISO/IEC
8652:1995/COR.1:2001 Programming languages -- Ada Technical
Corrigendum 1