ACATS Modification List for ACATS 4.1 Version 4.1B November 18, 2016 What is this? This is the ACATS modification list for ACATS 4.1. This document includes test corrections, test removals, and new tests for ACATS 4.1. ACATS 4.1 is the current test suite for Ada, covering the third edition of Ada, ISO/IEC 8652:2012 (usually known as "Ada 2012"). ACATS Modification List version 3.1N for ACATS 3.1 is the current list for that test suite. ACATS 3.1 should be used for testing implementations that support Amendment 1 (ISO/IEC 8652:1995/AMD 1:2007, often known as "Ada 2005"), and will continue to be maintained as necessary for the foreseeable future. ACATS Modification List version 2.6D for ACATS 2.6 is the current list for that test suite. ACATS 2.6 should be used for testing implementations of the second edition of Ada, ISO/IEC 8652:1995 (usually known as "Ada 95"); it also will continue to be maintained as necessary for the foreseeable future. Changes from the last list: A number of tests changed status on July 1st and October 1st, this list reflects those changes. New test B611014 checks that prefixes of Old attribute references that are potentially unevaluated must statically denote an object. New test B611015 checks that prefixes of Old attributes cannot have a limited type. New test B611016 checks that Pre'Class and Post'Class cannot be given on entries and protected operations even if the type is tagged. New test B611017 checks that Pre'Class cannot be given on an overriding subprogram if none of the original subprogram give Pre'Class. New test B611018 checks 6.1.1(10-16/3). New test B611019 checks 6.1.1(17/3). New tests posted between October 1, 2016 and December 31, 2016 will become required for conformity assessments on July 1, 2017. New tests posted between July 1, 2016 and September 30, 2016 will become required for conformity assessments on April 1, 2017. New tests posted between April 1, 2016 and June 30, 2016 will become required for conformity assessments on January 1, 2017. New tests posted before April 1, 2016 are required for conformity assessments. The effective dates of tests are marked in the lists. Questions about this document? Contact the ACAA Technical Agent at agent@ada-auth.org. ACATS 4.1 includes optional Test Grading tools ACATS 4.1 introduces new test grading tools. When the ACATS was designed (as the ACVC in the early 1980s), the intention always was that running it would give a simple and clear Pass or Fail result. However, grading of tests (particularly of B and L Tests) is somewhat subjective and very time-consuming. (Test grading typically involved poring over compiler listings of the entire ACATS with a large highlighter.) The subjectivity was acceptable in the days of common formal testing, as the formal testers would enforce a common grading standard (even if no one could describe what that was), but in an age of self-testing, it reduces the value of the ACATS. As such, we're introducing a long-overdue set of automated grading tools. These tools are intended to: * Greatly reduce the effort to grade an ACATS test run (and, hopefully to reduce the cost of formal testing to make it a more practical option); * Ensure that all users of the ACATS grade tests in the same way, thus making it more valid to compare ACATS results between different implementations and different testers; * Give implementers certainty as to whether a particular test result is passing (whether the location of an error is “close enough” to the canonical position); * Be able to detect process errors (especially errors of omission) – that is, tests improperly run (for instance, files compiled in the wrong order where the ACATS requires an order), or not run at all (if one is checking off lines in a listing, it's hard to notice that some test or portion of a test is missing). The tools also will optionally give feedback to implementers about quality of error message positioning and of error cascades (does one error generate multiple messages?). (Since “goodness of an implementation” is not something that the ACATS tests for, this is not an official goal for the tools.) This is an ambitious set of goals. It's unlikely that they'll be completely met by the initial versions of the tools. As such, their use will be completely optional (but encouraged!) for ACATS 4.1. The tools will be updated periodically by the normal process for updating the ACATS – an ACATS Modification List like this one. Details on the operation of the tools can be found in the ACATS 4.1 documentation (that documentation can be found at http://www.ada-auth.org/acats-files/4.1/docs/UG-6.HTM). As always, if you have questions, contact the ACAA Technical Agent at agent@ada-auth.org. Notice: The ACAA has created and maintains the Ada Conformity Assessment Test Suite for the purpose of conformity assessments conducted in accordance with the International Standard ISO/IEC 18009 - Ada: Conformity assessment of a language processor. This test suite should not be used to make claims of conformance unless used in accordance with ISO/IEC 18009 and any applicable ACAA procedures. List explanation: Test numbers given are for the basic test. Categories are “Withdrawn” (the test should not be processed for Ada conformity assessments); “Allowed Modification” (either the original test or the modified one should be processed for Ada conformity assessments); “Modified” (the modified test must be processed for Ada conformity assessments); “Pending New” (the test does not need to be processed for Ada conformity assessments, but will be added in the future); “New” (the test is processed for Ada conformity assessments). ACATS version labels are provided with all new and modified tests. Use this label to retrieve the test from the ACATS VCS. See “How to get new and modified tests”. In citations, line numbers are given relative to the start of a test file in the format of '@'; if a test program comprises more than one file, then the particular file will be indicated by "fn", where "n" is the test-program name's file-designator numeral. E.g., "B38103E : @f1-31, ... " denotes test file b38103e1.ada. In cases where there are two or more rationales for different errors in a test program, these different rationales with their respective line citations are listed on separate lines. References to the Ada standards are given in the format: .[.]: These references are to the Ada standard as available at http://www.adaic.org/resources/add-contentstandards/12rm/html/RM-TOC.html. Main list: ===== Core Language ===== Name Category Version label (effective date; old version label) Short description. B38003D [Modified] VCS Label: A4_1A Corrected error tags @70,76,82,88 so “optional” and “error” are on the same line, as required by the test standards. C3A0030 [New] VCS Label: A4_1A Check that an access discriminant is only null excluding if it includes a null exclusion. B3A1001 [Modified] VCS Label: A4_1A Split Pack6 so possible error sets don't intersect; fixed overlong lines. B3A1002 [Modified] VCS Label: A4_1A Split Pack6 so possible error sets don't intersect; fixed overlong lines. B455002 [Modified] VCS Label: A4_1A Corrected casing of error tags and fixed overlong lines. C480001 [New] VCS Label: A4_1A Check that an initialized allocator for a class-wide type does an accessibility check on the tag of the initializing object. C52101A [Modified] VCS Label: A4_1A Corrected comment @49 not to have the form of an error tag. B611001 [New] VCS Label: A4_1A Check that Pre and Post are only allowed on subprograms and entries (and not on subprogram instances). B611002 [New] VCS Label: A4_1A Check that Pre and Post are not allowed on subprograms acting as completions. B611003 [New] VCS Label: A4_1A Check that Pre'Class and Post'Class are only allowed on primitive subprograms of tagged types. (Normal tagged types and interfaces.) B611004 [New] VCS Label: A4_1A Check that Pre'Class and Post'Class are not allowed on subprograms acting as completions. B611005 [New] VCS Label: A4_1A Check that Pre and Post are not allowed on abstract subprograms and null procedures. B611006 [New] VCS Label: A4_1A Check that the restrictions on Pre'Class and Post'Class expressions added by AI12-0113-1 are enforced. B611007 [New] VCS Label: A4_1A Check that Pre'Class and Post'Class are not allowed on entries and protected subprograms of untagged task and protected types. B611008 [New] VCS Label: A4_1A Check various rules about the use of the Result attribute. B611009 [New] VCS Label: A4_1A Check that F'Result is not allowed in the body of F. B611010 [New] VCS Label: A4_1A Check that the Old attribute is not allowed in a subprogram or entry body, or in an accept statement. B611011 [New] VCS Label: A4_1A Check the Old attribute cannot be used within a precondition expression nor in a generic specification outside of a postcondition expression. B611012 [New] VCS Label: A4_1A Check that the prefix of an Old attribute cannot contain the Result attribute, another Old attribute, nor the index of a quantified expression. B611013 [New] VCS Label: A4_1A For a discrete X, check that X'Old has the nominal subtype of X. B611014 [Pending New] New VCS Label: A4_1B Required after 2017-07-01 If an Old attribute reference occurs in at a place where it is potentially unevaluated, check that the prefix must statically denote an object. B611015 [Pending New] New VCS Label: A4_1B Required after 2017-07-01 Check that the prefix of an Old attribute cannot have a limited type. B611016 [Pending New] New VCS Label: A4_1B Required after 2017-07-01 Check that tagged task and tagged protected types do not allow Pre'Class and Post'Class, nor are they allow on operations of those types. B611017 [Pending New] New VCS Label: A4_1B Required after 2017-07-01 Check that Pre'Class cannot be specified for an overriding of a subprogram that does not specify Pre'Class. B611018 [Pending New] New VCS Label: A4_1B Required after 2017-07-01 Check 6.1.1(10-16/3). B611019 [Pending New] New VCS Label: A4_1B Required after 2017-07-01 Check 6.1.1(17/3). C611A01 [New] VCS Label: A4_1A Check that preconditions and postconditions where both specific and class-wide contracts apply are checked properly. C611A02 [New] VCS Label: A4_1A Check that the correct expressions are evaluated for specific preconditions and postconditions, and that they are evaluated at the correct point. C611A03 [New] VCS Label: A4_1A Check that the correct expressions are evaluated for class-wide preconditions and postconditions, and that they are evaluated at the correct point. C611B01 [New] VCS Label: A4_1A Check that X'Old has the tag of X and not the tag of the nominal subtype of X (for Post). C611B02 [New] VCS Label: A4_1A Check that X'Old has the tag of X and not the tag of the nominal subtype of X (for Post'Class). B62001C [Modified] VCS Label: A4_1A Changed error @35 to be optional, as the syntax error @33 is required (and the comment that only one error need be detected is bogus). B62001D [Modified] VCS Label: A4_1A Changed error @41 to be optional, as the syntax error @33 is required (and the comment that only one error need be detected is bogus). C650B04 [New] VCS Label: A4_1A Check that the result of a function that returns a class-wide tagged type does an accessibility check on the tag of the result object when the result expression is class-wide. B740003 [Modified] VCS Label: A4_1A Modified criteria @75 so that it doesn't look like an error tag to the grading tools; also fixed an overlong line. B750A07 [New] VCS Label: A4_1A Check that only legal expressions are allowed in the return expression of an expression function that returns a limited type. B83F02C [Modified] VCS Label: A4_1A Updated many optional error markers to match ACATS standards. B95007B [Modified] VCS Label: A4_1A Fixed error markers @62,66,70 to match ACATS standards. BB42001 [New] VCS Label: A4_1A Check that the arguments of a pragma Assert have to have the correct types. CB42001 [New] VCS Label: A4_1A Check the usual operation of pragma Assert, including the optional message. BB42002 [New] New VCS Label: A4_1A Check that the arguments of a pragma Assert cannot be reordered even if named notation is used. CB42002 [New] New VCS Label: A4_1A Check that if the assertion policy is Ignore, none of the expressions of a pragma Assert are evaluated and no exception is raised by the pragma. BB42003 [New] VCS Label: A4_1A Check that a pragma Assertion_Policy is illegal if the policy is not recognized. CB42003 [New] VCS Label: A4_1A Check that procedure Ada.Assertions.Assert evaluates its arguments and raises Assertion_Error regardless of the assertion policy. CD11001 [New] VCS Label: A4_1A Check that appropriate entities are visible in an aspect specification. BD4003A [Modified] VCS Label: A4_1A Moved error marker from @66 to @67 so it is on the incorrect construct. CXB3023 [Pending New] New VCS Label: A4_1A Required after 2017-01-01 Check that a value of an unchecked union type (defined with pragma Unchecked_Union) can be passed to a C function. CXB3024 [Pending New] New VCS Label: A4_1A Required after 2017-01-01 Check that a value of an unchecked union type (defined with aspect Unchecked_Union) can be passed to a C function. ===== Annex C ===== [None.] ===== Annex D ===== [None.] ===== Annex E ===== [None.] ===== Annex F ===== [None.] ===== Annex G ===== [None.] ===== Annex H ===== [None.] ===== Support ===== F611A00 [New] VCS Label: A4_1A Foundation for tests C611A01, C611A02, and C611A03. F611B00 [New] VCS Label: A4_1A Foundation for tests C611B01 and C611B02. How to get new and modified tests: The official home of the ACATS test suite is the ACAA web site. The site can be accessed at www.ada-auth.org. The complete set of official new and modified tests are available in the Mod_4_1B.Zip and Tar files on the ACAA web site. Individual new and modified tests can be retrieved from the ACATS VCS. Simply navigate to the appropriate directory (folder), and select the test you want. Select the revision you want (usually the most recent) and it will be retrieved for you. Tests are stored in directories that correspond to the test’s prefix. For instance, the test CD92001.A will be found in the CD directory. You can also browse the differences between versions of a test. [In case of a conflict, the files in the ACATS VCS should be considered the 'official' versions.] If you cannot get the files from the official site or you cannot get the files in a format appropriate for your processor, please contact your ACAL. The baseline ACATS 4.1 is available in various formats on the ACAA web site. Summary lists: Withdrawn Tests (Refer to the main list for details) None yet. Allowed Modification Tests (Refer to the main list for details) None currently. Modified Tests (Refer to the main list for details) B38003D B3A1001 B3A1002 B455002 B62001C B62001D B740003 B83F02C B95007B BD4003A C52101A New Tests (Refer to the main list for details) B611001 B611002 B611003 B611004 B611005 B611006 B611007 B611008 B611009 B611010 B611011 B611012 B611013 B750A07 BB42001 BB42002 BB42003 C3A0030 C480001 C611A01 C611A02 C611A03 C611B01 C611B02 C650B04 CB42001 CB42002 CB42003 CD11001 F611A00 F611B00 Pending New Tests (Refer to the main list for details) B611014 2017-07-01 B611015 2017-07-01 B611016 2017-07-01 B611017 2017-07-01 B611018 2017-07-01 B611019 2017-07-01 CXB3023 2017-01-01 CXB3024 2017-01-01