CVS difference for acats/docs/ug-611.htm
--- acats/docs/ug-611.htm 2016/05/27 03:52:20 1.1
+++ acats/docs/ug-611.htm 2016/06/30 05:29:50 1.2
@@ -39,7 +39,10 @@
<A HREF="UG-51.HTM">5.1</A>, <A HREF="UG-52.HTM">5.2</A>, and <A HREF="UG-53.HTM">5.3</A>.
It is particular important that Macro Defs customization <A HREF="UG-522.HTM">5.2.2</A>
is accomplished <I>before</I> generating any test summaries, as the summary
-program is unaware of the macro syntax.</div>
+program is unaware of the macro syntax. Also, do not use the grading
+tool on the support tests in the CZ directory, as some of these include
+intentional failure messages that the grading tool is not prepared to
+handle.</div>
<div class="Indented2NarrowHanging-Term">2.</div><div class="Indented2NarrowHanging-Body">
Compile the Grading and Test Summary Tools, as described in <A HREF="UG-613.HTM">6.1.3</A>.</div>
<div class="Indented2NarrowHanging-Term">3.</div><div class="Indented2NarrowHanging-Body">
@@ -57,18 +60,19 @@
ACATS, but it probably is more manageable to create several smaller event
traces for portions of the ACATS. One obvious way to do that is to create
a single event trace for each subdirectory that contains ACATS tests
-in its default delivery structure.</div>
+in its default delivery structure. (Such event traces can be combined
+later, if desired.)</div>
<div class="Indented2NarrowHanging-Term">4b.</div><div class="Indented2NarrowHanging-Body">
Create command scripts (as described in <A HREF="UG-54.HTM">5.4</A>)
to process the ACATS. Include in those use of the listing conversion
-tool to make event traces. The goto step 5.</div>
+tool to make event traces. Then go to step 5.</div>
<div class="Indented2NarrowHanging-Term">5.</div><div class="Indented2NarrowHanging-Body">
Create test summaries for each grading segment. If, for instance, you
will grade each directory individually, then you will need a test summary
file for each directory. These files can be generated by running the
Test Summary tool on each source file in the directory using a single
summary file as output. On most operating systems, this is easily accomplished
-with a script command. (Some possibilities are discussed in <A HREF="UG-615.HTM">6.1.5</A>.</div>
+with a script command. (Some possibilities are discussed in <A HREF="UG-615.HTM">6.1.5</A>).</div>
<div class="Indented2NarrowHanging-Term"></div><div class="Indented2NarrowHanging-Body">
The Test Summary Files will only need to be regenerated if the ACATS
tests change in some way (typically, when an ACATS Modification List
@@ -97,17 +101,18 @@
If the failure reason is a process failure or a missing compilation,
most likely there is a problem with the scripts that process the ACATS.
Make sure that the test files are compiled in the appropriate order and
-none are missing.</div>
+no test files are missing. A missing compilation might also mean that
+the test needs to be split. See item B.</div>
<div class="Indented4NarrowHanging-Term">B.</div><div class="Indented4NarrowHanging-Body">
If the failure reason is extra or missing errors, grade the test manually
(see <A HREF="UG-56.HTM">5.6</A>) to see if the problem is with the implementation
-or the Grading Tool being too strict about error positioning. If manual
+or the Grading Tool being too strict about error locations. If manual
grading indicates the test passed, add the test to your Manual Grading
Request file - again, see <A HREF="UG-64.HTM">6.4</A> (preferably with
-a comment explaining why it was added. Note that it is not necessary
+a comment explaining why it was added). Note that it is not necessary
to remove tests from this list: if the grading tool determines that the
test grades as Passed or Not Applicable, it will not request manual grading
-for the test.</div>
+for the test even if it appears in this list.</div>
<div class="Indented4NarrowHanging-Term"></div><div class="Indented4NarrowHanging-Body">
If the manual grading indicates that the test needs to be split, do the
following. First, add the test to your Manual Grading Request file -
@@ -121,7 +126,9 @@
<div class="Indented4NarrowHanging-Term">C.</div><div class="Indented4NarrowHanging-Body">
For other failure reasons, most likely the implementation is at fault.
Fixing the implementation is likely the only way to meaningfully change
-the result. </div>
+the result. In the unlikely event that there is a problem with a test,
+the procedure for test challenges are outlined in <A HREF="UG-572.HTM">5.7.2</A>.
+</div>
<div class="Indented2NarrowHanging-Term"></div><div class="Indented2NarrowHanging-Body">
You'll also need to handle any special handling tests, including any
tests that require manual grading. (This is one good reason to keep the
@@ -130,7 +137,9 @@
Then return to step 7 and repeat the test run. </div>
<div class="Normal">Using this procedure, the vast majority of tests
will not require hand grading. Future ACATS updates may improve tests
-that are particulary difficult to grade automatically.</div>
+that are particulary difficult to grade automatically. <SPAN STYLE="color: rgb(153,0,0)">The
+ACAA is interested in which tests need manual grading for your implementation
+- see <A HREF="UG-64.HTM">6.4</A>.</SPAN></div>
<HR>
<div style="margin-top: 0.0em; margin-bottom: 0.6em"><A HREF="UG-TOC.HTM"><IMG SRC="CONT.GIF" ALT="Contents" BORDER=0></A>
Questions? Ask the ACAA Technical Agent