Version 1.10 of ais/ai-00283.txt

Unformatted version of ais/ai-00283.txt version 1.10
Other versions for file ais/ai-00283.txt

!standard A.12.01 (28)          03-02-18 AI95-00283/02
!standard A.08.02 (03)
!standard A.08.02 (10)
!standard A.08.02 (16)
!standard A.08 (01)
!class binding interpretation 02-06-10
!status Amendment 200Y 03-02-18
!status ARG Approved 5-0-3 03-02-09
!status work item 02-06-10
!status received 01-12-14
!qualifier Clarification
!priority Low
!difficulty Medium
!subject Truncation of stream files by Close and Reset
!summary
Stream files are not truncated by Close or Reset.
!question
For Stream_IO, the standard indicates that Reset truncates a file after the last write position and it is painful (and sometimes erroneous) to do a dummy Write at the end of the file before a Reset on this file from Out_File mode to In_File mode. There is similar behavior for Close. Is this behavior intended? (No.)
!recommendation
Stream_IO files are not sequential files, and are not truncated on Close and Reset as described in A.8.2(10) and A.8.2(16).
!wording
Italicize "stream file" in A.8(1) to signify a definition.
Add "stream" to the list of kinds of input-output with Out_File as the default parameter in A.8.2(3).
Replace A.12.2(28) by:
The subprograms given in subclause A.8.2 for the control of external files (Create, Open, Close, Delete, Reset, Mode, Name, Form, and Is_Open) are available for stream files.
The End_of_File function:
* Propagates Mode_Error if the mode of the file is not In_File;
* If positioning is supported for the given external file, the function returns True if the current index exceeds the size of the external file; otherwise it returns False;
* Otherwise, the function returns True if no more elements can be read from the given file; otherwise it returns False.
!discussion
A compiler survey shows that most Ada 95 compilers truncate stream files on Open. This is much cheaper on Unix systems than truncating on Close as is described in the standard.
However, truncating on Open means that all of the data in the file is discarded. If the application opens a stream file, positions it, writes a few items, positions it to the end, and closes it, all of the data not written by this set of operations should remain intact. Thus, the behavior of these compilers is incorrect.
We considered requiring truncation as described in the RM. However, this is expensive on Unix systems (which do not have a truncate file operation), and no compiler currently implements it. Moreover, this is not particularly useful functionality. The behavior of existing compilers is easily accomplished by a Delete followed by Create sequence.
The AARM indicates that stream files are not sequential files. This implies that behavior specific to sequential files (such as truncation) is not intended. On the other hand, if stream files are not sequential files for the purposes of A.8.2, what are they? The standard specifically does not define "stream file" as a term. And the equivalence in A.12.1(28) specifies "Sequential_IO", so arguably the behavior of sequential files applies.
Thus we correct the oversight of defining the term "stream file", and we remove the reference to Sequential_IO from the specification of these routines. The wording used is similar to that used in the definition of Text_IO (A.10.2(1)). Having done so, the specific requirements in A.8.2 no longer apply to stream files. We also do a minor repair of the wording to describe the default mode for Create for stream files. Otherwise, no changes to A.8.2 are necessary.
Finally, we define the behavior of End_of_File directly in A.12.1. Since stream files can be positioned, the equivalence given in the RM means that the behavior of End_of_File is unspecified if the file has been positioned after reaching the end, or if it positioned beyond the end. Clearly, that is not intended.
!corrigendum A.8(01)
Replace the paragraph:
Two kinds of access to external files are defined in this subclause: sequential access and direct access. The corresponding file types and the associated operations are provided by the generic packages Sequential_IO and Direct_IO. A file object to be used for sequential access is called a sequential file, and one to be used for direct access is called a direct file. Access to stream files is described in A.12.1.
by:
Two kinds of access to external files are defined in this subclause: sequential access and direct access. The corresponding file types and the associated operations are provided by the generic packages Sequential_IO and Direct_IO. A file object to be used for sequential access is called a sequential file, and one to be used for direct access is called a direct file. Access to stream files is described in A.12.1.
!corrigendum A.8.2(03)
Replace the paragraph:
Establishes a new external file, with the given name and form, and associates this external file with the given file. The given file is left open. The current mode of the given file is set to the given access mode. The default access mode is the mode Out_File for sequential and text input-output; it is the mode Inout_File for direct input-output. For direct access, the size of the created file is implementation defined.
by:
Establishes a new external file, with the given name and form, and associates this external file with the given file. The given file is left open. The current mode of the given file is set to the given access mode. The default access mode is the mode Out_File for sequential, stream, and text input-output; it is the mode Inout_File for direct input-output. For direct access, the size of the created file is implementation defined.
!corrigendum A.12.1(28)
Replace the paragraph:
The subprograms Create, Open, Close, Delete, Reset, Mode, Name, Form, Is_Open, and End_of_File have the same effect as the corresponding subprograms in Sequential_IO (see A.8.2).
by:
The subprograms given in subclause A.8.2 for the control of external files (Create, Open, Close, Delete, Reset, Mode, Name, Form, and Is_Open) are available for stream files.
The End_of_File function:
!ACATS test
An ACATS test needs to be created to insure that stream files are not truncated except on Create.
!appendix

!topic Inout_File mode missing for Ada.Streams.Stream_Io
!reference RM95-A.12.1(6)
!from Bernard Maudry
!discussion

Would it be possible to add the "Inout_File" mode to
Ada.Streams.Stream_Io.File_Type as Reset truncates a file after the last write
position and it is painful (and sometimes erroneous) to do a dummy Write at the
end of the file before a Reset on this file from Out_File mode to In_File mode.

*************************************************************

From: Randy Brukardt
Date: Friday, December 14, 2001  5:08 PM

This seems like a clear bug in the standard (as modified by the
Corrigendum). We certainly don't need to add a new mode to fix it.

In the discussions for the Corrigendum, we decided that positionable
Stream_IO files acted essentially as direct files. Only non-positionable
files act like sequential files. We thought that we'd fixed the wording to
make that happen, but apparently we missed something.

To actually implement that rule (A.8.2(16), the sentence starting with "In
addition,", which is included by A.12.1(28)) as written, an implementation
would have to keep track of the highest position written since the last
Create/Open/Reset. For a non-positionable file, that is the same as the
current position, but for a positionable file, it could be anywhere. I don't
think it was intended that there was a requirement to save/check the write
location after every write.

So, I think that that sentence in A.8.2(16) is intended to apply only to
non-positionable stream files. This is the only rule in A.8.2 that applies
only to sequential files, so I don't think that there are any more examples
of this particular problem lurking.

There may be good arguments for adding an Inout_File mode to Stream_IO, but
I don't think this is one.

*************************************************************

From: Randy Brukardt
Date: Friday, December 14, 2001  5:15 PM

Oops, of course Close also has this behavior.

And that also is a problem. If you are representing a database index as a
stream file, if you insert a record (and write a couple of items in the
middle), you certainly don't want to have to write at the end so that the
silly system doesn't truncate the file.

Perhaps we need a Truncate operation in Stream_IO for the times when you do
want to truncate?

*************************************************************

From: Pascal Leroy
Date: Thursday, December 20, 2001  4:50 AM

This issue looks like a red herring to me.

If you want to change the mode without truncating, use Set_Mode, not Reset.  It
seems that the only difference between the two is that Set_Mode does not
truncate, but Reset does.

If you want to close the file without truncating, use Set_Mode to change the
mode to In_File before closing.  OK, this means that you need to perform two
operations (Set_Mode + Close).  But the alternative model suggested by Randy
(where Close doesn't truncate and we have an extra Truncate operation) also
requires two operations in the case where you want to truncate-and-close.

Actually, I find the current model (as stated by RM95+TC1) rather natural and
certainly consistent.  But evidently it should be tested by the ACATS since
it's an area where differences between implementations could cause lots of
trouble.

*************************************************************

From: Randy Brukardt
Date: Friday, December 28, 2001  5:23 PM

Since compatibility with existing behavior will be an important part of the
resolution of this issue (truncation of Stream_IO files with mode Out_File),
I've created a test and distributed it to implementors. (If anyone out there
would like to try it themselves, just drop me a line and I'd be happy to send
it. I'm not going to post it to the list as it is 535 lines long.) I'll post a
summary of results in a week or so.

I've run the test on 4 popular compilers for Windows, and I have a few
preliminary conclusions. (I'm writing this up now, rather than waiting for
additional results, so I don't forget all of this over the long holiday
weekend. I expect to have fewer brain cells next Wednesday... :-)

-- Most implementors do in fact expect to have to truncate Stream_IO files on
   Close. Apparently, I'm the only implementor who finds this bizarre.

-- None of the compilers I tested supports Pascal's suggested "workaround",
   including Pascal's compiler. :-) That suggests if this topic is handled with
   a confirmation AI, a strong ACATS test is needed to insure that the
   workaround actually works.

-- When positioning is added to the mix, the results vary wildly. I'll need
   more results to form any opinion here. It appears to be very dangerous to
   open a Stream_IO file with mode Out_File and write only a few elements
   before closing with current compilers.

-- Stream_IO is still buggy: several implementation raise End_Error when
   Stream_IO.Read reads past the end of a file; one implementation deletes the
   entire contents of the file when it is opened with mode Out_File (even if it
   is Set_Moded to In_File and closed without other operations).


It is interesting to note that although implementors mostly believe that
truncation is required on Close and Reset, it is possible to read the RM so
this is not a requirement.

A.12.1(28) says that "... Close, ... Reset, ... has the same effect as the
corresponding subprograms in Sequential_IO."  A.8.2(10) says, "In addition, for
sequential files, if the file being closed has mode Out_File or Append_File,
then the last element written since the most recent open or reset is the last
element that can be read from the file." (The language for Reset in A.8.2(16)
is almost identical.) A.8(1) says that a file used for sequential access is a
sequential file. The AARM also says specificially that stream files are not
sequential files (see A.8(2.a-b)).

So, we can argue both ways:
Since Sequential_IO only works on sequential files, the truncation requirement
also applies to Stream_IO by a literal reading of A.12.1(28).

Alternatively, since the description of Close and Reset say that truncation
only applies to sequential files, and the clear language designer's intent is
that stream files are not sequential files, then the truncation does not apply
to stream files.

Both interpretations seem valid to me, but of course are completely different.


A common implementation of truncation for sequential files (in Sequential_IO)
is to empty the file when it is opened. This is more efficient than truncating
the file on Close (most sequential files are Created, not Opened with Out_File,
and usually no truncation is needed), and it works even on operating systems
that do not support truncating files (because the file can be deleted on Open
to provide the correct semantics).

However, a direct copying of this implementation to stream files leads to
disaster. For instance, if Stream_IO is used to maintain a database, updating a
single record by opening the file with mode Out_File, writing the record, and
closing the file would discard the contents of the entire database! At least
one existing implementation appears to have made this mistake.

It is also worth noting that a correct implementation of Stream_IO truncation
on an operating system that does not support file truncation would be very
expensive (requiring an under-the-covers copy of the file). I'm not sure that
this is a real issue (the OSes that I used that didn't support truncation are
long obsolete), but I think we need to consider it.


Finally, exactly how the truncation is done when positioning is involved is
unclear. Recall the wording of A.8.2(10) says that the "last element written"
is the last one in the file after a Close. Does this mean the "most recent
element written" or "the element with the largest Index written"? A natural
English reading would imply the former; however implementations seem to favor
the latter.


In the absence of compatibility issues, I'd suggest reading the RM to say that
truncation is not required. However, given the trend of implementations, that
probably would be a bad idea. Thus we're going to have to figure out how
something intended solely for sequential files (truncation) applies to files
with positioning.

Sigh. Stream_IO is still a mess.

*************************************************************

From: Randy Brukardt
Date: Monday, February  4, 2002  7:56 PM

You may recall that back in December, I created and distributed a test to
determine the current behavior of Ada compilers when closing or resetting a
stream file with mode Out_File. Here is the promised summary of the
results -- they're pretty grim in that compilers are very inconsistent, even
compilers from the same vendor.

I received results from most of the compiler vendors, but I didn't receive
results from ICC, despite repeated requests. All other compilers are listed in
alphabetical order of the product name.

Aversomething AdaMagic (latest development version):

   -- Truncates sequential-access stream files on close/reset: No.
   -- Truncates stream files opened with In_File mode, then Reset to Out_File: No.
   -- Supports Set_Mode "workaround" to avoid truncation: Yes.
   -- Truncates positional-access stream files on close/reset: No.
   -- Discards data on Open for Out_File for positional-access stream files: No.
   -- No Truncation.
   -- File opened with mode Append_File truncates after positioning and writing in middle:
		Positioning not allowed.

Rational Apex for Sun Solaris (latest development version):

   -- Truncates sequential-access stream files on close/reset: Yes.
   -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes.
   -- Supports Set_Mode "workaround" to avoid truncation: No.
   -- Truncates positional-access stream files on close/reset: Yes.
   -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad).
   -- Truncation is at the largest position element written.
   -- File opened with mode Append_File truncates after positioning and writing in middle: No.
   --    (This implementation appears to violate AI-85 by allowing positioning,
   --	  but ignoring the result and writing at the end anyway.)

Rational Apex for Windows NT (3.2.0c):

   -- Truncates sequential-access stream files on close/reset: Yes.
   -- Truncates stream files opened with In_File mode, then Reset to Out_File: No.
   -- Supports Set_Mode "workaround" to avoid truncation: No.
   -- Truncates positional-access stream files on close/reset: Yes.
   -- Discards data on Open for Out_File for positional-access stream files: No.
   -- Truncation is at the largest position element written.
   -- File opened with mode Append_File truncates after positioning and writing in middle: No.
   (Implementation incorrectly raises End_Error when reading past the end of the file
    with Stream_IO.Read)

GNAT for Sun Solaris (latest development version):

   -- Truncates sequential-access stream files on close/reset: Yes.
   -- Truncates stream files opened with In_File mode, then Reset to Out_File: No.
   -- Supports Set_Mode "workaround" to avoid truncation: No.
   -- Truncates positional-access stream files on close/reset: Yes.
   -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad).
   -- Truncation is at the largest position element written.
   -- File opened with mode Append_File truncates after positioning and writing in middle: No.

GNAT for Windows NT (3.14a1):

   -- Truncates sequential-access stream files on close/reset: Yes.
   -- Truncates stream files opened with In_File mode, then Reset to Out_File: No.
   -- Supports Set_Mode "workaround" to avoid truncation: No.
   -- Truncates positional-access stream files on close/reset: Yes.
   -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad).
   -- Truncation is at the largest position element written.
   -- File opened with mode Append_File truncates after positioning and writing in middle: No.

Janus/Ada for Windows NT (3.1.2 development)

   -- Truncates sequential-access stream files on close/reset: No.
   -- Truncates stream files opened with In_File mode, then Reset to Out_File: No.
   -- Supports Set_Mode "workaround" to avoid truncation: Yes.
   -- Truncates positional-access stream files on close/reset: No.
   -- Discards data on Open for Out_File for positional-access stream files: No.
   -- No Truncation.
   -- File opened with mode Append_File truncates after positioning and writing in middle: No.

Aonix ObjectAda for Sun Solaris (version 7.2.1)

   -- Truncates sequential-access stream files on close/reset: Yes.
   -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes.
   -- Supports Set_Mode "workaround" to avoid truncation: No.
   -- Truncates positional-access stream files on close/reset: Yes (bizarre behavior: truncates
         to the number of elements written, ignoring the positioning
         --  except for the basic case of immediate Close after writing, which
         works as expected)).
   -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad).
   -- Can't tell truncation point (due to bizarre behavior).
   -- File opened with mode Append_File truncates after positioning and writing in middle: No.
   --    (This implementation appears to violate AI-85 by allowing positioning,
   --     but ignoring the result and writing at the end anyway.)

Aonix ObjectAda for Windows NT (version 7.2.1)

   -- Truncates sequential-access stream files on close/reset: Yes.
   -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes.
   -- Supports Set_Mode "workaround" to avoid truncation: No.
   -- Truncates positional-access stream files on close/reset: Yes (bizarre
         behavior: truncates to the number of elements written, ignoring the positioning).
   -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad).
   -- Can't tell truncation point (due to bizarre behavior).
   -- File opened with mode Append_File truncates after positioning and writing in middle: Yes.

OCS PowerAda/6000 Compiler (Version 4.3)

   -- Truncates sequential-access stream files on close/reset: Yes.
   -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes.
   -- Supports Set_Mode "workaround" to avoid truncation: No.
         (Set_Mode unexpectedly raised Use_Error at this point [with an Exception_Message
          of "Set_Mode not yet supported"], which terminated the test.)

DDCI SCORE 2.2 Solaris native:

    -- Truncates sequential-access stream files on close/reset: Yes.
    -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes.
    -- Supports Set_Mode "workaround" to avoid truncation: No.
    -- Truncates positional-access stream files on close/reset: Yes.
    -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad).
    -- Truncation is at the largest position element written.
    -- File opened with mode Append_File truncates after positioning and writing in middle: No.

The main conclusion that can be drawn from this is that there is no
consistency, even between the same vendor's implementations. Probably some
sort of truncation for sequential-access files should be required (since
most vendors do it, even though neither the AARM nor Tucker support the
requirement), but beyond that it is very murky.

OTOH, the large number of implementations that discard data when a file is
Opened with mode Out_File (even if later access is simply an update of a few
elements) is horrifying. It means that Stream_IO cannot be used in
applications requiring in-place updating -- an application that was clearly
intended by the language designers.

Also, hardly any of the implementations support the workaround suggested to
avoid truncation. So, if a user want to avoid this behavior (for in-place
updating, for example), they're out of luck with most implementations.

We'll have to discuss the best approach at the upcoming meeting.

*************************************************************

From Minutes of ARG meeting 15, February 2002, Cupertino CA

Randy describes the problem and the result of his compiler survey (most
compilers use the same algorithm for truncation as for Sequential_IO: truncate
on open for Out_File).

The current behavior of most compilers is a bug: it causes any data already in
the file to be discarded, even if it is not going to be overwritten and even if
the user has taken precautions to move the current index before closing. The
group agrees that this is wrong.

Tucker comments that if you don't truncate on open, it is expensive to truncate
at close on Unix systems. There are probably other systems with this property.

Pascal notes that if we don't truncate, we need to provide a way for the user
to do it. Tucker notes that no existing implementation truncates on close, so
we only need to provide what is currently available. Delete followed by Create
would accomplish that.

Therefore, the group settles on no truncation of Stream_IO files, except of
course by Create in In_File or Out_File modes. This should be accomplished by
fixing the equivalence so it clearly doesn't imply truncation.

Approve intent of the AI: 3-0-3

Randy will write the wording.

*************************************************************


Questions? Ask the ACAA Technical Agent