!standard A.12.01 (28) 05-08-11 AI95-00283/04 !standard A.08.02 (03) !standard A.08.02 (10) !standard A.08.02 (16) !standard A.08 (01) !class binding interpretation 02-06-10 !status Amendment 200Y 03-02-18 !status WG9 approved 03-06-20 !status ARG Approved 5-0-3 03-02-09 !status work item 02-06-10 !status received 01-12-14 !qualifier Error !priority Low !difficulty Medium !subject Truncation of stream files by Close and Reset !summary Stream files are not truncated by Close or Reset. !question For Stream_IO, the standard indicates that Reset truncates a file after the last write position and it is painful (and sometimes erroneous) to do a dummy Write at the end of the file before a Reset on this file from Out_File mode to In_File mode. There is similar behavior for Close. Is this behavior intended? (No.) !recommendation Stream_IO files are not sequential files, and are not truncated on Close and Reset as described for sequential files in A.8.2(10) and A.8.2(16). !wording Italicize "stream file" in A.8(1) to signify a definition. Add "stream" to the list of kinds of input-output with Out_File as the default parameter in A.8.2(3). Replace A.12.2(28) by: The subprograms given in subclause A.8.2 for the control of external files (Create, Open, Close, Delete, Reset, Mode, Name, Form, and Is_Open) are available for stream files. The End_Of_File function: * Propagates Mode_Error if the mode of the file is not In_File; * If positioning is supported for the given external file, the function returns True if the current index exceeds the size of the external file; otherwise it returns False; * If positioning is not supported for the given external file, the function returns True if no more elements can be read from the given file; otherwise it returns False. !discussion A compiler survey shows that most Ada 95 compilers truncate stream files on Open. This is much cheaper on Unix systems than truncating on Close as is described in the standard. However, truncating on Open means that all of the data in the file is discarded. If the application opens a stream file, positions it, writes a few items, positions it to the end, and closes it, all of the data not written by this set of operations should remain intact. Thus, the behavior of these compilers is incorrect. We considered requiring truncation as described in the standard. However, this is expensive on Unix systems (which do not have a truncate file operation), and no compiler currently implements it. Moreover, this is not particularly useful functionality. When it is needed, the behavior of existing compilers is easily accomplished by a Delete followed by Create sequence, so a rule like the existing one is not necessary. The AARM indicates that stream files are not sequential files. This implies that behavior specific to sequential files (such as truncation) is not intended. On the other hand, if stream files are not sequential files for the purposes of A.8.2, what are they? The standard specifically does not define "stream file" as a term. And the equivalence in A.12.1(28) specifies "Sequential_IO", so arguably the behavior of sequential files applies. Thus we correct the oversight by defining the term "stream file", and we remove the reference to Sequential_IO from the specification of these routines. The wording used is similar to that used in the definition of Text_IO (A.10.2(1)). Having done so, the specific requirements in A.8.2 no longer apply to stream files. We also do a minor repair of the wording to describe the default mode for Create for stream files. Otherwise, no changes to A.8.2 are necessary. Finally, we define the behavior of End_of_File directly in A.12.1. Since stream files can be positioned, the equivalence given in the RM means that the behavior of End_of_File is unspecified if the file has been positioned after reaching the end, or if it is positioned beyond the end. Clearly, that is not intended. !corrigendum A.8(01) @drepl Two kinds of access to external files are defined in this subclause: @i and @i. The corresponding file types and the associated operations are provided by the generic packages Sequential_IO and Direct_IO. A file object to be used for sequential access is called a @i, and one to be used for direct access is called a @i. Access to stream files is described in A.12.1. @dby Two kinds of access to external files are defined in this subclause: @i and @i. The corresponding file types and the associated operations are provided by the generic packages Sequential_IO and Direct_IO. A file object to be used for sequential access is called a @i, and one to be used for direct access is called a @i. Access to @is is described in A.12.1. !corrigendum A.8.2(03) @drepl Establishes a new external file, with the given name and form, and associates this external file with the given file. The given file is left open. The current mode of the given file is set to the given access mode. The default access mode is the mode Out_File for sequential and text input-output; it is the mode Inout_File for direct input-output. For direct access, the size of the created file is implementation defined. @dby Establishes a new external file, with the given name and form, and associates this external file with the given file. The given file is left open. The current mode of the given file is set to the given access mode. The default access mode is the mode Out_File for sequential, stream, and text input-output; it is the mode Inout_File for direct input-output. For direct access, the size of the created file is implementation defined. !corrigendum A.12.1(28) @drepl The subprograms Create, Open, Close, Delete, Reset, Mode, Name, Form, Is_Open, and End_of_File have the same effect as the corresponding subprograms in Sequential_IO (see A.8.2). @dby The subprograms given in subclause A.8.2 for the control of external files (Create, Open, Close, Delete, Reset, Mode, Name, Form, and Is_Open) are available for stream files. The End_Of_File function: @xbullet @xbullet @xbullet !ACATS test An ACATS test needs to be created to insure that stream files are not truncated except on Create. !appendix !topic Inout_File mode missing for Ada.Streams.Stream_Io !reference RM95-A.12.1(6) !from Bernard Maudry !discussion Would it be possible to add the "Inout_File" mode to Ada.Streams.Stream_Io.File_Type as Reset truncates a file after the last write position and it is painful (and sometimes erroneous) to do a dummy Write at the end of the file before a Reset on this file from Out_File mode to In_File mode. ************************************************************* From: Randy Brukardt Date: Friday, December 14, 2001 5:08 PM This seems like a clear bug in the standard (as modified by the Corrigendum). We certainly don't need to add a new mode to fix it. In the discussions for the Corrigendum, we decided that positionable Stream_IO files acted essentially as direct files. Only non-positionable files act like sequential files. We thought that we'd fixed the wording to make that happen, but apparently we missed something. To actually implement that rule (A.8.2(16), the sentence starting with "In addition,", which is included by A.12.1(28)) as written, an implementation would have to keep track of the highest position written since the last Create/Open/Reset. For a non-positionable file, that is the same as the current position, but for a positionable file, it could be anywhere. I don't think it was intended that there was a requirement to save/check the write location after every write. So, I think that that sentence in A.8.2(16) is intended to apply only to non-positionable stream files. This is the only rule in A.8.2 that applies only to sequential files, so I don't think that there are any more examples of this particular problem lurking. There may be good arguments for adding an Inout_File mode to Stream_IO, but I don't think this is one. ************************************************************* From: Randy Brukardt Date: Friday, December 14, 2001 5:15 PM Oops, of course Close also has this behavior. And that also is a problem. If you are representing a database index as a stream file, if you insert a record (and write a couple of items in the middle), you certainly don't want to have to write at the end so that the silly system doesn't truncate the file. Perhaps we need a Truncate operation in Stream_IO for the times when you do want to truncate? ************************************************************* From: Pascal Leroy Date: Thursday, December 20, 2001 4:50 AM This issue looks like a red herring to me. If you want to change the mode without truncating, use Set_Mode, not Reset. It seems that the only difference between the two is that Set_Mode does not truncate, but Reset does. If you want to close the file without truncating, use Set_Mode to change the mode to In_File before closing. OK, this means that you need to perform two operations (Set_Mode + Close). But the alternative model suggested by Randy (where Close doesn't truncate and we have an extra Truncate operation) also requires two operations in the case where you want to truncate-and-close. Actually, I find the current model (as stated by RM95+TC1) rather natural and certainly consistent. But evidently it should be tested by the ACATS since it's an area where differences between implementations could cause lots of trouble. ************************************************************* From: Randy Brukardt Date: Friday, December 28, 2001 5:23 PM Since compatibility with existing behavior will be an important part of the resolution of this issue (truncation of Stream_IO files with mode Out_File), I've created a test and distributed it to implementors. (If anyone out there would like to try it themselves, just drop me a line and I'd be happy to send it. I'm not going to post it to the list as it is 535 lines long.) I'll post a summary of results in a week or so. I've run the test on 4 popular compilers for Windows, and I have a few preliminary conclusions. (I'm writing this up now, rather than waiting for additional results, so I don't forget all of this over the long holiday weekend. I expect to have fewer brain cells next Wednesday... :-) -- Most implementors do in fact expect to have to truncate Stream_IO files on Close. Apparently, I'm the only implementor who finds this bizarre. -- None of the compilers I tested supports Pascal's suggested "workaround", including Pascal's compiler. :-) That suggests if this topic is handled with a confirmation AI, a strong ACATS test is needed to insure that the workaround actually works. -- When positioning is added to the mix, the results vary wildly. I'll need more results to form any opinion here. It appears to be very dangerous to open a Stream_IO file with mode Out_File and write only a few elements before closing with current compilers. -- Stream_IO is still buggy: several implementation raise End_Error when Stream_IO.Read reads past the end of a file; one implementation deletes the entire contents of the file when it is opened with mode Out_File (even if it is Set_Moded to In_File and closed without other operations). It is interesting to note that although implementors mostly believe that truncation is required on Close and Reset, it is possible to read the RM so this is not a requirement. A.12.1(28) says that "... Close, ... Reset, ... has the same effect as the corresponding subprograms in Sequential_IO." A.8.2(10) says, "In addition, for sequential files, if the file being closed has mode Out_File or Append_File, then the last element written since the most recent open or reset is the last element that can be read from the file." (The language for Reset in A.8.2(16) is almost identical.) A.8(1) says that a file used for sequential access is a sequential file. The AARM also says specificially that stream files are not sequential files (see A.8(2.a-b)). So, we can argue both ways: Since Sequential_IO only works on sequential files, the truncation requirement also applies to Stream_IO by a literal reading of A.12.1(28). Alternatively, since the description of Close and Reset say that truncation only applies to sequential files, and the clear language designer's intent is that stream files are not sequential files, then the truncation does not apply to stream files. Both interpretations seem valid to me, but of course are completely different. A common implementation of truncation for sequential files (in Sequential_IO) is to empty the file when it is opened. This is more efficient than truncating the file on Close (most sequential files are Created, not Opened with Out_File, and usually no truncation is needed), and it works even on operating systems that do not support truncating files (because the file can be deleted on Open to provide the correct semantics). However, a direct copying of this implementation to stream files leads to disaster. For instance, if Stream_IO is used to maintain a database, updating a single record by opening the file with mode Out_File, writing the record, and closing the file would discard the contents of the entire database! At least one existing implementation appears to have made this mistake. It is also worth noting that a correct implementation of Stream_IO truncation on an operating system that does not support file truncation would be very expensive (requiring an under-the-covers copy of the file). I'm not sure that this is a real issue (the OSes that I used that didn't support truncation are long obsolete), but I think we need to consider it. Finally, exactly how the truncation is done when positioning is involved is unclear. Recall the wording of A.8.2(10) says that the "last element written" is the last one in the file after a Close. Does this mean the "most recent element written" or "the element with the largest Index written"? A natural English reading would imply the former; however implementations seem to favor the latter. In the absence of compatibility issues, I'd suggest reading the RM to say that truncation is not required. However, given the trend of implementations, that probably would be a bad idea. Thus we're going to have to figure out how something intended solely for sequential files (truncation) applies to files with positioning. Sigh. Stream_IO is still a mess. ************************************************************* From: Randy Brukardt Date: Monday, February 4, 2002 7:56 PM You may recall that back in December, I created and distributed a test to determine the current behavior of Ada compilers when closing or resetting a stream file with mode Out_File. Here is the promised summary of the results -- they're pretty grim in that compilers are very inconsistent, even compilers from the same vendor. I received results from most of the compiler vendors, but I didn't receive results from ICC, despite repeated requests. All other compilers are listed in alphabetical order of the product name. Aversomething AdaMagic (latest development version): -- Truncates sequential-access stream files on close/reset: No. -- Truncates stream files opened with In_File mode, then Reset to Out_File: No. -- Supports Set_Mode "workaround" to avoid truncation: Yes. -- Truncates positional-access stream files on close/reset: No. -- Discards data on Open for Out_File for positional-access stream files: No. -- No Truncation. -- File opened with mode Append_File truncates after positioning and writing in middle: Positioning not allowed. Rational Apex for Sun Solaris (latest development version): -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes. -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad). -- Truncation is at the largest position element written. -- File opened with mode Append_File truncates after positioning and writing in middle: No. -- (This implementation appears to violate AI-85 by allowing positioning, -- but ignoring the result and writing at the end anyway.) Rational Apex for Windows NT (3.2.0c): -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: No. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes. -- Discards data on Open for Out_File for positional-access stream files: No. -- Truncation is at the largest position element written. -- File opened with mode Append_File truncates after positioning and writing in middle: No. (Implementation incorrectly raises End_Error when reading past the end of the file with Stream_IO.Read) GNAT for Sun Solaris (latest development version): -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: No. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes. -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad). -- Truncation is at the largest position element written. -- File opened with mode Append_File truncates after positioning and writing in middle: No. GNAT for Windows NT (3.14a1): -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: No. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes. -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad). -- Truncation is at the largest position element written. -- File opened with mode Append_File truncates after positioning and writing in middle: No. Janus/Ada for Windows NT (3.1.2 development) -- Truncates sequential-access stream files on close/reset: No. -- Truncates stream files opened with In_File mode, then Reset to Out_File: No. -- Supports Set_Mode "workaround" to avoid truncation: Yes. -- Truncates positional-access stream files on close/reset: No. -- Discards data on Open for Out_File for positional-access stream files: No. -- No Truncation. -- File opened with mode Append_File truncates after positioning and writing in middle: No. Aonix ObjectAda for Sun Solaris (version 7.2.1) -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes (bizarre behavior: truncates to the number of elements written, ignoring the positioning -- except for the basic case of immediate Close after writing, which works as expected)). -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad). -- Can't tell truncation point (due to bizarre behavior). -- File opened with mode Append_File truncates after positioning and writing in middle: No. -- (This implementation appears to violate AI-85 by allowing positioning, -- but ignoring the result and writing at the end anyway.) Aonix ObjectAda for Windows NT (version 7.2.1) -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes (bizarre behavior: truncates to the number of elements written, ignoring the positioning). -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad). -- Can't tell truncation point (due to bizarre behavior). -- File opened with mode Append_File truncates after positioning and writing in middle: Yes. OCS PowerAda/6000 Compiler (Version 4.3) -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes. -- Supports Set_Mode "workaround" to avoid truncation: No. (Set_Mode unexpectedly raised Use_Error at this point [with an Exception_Message of "Set_Mode not yet supported"], which terminated the test.) DDCI SCORE 2.2 Solaris native: -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes. -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad). -- Truncation is at the largest position element written. -- File opened with mode Append_File truncates after positioning and writing in middle: No. The main conclusion that can be drawn from this is that there is no consistency, even between the same vendor's implementations. Probably some sort of truncation for sequential-access files should be required (since most vendors do it, even though neither the AARM nor Tucker support the requirement), but beyond that it is very murky. OTOH, the large number of implementations that discard data when a file is Opened with mode Out_File (even if later access is simply an update of a few elements) is horrifying. It means that Stream_IO cannot be used in applications requiring in-place updating -- an application that was clearly intended by the language designers. Also, hardly any of the implementations support the workaround suggested to avoid truncation. So, if a user want to avoid this behavior (for in-place updating, for example), they're out of luck with most implementations. We'll have to discuss the best approach at the upcoming meeting. ************************************************************* From Minutes of ARG meeting 15, February 2002, Cupertino CA Randy describes the problem and the result of his compiler survey (most compilers use the same algorithm for truncation as for Sequential_IO: truncate on open for Out_File). The current behavior of most compilers is a bug: it causes any data already in the file to be discarded, even if it is not going to be overwritten and even if the user has taken precautions to move the current index before closing. The group agrees that this is wrong. Tucker comments that if you don't truncate on open, it is expensive to truncate at close on Unix systems. There are probably other systems with this property. Pascal notes that if we don't truncate, we need to provide a way for the user to do it. Tucker notes that no existing implementation truncates on close, so we only need to provide what is currently available. Delete followed by Create would accomplish that. Therefore, the group settles on no truncation of Stream_IO files, except of course by Create in In_File or Out_File modes. This should be accomplished by fixing the equivalence so it clearly doesn't imply truncation. Approve intent of the AI: 3-0-3 Randy will write the wording. *************************************************************