!standard A.12.01 (28) 01-12-26 AI95-00283/00 !standard A.08.02 (10) !standard A.08.02 (16) !class amendment 01-12-27 !status received 01-12-14 !priority Low !difficulty Medium !subject Truncation of stream files by Close and Reset !summary (TBD.) !problem !proposal !discussion !example !ACATS test !appendix !topic Inout_File mode missing for Ada.Streams.Stream_Io !reference RM95-A.12.1(6) !from Bernard Maudry !discussion Would it be possible to add the "Inout_File" mode to Ada.Streams.Stream_Io.File_Type as Reset truncates a file after the last write position and it is painful (and sometimes erroneous) to do a dummy Write at the end of the file before a Reset on this file from Out_File mode to In_File mode. ************************************************************* From: Randy Brukardt Date: Friday, December 14, 2001 5:08 PM This seems like a clear bug in the standard (as modified by the Corrigendum). We certainly don't need to add a new mode to fix it. In the discussions for the Corrigendum, we decided that positionable Stream_IO files acted essentially as direct files. Only non-positionable files act like sequential files. We thought that we'd fixed the wording to make that happen, but apparently we missed something. To actually implement that rule (A.8.2(16), the sentence starting with "In addition,", which is included by A.12.1(28)) as written, an implementation would have to keep track of the highest position written since the last Create/Open/Reset. For a non-positionable file, that is the same as the current position, but for a positionable file, it could be anywhere. I don't think it was intended that there was a requirement to save/check the write location after every write. So, I think that that sentence in A.8.2(16) is intended to apply only to non-positionable stream files. This is the only rule in A.8.2 that applies only to sequential files, so I don't think that there are any more examples of this particular problem lurking. There may be good arguments for adding an Inout_File mode to Stream_IO, but I don't think this is one. ************************************************************* From: Randy Brukardt Date: Friday, December 14, 2001 5:15 PM Oops, of course Close also has this behavior. And that also is a problem. If you are representing a database index as a stream file, if you insert a record (and write a couple of items in the middle), you certainly don't want to have to write at the end so that the silly system doesn't truncate the file. Perhaps we need a Truncate operation in Stream_IO for the times when you do want to truncate? ************************************************************* From: Pascal Leroy Date: Thursday, December 20, 2001 4:50 AM This issue looks like a red herring to me. If you want to change the mode without truncating, use Set_Mode, not Reset. It seems that the only difference between the two is that Set_Mode does not truncate, but Reset does. If you want to close the file without truncating, use Set_Mode to change the mode to In_File before closing. OK, this means that you need to perform two operations (Set_Mode + Close). But the alternative model suggested by Randy (where Close doesn't truncate and we have an extra Truncate operation) also requires two operations in the case where you want to truncate-and-close. Actually, I find the current model (as stated by RM95+TC1) rather natural and certainly consistent. But evidently it should be tested by the ACATS since it's an area where differences between implementations could cause lots of trouble. ************************************************************* From: Randy Brukardt Date: Friday, December 28, 2001 5:23 PM Since compatibility with existing behavior will be an important part of the resolution of this issue (truncation of Stream_IO files with mode Out_File), I've created a test and distributed it to implementors. (If anyone out there would like to try it themselves, just drop me a line and I'd be happy to send it. I'm not going to post it to the list as it is 535 lines long.) I'll post a summary of results in a week or so. I've run the test on 4 popular compilers for Windows, and I have a few preliminary conclusions. (I'm writing this up now, rather than waiting for additional results, so I don't forget all of this over the long holiday weekend. I expect to have fewer brain cells next Wednesday... :-) -- Most implementors do in fact expect to have to truncate Stream_IO files on Close. Apparently, I'm the only implementor who finds this bizarre. -- None of the compilers I tested supports Pascal's suggested "workaround", including Pascal's compiler. :-) That suggests if this topic is handled with a confirmation AI, a strong ACATS test is needed to insure that the workaround actually works. -- When positioning is added to the mix, the results vary wildly. I'll need more results to form any opinion here. It appears to be very dangerous to open a Stream_IO file with mode Out_File and write only a few elements before closing with current compilers. -- Stream_IO is still buggy: several implementation raise End_Error when Stream_IO.Read reads past the end of a file; one implementation deletes the entire contents of the file when it is opened with mode Out_File (even if it is Set_Moded to In_File and closed without other operations). It is interesting to note that although implementors mostly believe that truncation is required on Close and Reset, it is possible to read the RM so this is not a requirement. A.12.1(28) says that "... Close, ... Reset, ... has the same effect as the corresponding subprograms in Sequential_IO." A.8.2(10) says, "In addition, for sequential files, if the file being closed has mode Out_File or Append_File, then the last element written since the most recent open or reset is the last element that can be read from the file." (The language for Reset in A.8.2(16) is almost identical.) A.8(1) says that a file used for sequential access is a sequential file. The AARM also says specificially that stream files are not sequential files (see A.8(2.a-b)). So, we can argue both ways: Since Sequential_IO only works on sequential files, the truncation requirement also applies to Stream_IO by a literal reading of A.12.1(28). Alternatively, since the description of Close and Reset say that truncation only applies to sequential files, and the clear language designer's intent is that stream files are not sequential files, then the truncation does not apply to stream files. Both interpretations seem valid to me, but of course are completely different. A common implementation of truncation for sequential files (in Sequential_IO) is to empty the file when it is opened. This is more efficient than truncating the file on Close (most sequential files are Created, not Opened with Out_File, and usually no truncation is needed), and it works even on operating systems that do not support truncating files (because the file can be deleted on Open to provide the correct semantics). However, a direct copying of this implementation to stream files leads to disaster. For instance, if Stream_IO is used to maintain a database, updating a single record by opening the file with mode Out_File, writing the record, and closing the file would discard the contents of the entire database! At least one existing implementation appears to have made this mistake. It is also worth noting that a correct implementation of Stream_IO truncation on an operating system that does not support file truncation would be very expensive (requiring an under-the-covers copy of the file). I'm not sure that this is a real issue (the OSes that I used that didn't support truncation are long obsolete), but I think we need to consider it. Finally, exactly how the truncation is done when positioning is involved is unclear. Recall the wording of A.8.2(10) says that the "last element written" is the last one in the file after a Close. Does this mean the "most recent element written" or "the element with the largest Index written"? A natural English reading would imply the former; however implementations seem to favor the latter. In the absence of compatibility issues, I'd suggest reading the RM to say that truncation is not required. However, given the trend of implementations, that probably would be a bad idea. Thus we're going to have to figure out how something intended solely for sequential files (truncation) applies to files with positioning. Sigh. Stream_IO is still a mess. ************************************************************* From: Randy Brukardt Date: Monday, February 4, 2002 7:56 PM You may recall that back in December, I created and distributed a test to determine the current behavior of Ada compilers when closing or resetting a stream file with mode Out_File. Here is the promised summary of the results -- they're pretty grim in that compilers are very inconsistent, even compilers from the same vendor. I received results from most of the compiler vendors, but I didn't receive results from ICC, despite repeated requests. All other compilers are listed in alphabetical order of the product name. Aversomething AdaMagic (latest development version): -- Truncates sequential-access stream files on close/reset: No. -- Truncates stream files opened with In_File mode, then Reset to Out_File: No. -- Supports Set_Mode "workaround" to avoid truncation: Yes. -- Truncates positional-access stream files on close/reset: No. -- Discards data on Open for Out_File for positional-access stream files: No. -- No Truncation. -- File opened with mode Append_File truncates after positioning and writing in middle: Positioning not allowed. Rational Apex for Sun Solaris (latest development version): -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes. -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad). -- Truncation is at the largest position element written. -- File opened with mode Append_File truncates after positioning and writing in middle: No. -- (This implementation appears to violate AI-85 by allowing positioning, -- but ignoring the result and writing at the end anyway.) Rational Apex for Windows NT (3.2.0c): -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: No. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes. -- Discards data on Open for Out_File for positional-access stream files: No. -- Truncation is at the largest position element written. -- File opened with mode Append_File truncates after positioning and writing in middle: No. (Implementation incorrectly raises End_Error when reading past the end of the file with Stream_IO.Read) GNAT for Sun Solaris (latest development version): -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: No. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes. -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad). -- Truncation is at the largest position element written. -- File opened with mode Append_File truncates after positioning and writing in middle: No. GNAT for Windows NT (3.14a1): -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: No. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes. -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad). -- Truncation is at the largest position element written. -- File opened with mode Append_File truncates after positioning and writing in middle: No. Janus/Ada for Windows NT (3.1.2 development) -- Truncates sequential-access stream files on close/reset: No. -- Truncates stream files opened with In_File mode, then Reset to Out_File: No. -- Supports Set_Mode "workaround" to avoid truncation: Yes. -- Truncates positional-access stream files on close/reset: No. -- Discards data on Open for Out_File for positional-access stream files: No. -- No Truncation. -- File opened with mode Append_File truncates after positioning and writing in middle: No. Aonix ObjectAda for Sun Solaris (version 7.2.1) -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes (bizarre behavior: truncates to the number of elements written, ignoring the positioning -- except for the basic case of immediate Close after writing, which works as expected)). -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad). -- Can't tell truncation point (due to bizarre behavior). -- File opened with mode Append_File truncates after positioning and writing in middle: No. -- (This implementation appears to violate AI-85 by allowing positioning, -- but ignoring the result and writing at the end anyway.) Aonix ObjectAda for Windows NT (version 7.2.1) -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes (bizarre behavior: truncates to the number of elements written, ignoring the positioning). -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad). -- Can't tell truncation point (due to bizarre behavior). -- File opened with mode Append_File truncates after positioning and writing in middle: Yes. OCS PowerAda/6000 Compiler (Version 4.3) -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes. -- Supports Set_Mode "workaround" to avoid truncation: No. (Set_Mode unexpectedly raised Use_Error at this point [with an Exception_Message of "Set_Mode not yet supported"], which terminated the test.) DDCI SCORE 2.2 Solaris native: -- Truncates sequential-access stream files on close/reset: Yes. -- Truncates stream files opened with In_File mode, then Reset to Out_File: Yes. -- Supports Set_Mode "workaround" to avoid truncation: No. -- Truncates positional-access stream files on close/reset: Yes. -- Discards data on Open for Out_File for positional-access stream files: Yes (very bad). -- Truncation is at the largest position element written. -- File opened with mode Append_File truncates after positioning and writing in middle: No. The main conclusion that can be drawn from this is that there is no consistency, even between the same vendor's implementations. Probably some sort of truncation for sequential-access files should be required (since most vendors do it, even though neither the AARM nor Tucker support the requirement), but beyond that it is very murky. OTOH, the large number of implementations that discard data when a file is Opened with mode Out_File (even if later access is simply an update of a few elements) is horrifying. It means that Stream_IO cannot be used in applications requiring in-place updating -- an application that was clearly intended by the language designers. Also, hardly any of the implementations support the workaround suggested to avoid truncation. So, if a user want to avoid this behavior (for in-place updating, for example), they're out of luck with most implementations. We'll have to discuss the best approach at the upcoming meeting. *************************************************************