Version 1.2 of ai05s/ai05-0283-1.txt

!topic IO packages should have preelaborate aspect
!reference Ada 2005 RM A.10.1 (2)
!from /Brad Moore 11-09-27/
!keywords Preelaborate Remote_Types Text_IO
!discussion

It would be nice if the IO packages (Ada.Text_IO, etc)
had the Preelaborate aspect. This would
facilitate use for Remote_Types. This would make it easier to
create RACW types that have persistence, provide remote file
access, and database access for direct IO, for example.
One cannot currently with IO packages from a Remote_Types package,
because the IO packages are not preelaborable.

****************************************************************

From: Randy Brukardt
Sent: Wednesday, September 28, 2011  7:33 PM

This problem was extensively discussed during the Ada 2005 work -- see
AI95-0362-1. Note that the problem statement in that AI specifically
mentions IO.

The discussion section of that AI gives a very detailed look at the results
of that discussion. It should be noted that the original version of the AI
proposed making Direct_IO, Sequential_IO and Stream_IO preelaborable
(Text_IO is not reasonable for reasons discussed in the AI). We circulated
that to implementers to get feedback, and various objections were recorded.
In the end, those proposals were dropped in order to get consensus on the
remainder of the AI.

Calendar and Text_IO are the biggest impediment to making most packages
Preelaborable, but it does not appear reasonable for either to be
Preelaborable. (And probably Calendar shouldn't be a Remote_Types package
even if it was Preelaborable.) There was a brief amount of discussion of
having a Preelaborable "Simple_Text_IO", but that appeared like a long
argument over exactly what to include (the original proposal not having any
file I/O, just StdOut, making it useless for logging, my #1 need).

A case can be made for the other I/O packages, but that case would have to
be stronger than the one I made in 2003. (Wow, was it really that long ago?)
I'm pretty sure "It would be nice" is not a strong case. :-) So if you have
more examples of places where I/O was needed in preelaborated units, it
would help consideration of your proposal.

****************************************************************

From: Brad Moore
Sent: Thursday, September 29, 2011  12:58 AM

> The discussion section of that AI gives a very detailed look at the results
> of that discussion. It should be noted that the original version of the AI
> proposed making Direct_IO, Sequential_IO and Stream_IO preelaborable
> (Text_IO is not reasonable for reasons discussed in the AI). We circulated
> that to implementers to get feedback, and various objections were recorded.
> In the end, those proposals were dropped in order to get consensus on the
> remainder of the AI.

In my particular case, I was looking for Direct_IO to have the
preelaborate pragma. I was implementing
a suite of buffer generics (essentially a deque container), including
persistent buffers where the storage for the buffer is
entirely in a file. The buffers were designed to support remote access
via Remote Access to Class Wide
types (RACW types). I ended up having to create my own IO package,
called Preelaborated_IO.
Note using GNAT_IO was not a possibility because;
    1) I needed the package to be generic, where the element type is a
generic formal parameter. GNAT_IO is non-generic and only
        deals with integers, and strings.
    2) I wanted the buffers to be portable to other compiler vendors.

> Calendar and Text_IO are the biggest impediment to making most packages
> Preelaborable, but it does not appear reasonable for either to be
> Preelaborable. (And probably Calendar shouldn't be a Remote_Types package
> even if it was Preelaborable.) There was a brief amount of discussion of
> having a Preelaborable "Simple_Text_IO", but that appeared like a long
> argument over exactly what to include (the original proposal not having any
> file I/O, just StdOut, making it useless for logging, my #1 need).

I think I'm OK with Text_IO not being preelaborable. I'm more interested
in Sequential_IO and Direct_IO.

> A case can be made for the other I/O packages, but that case would have to
> be stronger than the one I made in 2003. (Wow, was it really that long ago?)
> I'm pretty sure "It would be nice" is not a strong case. :-) So if you have
> more examples of places where I/O was needed in preelaborated units, it
> would help consideration of your proposal.

If I was asked before I had implemented my own package, my response
would have been  a lot stronger,
possibly, "It would be bleeping awesome!"

Also, all the buffer containers (including non-persistent ones) have a
Save and a Restore primitive that
can load or save the container to a file, which is another case where I
needed to use Preelaborated_IO.

Maybe such primitives might be worth considering some day for the
Standard Ada containers?
While a container can be streamed to a file, it is not as convenient,
nor likely as efficient, as having a Save and Restore primitive.

It seems a shame that every time someone wants to do something similar
with Remote_Types packages, they would have to end up
writing their own Preelaborated_IO package. (Or possibly use mine)

****************************************************************

From: Randy Brukardt
Sent: Thursday, September 29, 2011  1:18 AM

...
> If I was asked before I had implemented my own package, my
> response would have been  a lot stronger, possibly, "It would
> be bleeping awesome!"

<Grin>

> Also, all the buffer containers (including non-persistent
> ones) have a Save and a Restore primitive that can load or
> save the container to a file, which is another case where I
> needed to use Preelaborated_IO.

That sounds like a job for a (Preelaborated version of) Stream_IO.

> Maybe such primitives might be worth considering some day for
> the Standard Ada containers?
> While a container can be streamed to a file, it is not as
> convenient, nor likely as efficient, as having a Save and
> Restore primitive.

No, you just need a memory stream. Claw has such a stream (I think we called
it a marshalling stream); we use it for writing to/from the clipboard and
registry (but we also made it generally available). It's better to do that
than reinvent the wheel (for streaming).

I think the Standard should have something similar. Ada has Storage_IO for
this purpose, but that isn't likely to work with user-defined types
(certainly not if there is any indirection in the types).

> It seems a shame that every time someone wants to do
> something similar with Remote_Types packages, they would have
> to end up writing their own Preelaborated_IO package. (Or
> possibly use mine)

Probably we should bite the bullet and make Stream_IO preelaborated (you can
write all of the others using it, and most of the time you would be better
off using it anyway -- almost all non-text files are heterogeneous in
practice). Otherwise, we should create a preelaborated version of it, but
there doesn't seem to be much point in having two packages with the same
contents.

If we left the others alone, the pain would be fairly localized. And a
motivated user could write their own Sequential_IO and Direct_IO analogs.

Anyway, something to consider.

****************************************************************

From: Brad Moore
Sent: Saturday, October  1, 2011  10:39 AM

I agree with your suggestion. That sounds like the best way forward.

****************************************************************

Editor's note: The following was in the original Ada 2012 version of the
AI, but it is not appropriate in the Ada 2005 AI (because we're not likely
to add significant packages at this point).

---

The e-mail thread also touches on the need to be able to
serialize containers into memory-based structures. The best way to
do this is with a memory stream. Claw defines one to use for writing
to the registry and clipboard (and also makes it available to users).
Ada probably ought to have one. Claw's version looks like:

with Ada.Streams;
package Claw.Marshalling is

  -- Writes to a buffer will allocate additional memory from the
  -- heap as needed.  If you know reasonably accurately how much
  -- you'll need, it will be more efficient to make the
  -- Initial_Length big enough and thus avoid heap allocation.
  --
  -- Any heap allocation will be freed automatically when a Buffer
  -- ceases to exist.

  type Buffer_Type(Initial_Length : Ada.Streams.Stream_Element_Count)
    is new Ada.Streams.Root_Stream_Type with private;

  procedure Read(
    Stream : in out Buffer_Type;
    Item   :    out Ada.Streams.Stream_Element_Array;
    Last   :    out Ada.Streams.Stream_Element_Offset);

  procedure Write(
    Stream : in out Buffer_Type;
    Item   : in     Ada.Streams.Stream_Element_Array);

  function Length(Stream : in Buffer_Type)
      return Ada.Streams.Stream_Element_Count;
	-- Return the total length of data written into the buffer.

  procedure Clear(Stream : in out Buffer_Type);
	-- Empty the buffer.

private
  ...
end Claw.Marshalling;

Should we consider something on this line for Ada??

****************************************************************

From: Randy Brukardt
Sent: Tuesday, November 15, 2011  12:41 PM

The ARG would like input from implementers on the following topic:

What would the effect of declaring that Ada.Streams.Stream_IO is preelaborated
be on your implementation?

Background:

Currently, no I/O packages in the Ada Standard are considered preelaborated.
This means that tricks (using pragma Import to break the rules, or passing in
access-to-subprogram parameters to do the I/O) are necessary to use any I/O in
preelaborated packages. This makes common tasks such as logging and persistent
objects harder and much more obscure than otherwise would be required. This is
especially a problem in distributed (Annex E) programs, where large parts of
each partition is required to be Preelaborated to support distribution.

We considered making such a change to all of the I/O packages in Ada 2005, but
some of the features of the packages (especially Text_IO) made it difficult to
implement. There also was quite a bit of disruption to some existing Ada
implementations. In the end, we did not make any change in Ada 2005.

But of course the problem has not gone away. Therefore, the ARG is now
considering only changing Stream_IO (on top of which any needed I/O can be
constructed, and which typically would be used directly to implement
persistence) to be preelaborated, as it supports simple, (potentially)
hetrogenous I/O.

Investigations done during the recent ARG meeting showed that the change is not
significant for two widely-used implementations (AdaMagic and GNAT) --
essentially, the pragma needed to be applied to the Stream_IO package and a few
units that it uses. As such, the ARG would like to make this change in Ada 2012
(as it is only a one-line change to the Ada Standard) -- but it doesn't want to
do that if it causes hardships to many other Ada implementations.

Please note that we are *not* considering making Stream_IO a Remote_Types
package, as transporting file values (especially in a hetrogenous system) does
not make much sense. The only change contemplated is adding pragma
Preelaborate(Stream_IO) to the package Ada.Streams.Stream_IO.

Thank you for your help.

****************************************************************

From: Tucker Taft (SofCheck)
Sent: Tuesday, November 15, 2011  1:17 PM

As I mentioned during the ARG meeting,
this is no problem for SofCheck's AdaMagic product.

****************************************************************

From: Randy Brukardt (RRSoftware)
Sent: Tuesday, November 15, 2011  1:45 PM

Which was mentioned in the body of the message. But it doesn't hurt to repeat
it.

As for new information, (changing hats here), I just made this change in the
working version of Janus/Ada. I needed to make a small change to our core I/O
unit (Basic_IO) to only check the search path upon the first Open that needs it
rather than when Basic_IO elaborates. I don't expect that to have much impact
(it will increase the memory use of Open somewhat, but any program with tight
stack memory requirements that is doing file I/O is probably going to have
problems anyway). Otherwise, the change just required adding the pragma to the
two units (Stream_IO and the previously mentioned Basic_IO).

****************************************************************

From: Gary Dismukes (AdaCore)
Sent: Tuesday, November 15, 2011  2:18 PM

> Investigations done during the recent ARG meeting showed that the
> change is not significant for two widely-used implementations
> (AdaMagic and GNAT) -- essentially, the pragma needed to be applied to
> the Stream_IO package and a few units that it uses. ...

It turns out that there's a little more change than that required for GNAT.
Mostly its adding Preelaborate to a few packages, but there's a reference to an
imported max_path_len variable in one of the support packages (in a record
type's string component range) and one place where a warning needs to be
suppressed.  The string bound violation is an issue that can probably be
addressed in some way, so it still looks feasible for GNAT.

****************************************************************

From: Johan O. Nielsen (DDCI)
Sent: Wednesday, November 16, 2011  4:29 PM

The really short answer is No effect. SCORE Ada already has added a pragma
Preelaborate to Ada.Streams.Stream_Io.

SCORE Ada is still mostly Ada 95 but I don't see any changes in Ada 2005 or
proposed changes in Ada 2012 that would make it more difficult for our
Ada.Streams.Stream_Io to be preelaborated. The statements here are based on the
understanding that no other Stream_Io changes are introduced compared to Draft
14. I agree with the general desire to make as much of the language available in
preelaborated units so this change is welcomed.

It would be unproblematic for SCORE Ada to go a little further in this area and
add the pragma for Ada.Direct_Io and Ada.Sequential_Io as well. On the other
hand, I don't think we can hope to ever make the Text_Io packages preelaborated.

****************************************************************

From: Adam Beneschan (Irvine)
Sent: Wednesday, November 16, 2011  5:24 PM

I just tried this and adding this pragma isn't significant for us at all.  (In
fact, I added the pragma to the Stream_IO package and it compiled without any
other changes being needed.)

****************************************************************

From: Dan Eilers (Irvine)
Sent: Friday, November 18, 2011  10:37 AM

> The ARG would like input from implementers on the following topic:
>
> What would the effect of declaring that Ada.Streams.Stream_IO is
> preelaborated be on your implementation?

Adam responded directly to Randy that this works fine for us with no other
changes.

I notice that this issue is currently categorized as:
> AI12-0010-1/01   2011-11-08 --  Stream_IO should be preelaborated

under the heading in the Denver ARG agenda:
> Ada 2012 AIs (to fix issues in the Ada 2012 Standard, most likely
> after the Standard is issued):

I would strongly recommend changing this from an Ada 2012 AI to an Ada 2005 AI,
so it makes it into the Ada 2012 standard, since it provides a very simple
solution to an important problem that has no other good solutions available.
Perhaps it should even be a binding interpretation, so it can be implemented in
Ada 2005 compilers and used in Ada 2005 code.

The reason I think this is important is that I personally spent a considerable
amount of time debugging a very complicated piece of Ada 2005 software that
tried to work aound this problem by supplying a custom pre-elaborable version of
text_io, which turned out to be non-portable because it relied on the layout of
the Unix stat record.

****************************************************************

From: Randy Brukardt
Sent: Friday, November 18, 2011  1:13 PM

...
> I would strongly recommend changing this from an Ada 2012 AI to an Ada
> 2005 AI, so it makes it into the Ada 2005 standard, since it provides
> a very simple solution to an important problem that has no other good
> solutions available.  Perhaps it should even be a binding
> interpretation, so it can be implemented in Ada 2005 compilers and
> used in Ada 2005 code.

As noted in my original message, including it in Ada 2012 is our intent of
asking about it now. This is only a one-line change in the Standard, much
smaller than many editorial fixes, and it adds a lot of capability. It might be
a bit much for a BI (it changes the memory use of Open substantially in
Janus/Ada, for instance - I could easily imagine that breaking programs).

BTW, I've also heard from DDCI that this would not cause any problems for them.
So, so far, we've heard from 5 implementers with no reports of any serious
problems. Looks like a GO so far.

****************************************************************