!standard A.16(3/2) 16-06-07 AI12-0009-1/06 !standard A.16(36.1/3) !standard A.16(98/2) !standard A.16(112.1/3) !standard A.16(125/3) !standard A.17(3/2) !standard A.17(8/3) !standard A.17(23/3) !standard A.17(25/2) !standard A.17(28/2) !class Amendment 11-11-08 !status work item 11-11-08 !status received 11-10-01 !priority Low !difficulty Medium !subject Iterators for Directories and Environment_Variables !summary Support for generalized loop iteration for Ada.Directories and Ada.Environment_Variables is added. !problem Ada.Directories and Ada.Environment_Variables contain iterator procedures with call-back procedures. There should be a version of the Ada 2012 iterators for these subprograms, so that users can write a loop. !proposal (See wording.) !wording Modify A.16(3/2) "with Ada.IO_Exceptions; with Ada.Calendar; {with Ada.Iterator_Interfaces;} package Ada.Directories is" Append after A.16(36/3) the following -- Directory Iteration type Directory_Listing is tagged limited private with Constant_Indexing => Current_Entry, Default_Iterator => Iterate, Iterator_Element => Directory_Entry_Type; pragma Preelaborable_Initialization (Directory_Listing); function Entries (Directory : in String; Pattern : in String; Filter : in Filter_Type := (others => True)) return Directory_Listing; type Cursor is private; pragma Preelaborable_Initialization (Cursor); function Has_Entry (Position : in Cursor) return Boolean; package Directory_Iterators is new Ada.Iterator_Interfaces (Cursor, Has_Entry); function Iterate (Listing : in Directory_Listing) return Directory_Iterators.Forward_Iterator'Class; function Current_Entry (Entries : in Directory_Listing; Position : in Cursor) return Directory_Entry_Type with Pre => Has_Entry (Position); Modify A.16(98/2) The type Directory_Entry_Type represents a single item in a directory. These items can only be created {either }by the Get_Next_Entry procedure in this package{, or by generalized loop iteration on the return object of the Entries function in this package}. Information about the item can be obtained from the functions declared in this package. A default-initialized object of this type is invalid; objects returned from Get_Next_Entry {and by generalized loop iteration on the result of Entries} are valid. Append after A.16(112/3) type Directory_Listing is tagged limited private; The type Directory_Listing contains the state of a directory search. An object of type Directory_Listing can be used in generalized loop iteration (see 5.5.2) to visit each available directory entry for a particular directory search. A default-initialized Directory_Listing object has no entries available. The type Directory_Listing needs finalization. function Entries (Directory : in String; Pattern : in String; Filter : in Filter_Type := (others => True)) return Directory_Listing; Returns an object that represents the result of a search in the directory named by Directory for entries matching Pattern and Filter. Pattern represents a pattern for matching file names. If Pattern is the null string, all items in the directory are matched; otherwise, the interpretation of Pattern is implementation-defined. Only items that match Filter will be returned. After a successful call on Entries, the return object may have entries available, but it may have no entries available if no files or directories match Pattern and Filter. The exception Name_Error is propagated if the string given by Directory does not identify an existing directory, or if Pattern does not allow the identification of any possible external file or directory. The exception Use_Error is propagated if the external environment does not support the searching of the directory with the given name (in the absence of Name_Error). When Start_Search propagates Name_Error or Use_Error, the return object will have no entries available. type Cursor is private; A cursor is a reference to a directory entry within a directory listing. To Be Honest: There may not be an actual reference component needed, but the abstraction of a cursor into a directory listing is intuitive and is consistent with other container iterators in the standard. function Has_Entry (Position : in Cursor) return Boolean; Returns True if Position designates a directory entry, and returns False otherwise. function Iterate (Listing : Directory_Listing) return Directory_Iterators.Forward_Iterator'Class; Iterate returns a forward iterator object (see 5.5.1) that will generate a value for a loop parameter (see 5.5.2) designating each directory entry in Listing, starting with the first directory entry and moving the cursor as per the Next function. The iterator object needs finalization. function Current_Entry (Entries : Directory_Listing; Position : Cursor) return Directory_Entry_Type with Pre => Has_Entry (Position); If Entries is a default-initialized object then Program_Error is propagated. Otherwise, Current_Entry returns the Directory_Entry designated by Position when used with a generalized iterator or container element iterator. It is implementation-defined as to whether the results returned by this subprogram are altered if the contents of the directory are altered during the current iteration of the loop (for example, by another program). The exception Use_Error is propagated if the external environment does not support continued searching of the directory represented by Entries. Modify A.16(125/3) Start_Search{,} [and ]Search{, and Entries} should raise Name_Error if Pattern is malformed, but not if it could represent a file in the directory but does not actually do so. Modify A.16(125.a/3) Implementation Advice: Directories.Start_Search{,} [and ] Directories.Search{, and Directories.Entries} should raise Name_Error for malformed patterns. Modify A.17(3/2) "{with Ada.Iterator_Interfaces;} package Ada.Environment_Variables is pragma Preelaborate(Environment_Variables);" Append after A.17(8/3) the following package Name_Value_Pairs is type Name_Value_Pair_Type is tagged limited private; -- Operations on Name_Value_Pairs function Name (Name_Value_Pair : in Name_Value_Pair_Type) return String; function Value (Name_Value_Pair : in Name_Value_Pair_Type) return String; private ... -- not specified by the languages end Name_Value_Pairs; use Name_Value_Pairs; type Environment_Variable_Listing is tagged limited private with Constant_Indexing => Current_Variable, Default_Iterator => Iterate, Iterator_Element => Name_Value_Pair_Type; pragma Preelaborable_Initialization (Environment_Variable_Listing); function Environment return Environment_Variable_Listing; type Cursor is private; pragma Preelaborable_Initialization (Cursor); function Has_Environment_Variable (Position : in Cursor) return Boolean; package Environment_Variable_Iterators is new Ada.Iterator_Interfaces (Cursor, Has_Environment_Variable); function Iterate (Listing : in Environment_Variable_Listing) return Environment_Variable_Iterators.Forward_Iterator'Class; function Current_Variable (Variables : in Environment_Variable_Listing; Position : in Cursor) return Name_Value_Pair_Type with Pre => Has_Environment_Variable (Position); Append after A.17(24/2) type Name_Value_Pair_Type is limited private; The type Name_Value_Pair_Type represents a single environment variable of the execution environment. A default-initialized object of this type is invalid; objects returned by generalized loop iteration on the result of Entries are valid. function Name (Name_Value_Pair : in Name_Value_Pair_Type) return String; Returns the name of the environment variable represented by Name_Value_Pair. function Value (Name_Value_Pair : in Name_Value_Pair_Type) return String; Returns the value of the environment variable represented by Name_Value_Pair. type Environment_Variable_Listing is tagged limited private; The type Environment_Variable_Listing contains the state of an environment variable search for the execution environment. An object of type Environment_Variable_Listing can be used in generalized loop iteration (see 5.5.2) to visit each environment variable from the external environment. A default-initialized Environment_Variable_Listing object represents all environment variables from the external environment. The type Environment_Variable_Listing needs finalization. If the external execution environment does not support environment variables, then Program_Error is propagated for generalized loop iteration. function Environment return Environment_Variable_Listing; Returns an object that represents the state of an environment variable search for all environment variables from the execution environment. The exception Program_Error is propagated if the external execution environment does not support environment variables. type Cursor is private; A cursor is a reference to an environment variable name value pair within an environment variable listing. To Be Honest: There may not be an actual reference component needed, but the abstraction of a cursor into an environment variable listing is intuitive and is consistent with other container iterators in the standard. function Has_Environment_Variable (Position : in Cursor) return Boolean; Returns True if Position designates an environment variable name value pair, and returns False otherwise. function Iterate (Variables : Environment_Variable_Listing) return Environment_Variable_Iterators.Forward_Iterator'Class; Iterate returns a forward iterator object (see 5.5.1) that will generate a value for a loop parameter (see 5.5.2) designating each environment variable in Variables, starting with the first environment variable and moving the cursor as per the Next function. The iterator object needs finalization. function Current_Variable (Variables : Environment_Variable_Listing; Position : Cursor) return Name_Value_Pair with Pre => Has_Environment_Variable (Position); Current_Variable returns the Name_Value_Pair designated by Position that is associated with the current state of the generalized iterator or container element iterator. Modify A.17(25/2) It is a bounded error to call Value {if the call does not involve an object of type Name_Value_Pair_Type,} if more than one environment variable exists with the given name; the possible outcomes are that: !discussion There were two identified approaches to provide this functionality. One was to have a function that returns an iterator_name that may be used as a generalized iterator. The second approach was to provide a function that returns an object of an iterable container type, that could be used to generate a container element iterator. From the user's perspective, these two approaches would involve slightly different syntax, since generalized iterators use "in" in the iterator_specification while container element iterators use "of". For example, a loop involving the first approach might look like; for Dir_Entry in Ada.Directories.Entries ("/etc", "*.txt") loop Put_Line (Directories.Simple_Name (Dir_Entry)); end loop; whereas for the second approach the same loop would be written as; for Dir_Entry of Ada.Directories.Entries ("/etc", "*.txt") loop Put_Line (Directories.Simple_Name (Dir_Entry)); end loop; To implement the first approach, the solution found required the use of a cursor type with an access discriminant and the Implicit_Dereference aspect to designate the corresponding Directory_Entry_Type for the loop. This was needed because Directory_Entry_Type is a limited type, and cannot be returned directly as object of the Ada.Iterator_Interfaces subprograms. To implement this in the Ada.Directories package, the following would need to be added to the public part of the package. with Ada.Iterator_Interfaces; ... type Cursor (Directory_Entry : not null access constant Directory_Entry_Type) is private with Implicit_Dereference => Directory_Entry; function Has_Element (Directory_Entry : Cursor) return Boolean; package Directory_Iterator_Interfaces is new Ada.Iterator_Interfaces (Cursor => Cursor, Has_Element => Has_Element); function Entries (Directory : String; Pattern : String; Filter : Filter_Type := (others => True)) return Directory_Iterator_Interfaces.Forward_Iterator'Class; It was generally felt that the use of the Implicit_Dereference aspect here was viewed as being a bit of a trick. It would be ideal if a solution could be found that was less "tricky". Also, having to expose a public type with an access discriminant was seen as being undesirable. The second approach eliminated these concerns, because there is no need to involve access discriminants or the Implicit_Dererence aspect. In addition, the second approach is more flexible because it allows the user to write both forms of loops. The second form of the loop above could be written as; declare Listing : Ada.Directories.Directory_Listing := Ada.Directories.Entries ("/etc", "*.txt"); begin for Dir_Entry in Listing.Iterate loop Put_Line (Directories.Simple_Name (Directories.Current_Entry (Entries => Listing, EP => Dir_Entry))); end loop; end; It was also considered whether the container and the iterator type should be the same type. This seemed more confusing and would have required exposing more of the iterator interface in the public part of the package. It also would have been less clear whether one could restart an iteration on the same container object a second time. It was considered whether reverse iteration through the containers should be supported. Since the existing iteration support for Ada.Directories did not support reverse iteration, it was felt that there would be insufficient need for this capability for generalized loop iteration. The overall approach taken for Ada.Environment_Variables is similar to the one taken for Ada.Directories except that the type Name_Value_Pair_Type was made to be a tagged type to improve usability by allowing object prefix notation. for Pair of Ada.Environment_Variables.Environment loops Put_Line (Pair.Name & "=" & Pair.Value); end loop; rather than; for Pair of Ada.Environment_Variables.Environment loops Put_Line (Ada.Environment_Variables.Name (Pair) & "=" & Ada.Environment_Variables.Value (Pair)); end loop; Making Name_Value_Pair_Type tagged however, meant having to declare the type in a nested package, to avoid problems with having the call Current_Variable dispatching on more than one type, which is illegal. It would have been nice to do the same for Directory_Entry_Type, and make it a tagged type but that type already existed and changing that to be a tagged type would likely break backwards incompatibility in some way. !ACATS test ** TBD. !appendix From: Brad Moore Sent: Saturday, October 1, 2011 1:37 PM !topic Ada 2012 Iterators for Directory Searches in Ada.Directories !reference Ada 2012 RM A.16 (32/2, 36/2) !from /Brad Moore 11-10-01/ !keywords Iterators !discussion Ada 2012 introduces new iterator syntax for containers. A directory can be viewed as a container for files. The Ada.Directories package provides the Search related subprograms to iterate through the files in a directory. It seems we ought to be able to use the new iterator syntax to also iterate through the files in a directory. This would be more convenient, and express the iteration in a more natural way, similar to how the new syntax is helpful for iterating through containers. **************************************************************** From: Brad Moore Sent: Saturday, October 1, 2011 2:28 PM !topic Ada 2012 iterators for environment variables !reference Ada 2012 RM A.17 (8/3) !from /Brad Moore 11-10-01/ !keywords Iterators, Environment Variables !discussion Ada 2012 introduces new syntax for iterating through containers. The package Ada.Environment_Variables has a subprogram called Iterate that is used to iterate through the environment variables. It seems we ought to be able to use the new iterator syntax to iterate through environment variables as well. This likely would be more convenient and provide a more natural means of expressing this functionality similar to how the new syntax benefits iterating through containers. **************************************************************** From: Brad Moore Sent: Tuesday, September 29, 2015 9:15 PM I have been working on solutions to this AI, and have encountered a bit of a dilemma. I started looking at Ada.Directories, and found I was able to take the existing GNAT implementation, and get a working prototype by adding just the following to the public part of the package specification. type Cursor (Directory_Entry : not null access constant Directory_Entry_Type) is private with Implicit_Dereference => Directory_Entry; function Has_Element (Directory_Entry : Cursor) return Boolean; package Directory_Iterator_Interfaces is new Ada.Iterator_Interfaces (Cursor => Cursor, Has_Element => Has_Element); function Entries (Directory : String; Pattern : String; Filter : Filter_Type := (others => True)) return Directory_Iterator_Interfaces.Forward_Iterator'Class; I found the Implicit_Dereference aspect helpful here, as it allowed me to create a cursor that can designate the limited type Directory_Entry_Type. This allowed me to write: for Dir_Entry in Ada202x.Directories.Entries ("/home/brad", "*.txt") loop Put_Line(Directories.Simple_Name (Dir_Entry)); end loop; Where I have defined an iterator_name, rather than an iterable_name (see 5.5.2). Other than using the Rosen Trick, the implementation was quite easy and uses existing library functions defined by the standard. Looking at the meeting minutes, it was suggested that I look at having the Entries function return an array of Directory_Entry_Types, rather than an iterator object. I suspect this is partly because noone had yet considered using the Implicit_Dereference aspect as I have. The main difference to the programmer between what was suggested and what I implemented would be that the user would need to use "of" instead of "in" in the loop iterator specification. ie. for Dir_Entry of Ada202x.Directories.Entries ("/home/brad", "*.txt") loop Put_Line(Directories.Simple_Name (Dir_Entry)); end loop; Not a huge difference. But there are some others. Returning an array could be a problem if the directory consisted of many files. A few thousand files might cause stack overflow. The solution I came up with however scales to any number of files because the object being returned is just an iterator object, not an array. The iterator object only contains a single Directory_Entry_Type object component at a time. Another difference is that I think perhaps it is better to avoid "implicit" containers. There is no array object that the programmer can see. I think using iterable_names are good to use when you have an explicit container object declaration in the programmers code, but otherwise, an iterator_name seems more appropriate when there is no such explicit container. On the other hand, this (and the environment variables package which could be handled the same way) would be the first standard containers where the new Ada 2012 iterators are designed mainly for "in" loops rather than "of" loops. I think considering all the above, I am still leaning towards what I have, but wanted to get others opinions. Which would be better? Use "in" instead of "of" in the loop iterators, returning an iterator object rather than an array? Or the other way round? If it is helpful to look at the rest of the implementation I came up with, here is what I added to the private part of the specification: type Cursor (Directory_Entry : not null access constant Directory_Entry_Type) is record Done : Boolean; -- Control : Reference_Control_Type := -- raise Program_Error with "uninitialized reference"; -- The RM says, "The default initialization of an object of -- type Constant_Reference_Type or Reference_Type propagates -- Program_Error." end record; function Has_Element (Directory_Entry : Cursor) return Boolean is (not Directory_Entry.Done); type Directory_Iterator; type Holder (Reference : access Directory_Iterator) is limited null record; type Directory_Iterator is limited new Directory_Iterator_Interfaces.Forward_Iterator with record -- Rosen trick needed so that First and Next can -- modify the iterator object Self : Holder (Directory_Iterator'Access); Search : Search_Type; Directory_Entry : aliased Directory_Entry_Type; end record; overriding function First (Object : Directory_Iterator) return Cursor; overriding function Next (Object : Directory_Iterator; Value : Cursor) return Cursor; and in the body, I added.... function Entries (Directory : String; Pattern : String; Filter : Filter_Type := (others => True)) return Directory_Iterator_Interfaces.Forward_Iterator'Class is begin return It : Directory_Iterator do It.Done := False; Start_Search (Search => It.Search, Directory => Directory, Pattern => Pattern, Filter => Filter); end return; end Entries; function First (Object : Directory_Iterator) return Cursor is Result : Cursor (Directory_Entry => Object.Directory_Entry'Unchecked_Access); begin Get_Next_Entry (Search => Object.Self.Reference.Search, Directory_Entry => Object.Self.Reference.Directory_Entry); return Result; end First; function Next (Object : Directory_Iterator; Value : Cursor) return Cursor is Result : Cursor (Directory_Entry => Object.Directory_Entry'Unchecked_Access); begin if More_Entries (Search => Object.Self.Reference.Search) then Get_Next_Entry (Search => Object.Self.Reference.Search, Directory_Entry => Object.Self.Reference.Directory_Entry); else End_Search (Search => Object.Self.Reference.Search); Result.Done := True; end if; return Result; end Next; **************************************************************** From: Ed Schonberg Sent: Sunday, October 4, 2015 9:51 AM > Which would be better? > Use "in" instead of "of" in the loop iterators, returning an iterator > object rather than an array? Or the other way round? I think you make a good argument that using an iterator object is preferable to constructing a temporary array. This points to a construct that has become very popular in other languages: the generator. Of course we can write generators in Ada, but there is no single syntax that makes the idiom immediately available. Maybe some new aspect for a protected function? It would be great to have a simple way to describe a function where successive calls generate a stream of values from some collection, without building the collection explicitly. Language design invited… **************************************************************** From: Tucker Taft Sent: Sunday, October 4, 2015 8:34 PM > I think you make a good argument that using an iterator object is > preferable to constructing a temporary array. This points to a > construct that has become very popular in other languages: the generator. > Of course we can write generators in Ada, but there is no single syntax that > makes the idiom immediately available. I would think that an iterator object, or an iterable collection object, provides what you want. Can you give an example of what a generator would do that is not accomplishable with the First/Next sequence of iterators? > ... Maybe some new aspect for a protected function? I am not sure I understand why a protected function would be involved. > ... It would be great to have > a simple way to describe a function where successive calls generate a > stream of values from some collection, without building the collection > explicitly. Language design invited… I would certainly hope we could use one of the existing syntaxes "for I in XXX loop" or "for I of YYY loop" to accomplish this. The interesting question is what these would expand into for a "generator" vs. an "iterator" (hard for me since I am unsure of the difference). **************************************************************** From: Randy Brukardt Sent: Monday, October 5, 2015 6:08 PM ... > I would think that an iterator object, or an iterable collection > object, provides what you want. Can you give an example of what a > generator would do that is not accomplishable with the First/Next > sequence of iterators? I was equally puzzled by Ed's message. The only thing I can think of is generating objects without some sort of Cursor (pointer) involved. But that only works if: (A) Objects in the stream are not modifiable, as in a functional language. (You have to have some sort of access to the object in order to modify it within the loop.) (B) The order that the objects are to be returned is inherent in the data structure. (The cursor provides a "next" location; it's hard to imagine many uses where that isn't necessary.) And even then, the implementation of the object stream probably will need some sort of pointer, so using it as a cursor isn't a real issue. In the vast majority of cases, you're going to need the operations of a forward iterator, and since there are only two operations to implement, it's hard to imagine how it could be made much simpler. **************************************************************** From: Ed Schonberg Sent: Tuesday, October 6, 2015 8:42 AM What generators provide is lazy evaluation, when you are searching over a potentially unbounded collection and you want to generate its values one by one. Think of a search over prime numbers, or a crossword puzzle solver that will show you words of 9 letters when the middle one is a Z! You don't need to precompute the set, you want to have a construct with local memory that embodies the computation to go from one element of the set to the next. In Brad's example, the set of values exists in the directory so you can use cursors, but that may not be the case: in GPS for example there is a typing completion algorithm that generates dynamically suggestions based on keystrokes so far. It is not possible to precompute a set of completions in advance, and what you need is to encapsulate that set in a generator that will produce successive candidates on demand. **************************************************************** From: Tucker Taft Sent: Tuesday, October 6, 2015 10:03 AM > ... It is not possible to precompute a set of completions in advance, > and what you need is to encapsulate that set in a generator that will produce > successive candidates on demand. Is there something that requires an iterator to be static? Couldn't the call on "Next" compute or retrieve the next value? **************************************************************** From: Ed Schonberg Sent: Tuesday, October 6, 2015 10:21 AM Yes, but either the caller saves the previous value and Next takes it as an argument, or else Next needs local memory, which is why coroutines come up in these discussions. This is why I mentioned protected types earlier: the simplest realization of that model in current Ada is a protected object with one function and some local storage. **************************************************************** From: Randy Brukardt Sent: Tuesday, October 6, 2015 11:39 AM > Yes, but either the caller saves the previous value and Next takes it > as an argument, Which of course is exactly the model of Ada's iterators. For this case, the "cursor" would be whatever is needed to understand the previous value. > or else Next needs local memory, The iterator object provides the local state. (Next having two parameters.) > which is why coroutines come up in these discussions. This is why I > mentioned protected types earlier: the simplest realization of that > model in current Ada is a protected object with one function and some > local storage. Actually, the existing iterators map it pretty well so far as I can tell. I recall Bob asking about lazy evaluation of a list during the early design, and the intent was that they could be used that way. Perhaps a bit more complex than a single function, but not particularly bad. And of course, if you don't want the for loop syntax, you can just use a function and some local state. I don't see anything else that the language could do to help you in that case (I don't think Ada is getting lazy evaluation or coroutines anytime in the near future!). **************************************************************** From: Tucker Taft Sent: Monday, October 5, 2015 7:30 PM ... > Other than using the Rosen Trick, the implementation was quite easy > and uses existing library functions defined by the standard. What happens if you copy the value of a cursor and save it? It seems the cursor always points at the "current" directory entry. > Looking at the meeting minutes, it was suggested that I look at having > the Entries function return an array of Directory_Entry_Types, rather > than an iterator object. I suspect this is partly because noone had > yet considered using the Implicit_Dereference aspect as I have. > > The main difference to the programmer between what was suggested and > what I implemented would be that the user would need to use "of" > instead of "in" in the loop iterator specification. > > ie. > > for Dir_Entry of Ada202x.Directories.Entries ("/home/brad", "*.txt") loop > Put_Line(Directories.Simple_Name (Dir_Entry)); end loop; > > Not a huge difference. But there are some others. Returning an array > could be a problem if the directory consisted of many files. A few > thousand files might cause stack overflow. Do you mean secondary stack overflow, because these objects would almost certainly not reside on the "primary" stack? That seems relatively unlikely. Alternatively, did you consider creating a function that returns a (conceptual) directory-entry "container" rather than an actual array, which could be allocated on the heap, and use a cursor object containing a reference-counted pointer to the container, and an index into the container? > The solution I came up with however scales to any number of files > because the object being returned is just an iterator object, not an > array. The iterator object only contains a single Directory_Entry_Type > object component at a time. True, but then the "cursor" is a bit of a misnomer, as it is really just a level of indirection. > Another difference is that I think perhaps it is better to avoid > "implicit" containers. There is no array object that the programmer > can see. I had presumed that the function would be returning a container that was indexable, so in what sense is that "implicit"? It is a temporary, that is true, though presumably it could be built in one place and iterated multiple times. > ... I think using iterable_names are good to use when you have an > explicit container object declaration in the programmers code, but > otherwise, an iterator_name seems more appropriate when there is no > such explicit container. > > On the other hand, this (and the environment variables package which > could be handled the same way) would be the first standard containers > where the new Ada 2012 iterators are designed mainly for "in" loops rather than "of" loops. > > I think considering all the above, I am still leaning towards what I > have, but wanted to get others opinions. > > Which would be better? > Use "in" instead of "of" in the loop iterators, returning an > iterator object rather than an array? Or the other way round? ... I think "in" or "or" is not that big of an issue. The more important question might be what are the semantics of saving the (temporary) container, or saving a cursor? Does a cursor end up as a dangling reference. Do all cursor values point at the same object? **************************************************************** From: Randy Brukardt Sent: Monday, October 4, 2015 6:19 PM ... > The main difference to the programmer between what was suggested and > what I implemented would be that the user would need to use "of" > instead of "in" in the loop iterator specification. Actually, the main difference is that the implementation-defined behavior when a directory is altered is avoided. (See A.16(110/3).) The array would clearly be a snapshot of the directory at the time it is taken, so it wouldn't change afterwards. Also, as Tucker notes, copying indexes into the array would be harmless. Copying one of your cursors would most likely leave it being a dangling pointer after the completion of the iteration. Sounds like more erroneous execution possibilities (which I'd prefer to avoid). I could see ways to avoid cursors becoming erroneous if someone copies them, but they all boil down to making an "implicit container", which you say you don't want to do. **************************************************************** From: Brad Moore Sent: Thursday, October 8, 2015 1:23 AM >> This allowed me to write: >> >> for Dir_Entry in Ada202x.Directories.Entries ("/home/brad", "*.txt") loop >> Put_Line(Directories.Simple_Name (Dir_Entry)); end loop; >> > > What happens if you copy the value of a cursor and save it? It seems > the cursor always points at the "current" directory entry. > I was kind of hoping that the accessibility rules would somehow prevent squirreling a temporary access value from an inner scope to an outer scope. The cursor's discriminant is an access to a Directory_Entry_Type component of the iterator object. I tried this with GNAT however, and it didn't complain. Copying inside the loop body should be OK since any copy would be expected to designate the same "current" directory entry. Shouldn't the accessibility checks prevent such copying to variables declared in a global scope? declare DE : aliased Directories.Directory_Entry_Type; Squirrel : Directories.Cursor (DE'Access); begin for Dir_Entry in Ada202x.Directories.Entries ("/home/brad", "*.txt") loop Put_Line (Directories.Simple_Name (Dir_Entry)); Squirrel := Directories.Cursor'(Dir_Entry); end loop; end; **************************************************************** From: Randy Brukardt Sent: Thursday, October 8, 2015 1:07 PM ... > Shouldn't the accessibility checks prevent such copying to variables > declared in a global scope? > > declare > DE : aliased Directories.Directory_Entry_Type; > Squirrel : Directories.Cursor (DE'Access); begin > > for Dir_Entry in Ada202x.Directories.Entries ("/home/brad", "*.txt") loop > Put_Line (Directories.Simple_Name (Dir_Entry)); > Squirrel := Directories.Cursor'(Dir_Entry); > end loop; > end; The above is fine for accesibility (I think), but it should raise Constraint_Error because you can't change the discriminant of Squirrel (thus the discriminant check should fail). Try: (I didn't compile this) declare type Outer is access all Directories.Directory_Entry_Type; Squirrel : Outer; begin for Dir_Entry in Ada202x.Directories.Entries ("/home/brad", "*.txt") loop Put_Line (Directories.Simple_Name (Dir_Entry)); Squirrel := Dir_Entry.D; -- D should be whatever the name of your discriminant is. end loop; end; ... >I was kind of hoping that the accessibility rules would somehow prevent squirreling >a temporary access value from an inner scope to an outer scope. The cursor's >discriminant is an access to a Directory_Entry_Type component of the iterator object. They're supposed to do that. You shouldn't be able to copy the discriminant to any named access type that is outside of the loop. But Tucker's point is that you could squirrel them between iterations, since all of the iterations belong to a single master. The good news is I can't figure out how to write such a thing [if you add a block, you're at a different level]; maybe with a Holder container and a subprogram? Or maybe it's impossible and Tucker is just throwing FUD. **************************************************************** From: Tucker Taft Sent: Thursday, October 8, 2015 4:40 PM There are a couple of subtle issues here. Not surprising because we are tip-toeing very near the heart of darkness (3.10.2). I think we are probably OK, but convincing myself of that was not easy. The loop parameter Dir_Entry is of the Cursor type, and it is *initialized* by a call on First, so no constraint check. The subsequent iterations have Dir_Entry being *assigned* from the result of Next, so here there *would* be a constraint check, presumably, and if the access discriminant pointed elsewhere, then that should fail the discriminant check. So in fact, if you are going to use this approach, then the access discriminant of the cursor object must *always* point to the same object. A bit weird! As far as squirreling away, I agree that the accessibility level of the access discriminant's type should be the same as that of the loop parameter (per 3.10.2(12.5/3)), so GNAT should disallow the assignment to an access-type variable outside the loop if you are assigning from the access discriminant only, because of the required implicit or explicit conversion (per 4.6(24.17/4)); it should fail a discriminant check if you assign from the loop parameter as a whole. It should also disallow using the value of the loop parameter, or the access discriminant thereof, as the initial value in an initialized allocator for an access type declared outside of the loop, per 4.8(5.3/3, 10.1/3). So this looks OK, but it shows some issues with GNAT's discriminant checks (unless you are suppressing them, or you compiled but did not actually run the code). It will be interesting to see whether GNAT rejects Randy's example, where you are assigning from the access discriminant itself. After all of this, I am somewhat uncomfortable using this trick to provide an iterator. Alternatively, we should bless this approach and recommend it for other cases where you want to use the iterator syntax but not actually use the cursor as an index into anything, but rather just use it as a level of indirection into the iterator object, which has the current value stored within it. Calling it a "cursor" is of course a bit confusing then. It is more like a "handle" on the current object of the iteration, which is hidden somewhere within the iterator object itself (in this case) or something it references (in the container iterator case). We are relying on the access discriminant to prevent squirreling things away (thanks to the discriminant check), as well as giving us the implicit dereference semantics. I would be somewhat more comfortable if we hid the access discriminant, and perhaps the whole "cursor" notion completely. What would happen if we turned this into a container element iterator, by changing "Entries" to be something that could follow "of" rather than "in"? That is, "Entries" would return an iterable object that has a Constant_Indexing aspect. The Constant_Indexing aspect would return the "current" Directory_Entry (or a const reference to the current Directory_Entry if we want to avoid copying a large Directory_Entry), pretty much independent of the cursor value passed in. The cursor could be effectively a Boolean value, indicating whether or not there is another directory entry to be processed, so Has_Element could simply return the Boolean value of the cursor, and the Constant_Indexing attribute could raise an exception if passed a cursor with value False. For example: type Entry_Presence is new Boolean; -- the "cursor" type function Has_Element(EP : Entry_Presence) return Boolean is (Boolean(EP)); package Dir_Iterators is new Ada.Iterator_Interfaces(Entry_Presence, Has_Element); type Dir_Entry_Iterator is -- The iterable container type limited new Dir_Iterators.Forward_Iterator with Constant_Indexing => Current_Entry, Default_Iterator => Entries, Iterator_Element => Dir_Entry; function Entries -- Returns an iterable container (Dir_Name : String; Pattern : String) return Dir_Entry_Iterator; function First -- One of the forward iterator operations (Entries: Dir_Entry_Iterator) return Entry_Presence; function Next -- One of the forward iterator operations (Entries: Dir_Entry_Iterator; Prev : Entry_Presence) return Entry_Presence; function Current_Entry -- "indexing" operation (Entries: Dir_Entry_Iterator; EP : Entry_Presence) return Dir_Entry with Pre => Has_Element(EP); -- or could return an object that has an access discrim -- e.g.: -- type Dir_Ent_Ref(DE : not null access constant Dir_Entry) ... for Dir_Ent of Entries("/homes/stt", "*.txt") loop ... end loop; This would preserve the model of a directory as a vector of directory entries, while still allowing us to implement the iteration without building up that vector explicitly. It also eliminates the visible use of a reference object as a cursor, and the concerns about an attempt at squirreling away the cursor causing a run-time constraint error or a potentially confusing accessibility error. **************************************************************** From: Tucker Taft Sent: Thursday, October 8, 2015 5:00 PM I think I might have gotten myself confused with the "Default_Iterator" aspect. Stay tuned for another version of the actual source code (I might actually try to compile it next time ;-). I think the basic idea should still work, allowing us to use "of" rather than "in" and avoid some of the accessibility check issues. I just need to get my "iterable container" and "default iterator" and "iterator element" aspects all straightened out... **************************************************************** From: Tucker Taft Sent: Thursday, October 8, 2015 5:13 PM OK, the change is to make the Default_Iterator aspect denote an identity function, which takes the Dir_Entry_Iterator and returns it, since the Dir_Entry_Iterator is acting as both our "iterable container type" and as an "iterator type." Hence: type Dir_Entry_Iterator is -- The iterable container type new Dir_Iterators.Forward_Iterator with Constant_Indexing => Current_Entry, Default_Iterator => Identity, Iterator_Element => Dir_Entry; function Identity(Iter: Dir_Entry_Iterator) return Dir_Entry_Iterator is (Iter); Note that because we wanted to define the body of Identity using a simple expression function, rather than an aggregate, we had to change Dir_Entry_Iterator to be a non-limited type. Everything but the Default_Iterator and limitedness of Dir_Entry_Iterator remains the same. Sorry for the confusion. **************************************************************** From: Brad Moore Sent: Thursday, October 8, 2015 8:00 PM ... > As far as squirreling away, I agree that the accessibility level of > the access discriminant's type should be the same as that of the loop > parameter (per 3.10.2(12.5/3)), so GNAT should disallow the assignment > to an access-type variable outside the loop if you are assigning from > the access discriminant only, because of the required implicit or > explicit conversion (per 4.6(24.17/4)); it should fail a discriminant > check if you assign from the loop parameter as a whole. It should > also disallow using the value of the loop parameter, or the access > discriminant thereof, as the initial value in an initialized allocator > for an access type declared outside of the loop, per 4.8(5.3/3, 10.1/3). > > So this looks OK, but it shows some issues with GNAT's discriminant > checks (unless you are suppressing them, or you compiled but did not > actually run the code). Sorry, it was late last night, when I was trying this out. I compiled but did not run the code. When I run it today, I see the Constraint_Error being raised, as expected. **************************************************************** From: Brad Moore Sent: Thursday, October 8, 2015 11:46 PM I took your suggestion and ran with it, but found there were some further tweaks required. In particular, I found that I had to change Dir_Entry_Iterator back to being a limited type. This is because I needed to be able to modify an "in" parameter (in calls such as First, and Next), so I had to use the Rosen trick to do this. However, this required the type to be limited. As a result of making it limited, I had to involve two separate objects, the Container object and the Iterator object. Both are still the same type, (and maybe that's confusing). Because Identity needs to return a limited type, it cannot just return its "Iter" parameter. It has to construct a new return object from the "Iter" parameter. I decided to put the storage for the Directory_Entry in the Iterator object, but the Current_Entry function receives the Container object, not the Iterator object, and needs to ultimately get to the Iterator object, so I had to store an extra access value from the container to access the iterator object. Only the container object has this access value. The iterator object sets its own value for this component to null. All in all, the implementation was a bit tricky to get working, but it does seem to work. I have it compiling and executing with proper results in GNAT. The new content added to the public part of the Ada.Directories package is; type Entry_Presence is new Boolean; -- the "cursor" type function Has_Element (EP : Entry_Presence) return Boolean is (Boolean (EP)); package Directory_Iterator_Interfaces is new Ada.Iterator_Interfaces (Cursor => Entry_Presence, Has_Element => Has_Element); package Dir_Iterators is new Ada.Iterator_Interfaces (Entry_Presence, Has_Element); type Dir_Entry_Iterator is limited -- The iterable container type new Dir_Iterators.Forward_Iterator with private with Constant_Indexing => Current_Entry, Default_Iterator => Identity, Iterator_Element => Directory_Entry_Type; overriding function First (Iter : Dir_Entry_Iterator) return Entry_Presence; overriding function Next (Iter : Dir_Entry_Iterator; Value : Entry_Presence) return Entry_Presence; function Current_Entry -- "indexing" operation (Entries : Dir_Entry_Iterator; EP : Entry_Presence) return Directory_Entry_Type with Pre => Has_Element (EP); function Identity (Iter : Dir_Entry_Iterator) return Dir_Entry_Iterator; function Entries -- Returns an iterable container (Directory : String; Pattern : String; Filter : Filter_Type := (others => True)) return Dir_Entry_Iterator; I think I agree with you that this is probably a better approach than the "in" syntax version. **************************************************************** From: Brad Moore Sent: Saturday, October 10, 2015 12:26 PM I was able to go further and hide the "noise" parts of the iterator mechanism in the private part of the package. I ended up creating two separate types to distinguish the container from the iterator. The Container is a type called "Directory_Listing", and the Iterator type is defined in the private part of the package. I was able to move the "Identity" function and the "Current_Entry" function into the private part of the package as well, since the current freezing rules allowed me to do this. I think in this case, it a nice feature because those two functions are not intended to be called by programmers, and are only needed to support the iterator mechanism. Thus, I was able to reduce the additions needed for to the public part of the package to just; private with Ada.Iterator_Iterfaces; ... type Directory_Listing is tagged limited private with Constant_Indexing => Current_Entry, Default_Iterator => Identity, Iterator_Element => Directory_Entry_Type; function Entries (Directory : String; Pattern : String; Filter : Filter_Type := (others => True)) return Directory_Listing; **************************************************************** From: Brad Moore Sent: Saturday, October 10, 2015 5:59 PM Here is my writeup for the AI12-0009-1 [This is version /02 of the AI - Editor.] **************************************************************** From: Tucker Taft Sent: Saturday, October 10, 2015 8:44 PM ... > I think in this case, it a nice feature because those two functions > are not intended to be called by programmers, and are only needed to > support the iterator mechanism. Thus, I was able to reduce the > additions needed for to the public part of the package to just; > > private with Ada.Iterator_Iterfaces; > ... > > type Directory_Listing is tagged limited private > with Constant_Indexing => Current_Entry, > Default_Iterator => Identity, > Iterator_Element => Directory_Entry_Type; I don't think you can refer to a declaration that occurs in the private part in an aspect, if the aspect specification is in the visible part. See 13.1.1(11/3): "The usage names in an aspect_definition are not resolved at the point of the associated declaration, but rather are resolved at the end of the immediately enclosing declaration list." **************************************************************** From: Randy Brukardt Sent: Saturday, October 4, 2015 9:53 PM Yup. I just finished looking that up to write essentially the same message that you did. But I see you beat me to it, between the time I read Brad's message on my phone and the time I arrived in the office to answer it (but mainly to reprogram my handheld GPS for the northeast). **************************************************************** From: Bob Duff Sent: Sunday, October 11, 2015 6:11 AM > I don't think you can refer to a declaration that occurs in the > private part in an aspect, if the aspect specification is in the visible part. > See 13.1.1(11/3): You can in older versions of GNAT. This bug was fixed fairly recently. **************************************************************** From: Randy Brukardt Sent: Monday, October 12, 2015 3:11 PM There really ought to be an ACATS test for that. It's one of the higher priority untested things; any volunteers?? **************************************************************** From: Brad Moore Sent: Sunday, October 11, 2015 12:22 PM > I don't think you can refer to a declaration that occurs in the > private part in an aspect, if the aspect specification is in the visible part. > See 13.1.1(11/3): > > "The usage names in an aspect_definition are not resolved at the > point of the associated declaration, but rather are resolved at the > end of the immediately enclosing declaration list." > That's somewhat unfortunate as it means having to pull in more declarations from the private part that are mostly of no concern to the programmer, and having to deal with unnecessary use cases. For example for the function, function Current_Entry (Entries : Directory_Listing; EP : Entry_Presence) return Directory_Entry_Type with Pre => Has_Element (EP); we have to now deal with the possibility that a user might call this function explicitly with EP having a value of True, while the state of the Directory_Listing container might be in a state where the current directory entry is not present. It makes me think that there needs to be some capability to declare that a subprogram is not explicitly callable by the programmer, and only callable by the implementation. I'm thinking of a User_Callable aspect that can be applied to the subprogram as in; function Current_Entry (Entries : Directory_Listing; EP : Entry_Presence) return Directory_Entry_Type with Pre => Has_Element (EP), User_Callable => False; This would hopefully would mean that less semantics and wording for such a function would be needed in the RM. I think this helps also to separate the real public interface from the bits and pieces that are not intended to be called by user. **************************************************************** From: Brad Moore Sent: Sunday, October 11, 2015 3:09 PM Here is an updated version to replace what I submitted yesterday. [This is version /03 of the AI - Editor.] This version eliminates the "bug" where I was assuming I could defer the Contant_Indexing function and Default_Iterator functions to the private part of the packages. This version pulls the necessary declarations from the private part back into the public part. I have tested this implementation for the Ada.Directories package, and have a working implementation. I also renamed Identity to Iterate, as Identity is no longer true and Iterate is more consistent with the naming we use on other existing containers. For now, disregard my comment in my previous email about providing a User_Callable aspect. I don't think it is warranted here. I think we want to expose Iterate and Current_Entry as user callable. This allows one to write both forms of generalized iterator loops. (Using "in" or "of" in the loop iterator_specification) **************************************************************** From: Randy Brukardt Sent: Monday, October 12, 2015 3:43 PM > Here is an updated version to replace what I submitted yesterday. > This version eliminates the "bug" where I was assuming I could defer > the Contant_Indexing function and Default_Iterator functions to the > private part of the packages. I corrected a couple of typos in this as I was posting it. Specifically, ... > I also renamed Identity to Iterate, as Identity is no longer true and > Iterate is more consistent with the naming we use on other existing > containers. The Default_Iterator aspect was still set to Identity in A.16(36/3). You certainly meant that to be Iterate as well. Also, !question should be !problem in an !Amendment class AI. **************************************************************** From: Jeff Cousins Sent: Sunday, January 3, 2016 1:59 PM > Jeff Cousins: AI12-0009-1 (check wording in /03 version) There seems to have been a generalised replacement of key-words by their bold lower-case version, this has resulted in some sentences beginning with a lower-case letter, e.g.: for generalized loop iteration if Pattern is the null string when Start_Search propagates At the meeting we said that type Entry_Presence should be private not new Boolean. Presumably the same for type Variable_Presence too. type Directory_Listing The description of this seems a bit short compared with that for type Environment_Variable_Listing. Maybe insert a middle sentence "For generalized loop iteration (see 5.5.2), an object of the type Directory_Listing can be an iterable container object for the loop that will generate a forward container element iterator as the loop iterator and a loop cursor designating each environment variable of the execution environment." Otherwise it looks a bit odd referring to a cursor in the descriptions of Entries and Iterate when the cursor hasn't yet been introduced to the reader. function Current_Entry There are rather a lot of "and"s and "or"s in the first sentence, maybe it needs a few commas. type Environment_Variable_Listing is tagged limited private with Constant_Indexing => Current_Entry, should be type Environment_Variable_Listing is tagged limited private with Constant_Indexing => Current_Variable, Personally, I would prefer All_Environment_Variables to All_Variables. function Current_Variable Presumably the return type should be Name_Value_Pair_Type not Name_Value_Pair. type Name_Value_Pair_Type Spurious } in the description. **************************************************************** From: Randy Brukardt Sent: Thursday, January 7, 2016 10:39 PM Like Jeff, I had an action item to review the wording in AI12-0009-1. Presumably Brad will use our comments to create his next update. I'm not going to look too carefully at Jeff's comments; Brad can get to reconcile them (that's a lot of fun and I'm glad I don't have to do it here). [BTW, when I say "you" in here, I'm talking about the AI author, that is, Brad.] As discussed, Entry_Presence should be private. Replace: type Entry_Presence is new Boolean; function Has_Element (EP : in Entry_Presence) return Boolean is (Boolean (EP)); By: type Entry_Presence is private; function Has_Element (EP : in Entry_Presence) return Boolean; Note that we'll need a definition of what Has_Element does somewhere. BTW, I'd rename it to Has_Entry, since there is nothing named "Element" around here, just a bunch of Entries. Later, you have: function Entries (Directory : in String; Pattern : in String; Filter : in Filter_Type := (others => True)) return Directory_Listing; For generalized loop iteration (see 5.5.2), Entries returns an iterable container object for the loop that will generate a forward container element iterator as the loop iterator and a loop cursor designating each available directory entry in the return object. ... I don't understand this sentence. Entries does whatever it does in any context, so I don't see what "generalized loop iteration" has to do with anything. Directory_Listing is an iterable container object by definition, and how that works is defined by the Ada language (Iterate is called, etc.). So this is the sort of "says nothing" sentence that we're trying to get rid of in the Standard. I think you want to say something introductory rather than normative to start with, maybe something like: Function Entries returns an object that can be used in generalized loop iteration (see 5.5.2) to visit each available directory entry. And then continue with the details of how that gets done. I see Jeff suggested going in the other direction. I don't agree with that, but he might be right that the introductory text is better put on the type Directory_Listing. That is, something like: An object of type Directory_Listing can be used in generalized loop iteration (see 5.5.2) to visit each available directory entry for a particular directory search. We certainly don't need to say anything about how that happens; that's obvious from the aspects declared on the type declaration (at least if you're familiar with 5.5.1 and 5.5.2, and we can assume that in the RM). Then the description of Entries should echo that of Start_Search: Returns an object that represents the result of a search in the directory named by Directory for entries matching Pattern and Filter. As always, Simplify! (That's a lot easier to say as a reviewer than as an author. So please don't throw that back at me on my next AI ;-) --- function Current_Entry: ... or is not involved in generalized loop iteration ... Huh?? What does this mean? The default state is that there is no entries, so if it's not iterating, there are no entries. So there doesn't seem to be any need to mention this (which is good, 'cause it's nonsense - we don't care how the object is used, just that the sequence of calls makes sense). --- A.16(125.a/3): Implementation Advice: Directories.Start_Search{,} [and ] Directories.Search{, and Entries} should raise Name_Error for malformed patterns. The existing routines here give the package name, and the new one ought to as well - "Directories.Entries". This is the entry in the Implementation Advice annex, and a reader of that will not have any idea where this comes from, thus we have to be explicit. --- As Jeff mentions, "Variable_Presence" should be changed like "Entry_Presence" was above. And I'd change Has_Element to "Has_Pair" or "Has_Variable" or something like that. --- type Name_Value_Pair_Type is limited private; The type Name_Value_Pair_Type represents a single environment variable of the execution environment. These items can only be created by generalized loop iteration on the return object of the All_Variables function in this package}. Information about the item can be obtained from the functions declared in this package. A default-initialized object of this type is invalid; objects returned by generalized loop iteration on the result of Entries are valid. "These items can only be created by generalized loop iteration..."?? Huh? It looks to me like you can get some by calling All_Variables and then calling Current_Variable. As usual, anything that generalized iteration can do can be done explicitly! I'd probably focus on the use (which you did in the first sentence); the second and third sentences are obvious from the specification, so I'd just drop them. --- Environment_Variable_Listing: Again, simply and point out that it *can* be used in a generalized loop iteration, but surely there is no requirement to do so. And repeating the whole business about how those work doesn't add anything not in the spec. The less English, the better! My comments on "Entries" apply to "All_Variables" as well. --- The wording for Current_Variable depends on the previous fact that Variable_Presence is Boolean rather than private. And again, what the heck does "involved in generalized loop iteration" mean? What matters is the sequence of calls to All_Variables, Iterator, Current_Variable, and so on. --- A.17(25): you've got "involve" on the brain! Just say "have": It is a bounded error to call Value {if the call does not have a parameter of type Name_Value_Pair_Type,} if more than one environment variable exists with the given name; the possible outcomes are that: --- A.17(28/2): Again, you're ignoring the possibility of explicitly calling routines to create and use an object of Environment_Variable_Listing. It's irrelevant how the object got created or how it is being used, you'll need some abstract way to describe the state of when an object of that type is in use (between a call to Iterate and when Has_Pair becomes False). --- Jeff wrote: >There seems to have been a generalised replacement of key-words by >their bold lower-case version, this has resulted in some sentences >beginning with a lower-case letter, e.g.: > > for generalized loop iteration > > if Pattern is the null string > > when Start_Search propagates That's the same problem J-P complained about for AI12-0144-1. The autoformatter for AIs cannot handle the format of typical library subprogram definitions, as it starts with some code and then is followed by some indented text. *Never* worry about the formatted output -- *always* look at the raw text version before complaining about the "pretty" format. **************************************************************** From: Brad Moore Sent: Saturday, May 28, 2016 3:02 PM I have been looking at my homework regarding adding iterators to Ada.Directories. In short, I have a new implementation and API that I would like to propose. If this new API is favoured, I would write up a new version of the AI to match this API. In Vermont, Steve commented that the Entry_Presence type should be a private type. This type represents the cursor for the directory listing, and currently was implemented as a Boolean type. I found it improved the abstraction quite a bit to rename this type to be "Cursor" (as well as make it private), then it fits better with existing container types, where the Cursor is used to iterate through the directory listing, rather than a Boolean type. I then went further, and thought about someone wanting to iterate through the directory in a particular order, or to have better control over which files are iterated over. This led me to a new prototype, where one can optionally specify a sort order for the results, and a filter that gives much more control than the existing one, which only lets you specify file type. I have implemented this prototype in GNAT, and it appears to work. Also, it seems worth noting that my existing test program did not require any changes to switch over to this new API. Internally, the implementation changed from just wrapping calls to Search, to fetching a vector of matching directory entries, and storing that list in the Directory_Listing type. This also allows one to query how many items match the search criteria, via a Length subprogram. The Directory_Listing type ends up being a finalized type that looks more like one of the other standard Containers. A couple of other notes: I added Preelaborable_Initialization to both the Directory_Listing and the Cursor type, which wasn't in the previous writeup of the AI. Also. I think with Ada.Finalization; should be changed to with private Ada.Finalization; Since all the finalization details are all in the private part. With these changes, the additions to the Ada.Directories package would look like the following; -- Searches in the directory named by Directory for entries matching -- Pattern. The subprogram designated by Process is called with each -- matching entry in turn. Pattern represents a pattern for -- matching file names. If Pattern is null, all items in the -- directory are matched; -- otherwise, the interpretation of Pattern is implementation- -- defined. -- Only items that match Filter will be returned. A null Filter -- does not apply a filter, otherwise only items in the directory -- that return True from the filter, are included in the search. -- The ordering of the results is determined by the Sorting -- parameter. -- By default, the results are sorted in alphanumeric order, but the -- Sorting parameter can be specified which returns True if the Left -- Item should appear bnefore the Right item in the sort order. -- The exception Name_Error is propagated if the string given by -- Directory does not identify an existing directory, or if Pattern -- does not allow the identification of any possible external file -- or directory. -- The exception Use_Error is propagated if the external -- environment does not support the searching of the directory with -- the given name (in the absence of Name_Error). type Directory_Listing is tagged limited private with Preelaborable_Initialization, Constant_Indexing => Current_Entry, Default_Iterator => Iterate, Iterator_Element => Directory_Entry_Type; function Alphanumeric (Left, Right : Directory_Entry_Type) return Boolean; function Entries -- Returns an iterable container (Directory : String; Pattern : String; Filter : access function (Item : Directory_Entry_Type) return Boolean := null; Sorting : not null access function (Left, Right : Directory_Entry_Type) return Boolean := Alphanumeric'Access) return Directory_Listing; function Length (Container : Directory_Listing) return Ada.Containers.Count_Type; type Cursor is private with Preelaborable_Initialization; function Has_Element (Position : Cursor) return Boolean; package Directory_Iterators is new Ada.Iterator_Interfaces (Cursor, Has_Element); function Current_Entry -- "indexing" operation (Entries : Directory_Listing; Position : Cursor) return Directory_Entry_Type with Pre => Has_Element (Position); function Iterate (Listing : Directory_Listing) return Directory_Iterators.Forward_Iterator'Class; Example of Usage: for Dir_Entry of Ada.Directories.Entries ("/home/brad", "*.txt") loop Put_Line (Ada.Directories.Simple_Name (Dir_Entry))); end loop; I considered whether the Filter and Sort should be generic formals instead of anonymous subprograms, but I think this would be inconvenient for users to have to instantiate generics. Also, one might want to process various filters and sort criteria and it is nicer to be able to do that on the fly in my opinion. Do these changes look worthwhile to writeup? **************************************************************** From: Brad Moore Sent: Saturday, May 28, 2016 10:46 PM I now have a working implementation of Ada.Environment_Variables as well. I have a few notes and comments about this design as well that differs from the previous writeup. 1) Similar to how the pattern parameter of Ada.Directories is useful to limit the set of entries involved in the iteration, I added a similar pattern parameter to the new iterator for the Ada Environment Variables. In my experience, when I am looking for environment variables, I typically am only interested in those that match a certain pattern. Eg, "GTK*" could be used to find all the environment variables associated with GTK. The default for the pattern is "*" which retrieves all variables. 2) I find that when using the new Ada 2012 iterator syntax, I want to use the object prefix view notation. eg. for Variable of Ada.Environment_Variables.All_Variables (Pattern => "PARALLEL*") loop Put_Line (Variable.Name & "= " & Variable.Value); end loop; rather than for Variable of Ada.Environment_Variables.All_Variables (Pattern => "PARALLEL*") loop Put_Line (Ada.Environment_Variables.Name (Variable) & "= " & Ada.Environment_Variables.Value (Variable)); end loop; To achieve this, the Name_Value_Pair type needs to be a tagged type. One issue with having an iterator return an object of a tagged type, is you run into having to create an indexing function that is dispatching on more than one type, which doesn't compile. To get around this, I had to create a nested package for the Name_Value_Pair type, so that the function tied to the Constant_Indexing aspect of the Environment Variable listing can be dispatching only on the listing type. I don't mind that too much, because the name value pair is separated as a separate abstraction. It might even be worth considering moving that to its own package. Maybe someone has a better suggestion. Another benefit of the approach of retrieving the full list and storing in the listing object is that it removes the need for some of the Rosen Trick difficulty I ran into, in the previous implementations. It doesn't completely remove this, as the Rosen Trick is still needed for the Default_Iterator aspect, because that function has an "in" parameter for the container, but typically the iterator needs to store a reference to the container. This issue is actually common to all the standard containers, which is why I believe GNAT appears to be using 'Unrestricted_Access in all the standard containers for the Default_Iterator aspects. Using such an attribute would eliminate the need for using the Rosen Trick and also gets around the issue of having to make the container a limited type, but unfortunately, this is a non-portable technique. Presumably other compiler vendors would need to use a similar implementation defined attribute, or some other technique to get around this issue. Ideally, the parameter would have been "in out" which would have avoided these issues. Not sure what can be done about that now though... The Design I have for the Ada.Environment_Variables package adds the following to the public part of the spec... Unless I hear back any comments in the next couple of days, I will proceed to write up the new version of the AI to include this. package Name_Value_Pairs is type Name_Value_Pair is tagged private; ------------------------------------ -- Operations on Name_Value_Pairs -- ------------------------------------ function Create (Name, Value : String) return Name_Value_Pair; function Name (Name_Value : Name_Value_Pair) return String; function Value (Name_Value : Name_Value_Pair) return String; private ... end Name_Value_Pairs; use Name_Value_Pairs; type Environment_Variable_Listing is tagged limited private with Preelaborable_Initialization, Constant_Indexing => Current_Variable, Default_Iterator => Iterate, Iterator_Element => Name_Value_Pair; function All_Variables (Pattern : String := "*") return Environment_Variable_Listing; function Length (Container : Environment_Variable_Listing) return Ada.Containers.Count_Type; type Cursor is private with Preelaborable_Initialization; function Has_Element (Position : Cursor) return Boolean; package Environment_Iterators is new Ada.Iterator_Interfaces (Cursor => Cursor, Has_Element => Has_Element); function Current_Variable -- "indexing" operation (Variables : Environment_Variable_Listing; Position : Cursor) return Name_Value_Pair with Pre => Has_Element (Position); function Iterate (Listing : Environment_Variable_Listing) return Environment_Iterators.Forward_Iterator'Class; **************************************************************** From: Randy Brukardt Sent: Monday, May 30, 2016 5:55 PM ... > I then went further, and thought about someone wanting to iterate > through the directory in a particular order, or to have better control > over which files are iterated over. > > This led me to a new prototype, where one can optionally specify a > sort order for the results, and a filter that gives much more control > than the existing one, which only lets you specify file type. I think this is a bad idea. (See below.) > I have implemented this prototype in GNAT, and it appears to work. > Also, it seems worth noting that my existing test program did not > require any changes to switch over to this new API. > > Internally, the implementation changed from just wrapping calls to > Search, to fetching a vector of matching directory entries, and > storing that list in the Directory_Listing type. And this is why. This means that the memory requirements for a directory search are unpredictable, and potentially much, much more than the basic iterators as currently defined. In particular, on Windows, the search API uses a handle and some calls to implement the existing searching capabilities. There is no buffering needed other than for the value being currently returned. (I believe that Linux would be similar, but it's been long enough since I've done that I don't want to be definitive.) Contrast that to the (dynamic) memory requirements needed to store the entire directory. And remember that a directory can have nearly an unlimited set of files; I have directories with nearly 10,000 files (there is one with 9522 files as I write this). We surely don't want a disincentive to use the "improved" forms. **************************************************************** From: Brad Moore Sent: Tuesday, May 31, 2016 10:32 PM > I think this is a bad idea. (See below.) > >> I have implemented this prototype in GNAT, and it appears to work. >> Also, it seems worth noting that my existing test program did not >> require any changes to switch over to this new API. >> >> Internally, the implementation changed from just wrapping calls to >> Search, to fetching a vector of matching directory entries, and >> storing that list in the Directory_Listing type. > > And this is why. This means that the memory requirements for a > directory search are unpredictable, and potentially much, much more > than the basic iterators as currently defined. On linux, there are two functions. struct dirent *readdir(DIR *dirp); Which would be similar to the Windows API you describe, and the existing Ada API. There is also; int scandir(const char *dirp, struct dirent ***namelist, int (*filter)(const struct dirent *), int (*compar)(const struct dirent **, const struct dirent **)); Which is where I found the inspiration for the suggestion, and is very similar to what I proposed. I could see it being useful to have both these forms available. One for the case where the directories are large, and you generally want to process all the files, and the other when storage is not an issue, but a need for processing files in a certain order, or filtering on certain files, such as those within a certain date range, or file size. Only the filenames that satisfy the pattern and filter would actually get stored in the container. Maybe it is worth considering having both forms? For environment variables though, I think it is more reasonable to have the list stored in the "container", rather than the iterator, mostly because environment variables are not typically or even likely to be encountered in the numbers that you might find in a directory. I doubt it makes sense to have two forms for this package though. I'm guessing that you'd still dislike the storage requirements for this approach though, as having to copy and then finalize that copy of the environment may be undesirable, if the OS can in pass the environment, as it does for C programs, presumably more efficiently as an array of string pointers. So, nix the idea for environment variables, but what about having two forms for the directory iteration? **************************************************************** From: Randy Brukardt Sent: Tuesday, May 31, 2016 11:13 PM ... > So, nix the idea for environment variables, but what about having two > forms for the directory iteration? That seems like overkill to me. It's easy enough to write it yourself if you need it (given the existence of the indefinite vectors). The cost for many of my programs would be prohibitive (remember, one directory has 9522 files). And you've got a lot of RM text there just for one version. I'd probably be a bit less negative if we were just talking about a single procedure, as in: procedure Search ( Directory : in String; Pattern : in String; Sort_Order: in Sort_Order_Type; -- Not defaulted for compatibility. Filter : in Filter_Type := (others => True); Process : not null access procedure ( Directory_Entry : in Directory_Entry_Type)); This would work well with Tucker's (or is it Bob's) lambda loops. Note that we'll probably consider the lambda loops first, because if we decide to go that way, we'll probably leave Environment_Variables and Directories unchanged. (Making all of your work a waste, other than to prove it didn't work very well -- I guess an important data point that we needed anyway.) My recollection of the reason that Bob/Tucker came up with that idea was an alternative way to get the functionality that you were struggling with. **************************************************************** From: Brad Moore Sent: Sunday, June 5, 2016 5:42 PM Here is another part of my homework. [This is version /04 of the AI - Ed.] This is an update to an existing AI about adding Ada 2012 iterator support to Ada.Directories and Ada.Environment_Environment_Variables. I basically addressed Jeff and Randy's comments that arrived subsequent to our last meeting in Vermont. There were a few other minor changes as well, such as renaming the Entry_Presence type to a Cursor type, which I found reads a lot better now. **************************************************************** From: Randy Brukardt Sent: Monday, June 6, 2016 7:10 PM > AI23-009-1 I think I just had a Rip-van-Winkle moment. ;-) A couple of comments (I'll fix these in the posted draft, so you (Brad) don't have to): > type Directory_Listing is tagged limited private > with Preelaborable_Initialization, > Constant_Indexing => Current_Entry, > Default_Iterator => Iterate, > Iterator_Element => Directory_Entry_Type; Preelaborable_Initialization is not an aspect; it is view-specific which doesn't map well to aspects. We tried and punted on it for Ada 2012. Perhaps we should try again to make it an aspect (separate AI, of course), but it certainly isn't one now. So you have to use the pragma. As in: type Directory_Listing is tagged limited private with Constant_Indexing => Current_Entry, Default_Iterator => Iterate, Iterator_Element => Directory_Entry_Type; pragma Preelaborable_Initialization (Directory_Listing); There's some other places like that. In 112/3: >The type Directory_Listing need finalization. This should be "needs finalization". **************************************************************** From: Brad Moore Sent: Monday, June 6, 2016 9:53 PM > A couple of comments (I'll fix these in the posted draft, so you > (Brad) don't have to): Thanks, I also realized I forgot to mention why I made Name_Value_Pair_Type a tagged type, and thought it would be worth mentioning. Would it be possible to have you attach the following to replace the last sentence of the Discussion section, or should I resubmit? "A similar approach is applied to Ada.Environment_Variables, except that the type Name_Value_Pair_Type was made to be a tagged type to improve usability by allowing object prefix notation. for Pair of Ada.Environment_Variables.Environment loops Put_Line (Pair.Name & "=" & Pair.Value); end loop; rather than; for Pair of Ada.Environment_Variables.Environment loops Put_Line (Ada.Environment_Variables.Name (Pair) & "=" & Ada.Environment_Variables.Value (Pair)); end loop; Making Name_Value_Pair_Type tagged however, meant having to declare the type in a nested package, to avoid problems with having the call Current_Variable dispatching on more than one type, which is illegal. It would have been nice to do the same for Directory_Entry_Type, and make it a tagged type but that type already existed and changing that to be a tagged type would likely break backwards incompatibility in some way." >> type Directory_Listing is tagged limited private >> with Preelaborable_Initialization, >> Constant_Indexing => Current_Entry, >> Default_Iterator => Iterate, >> Iterator_Element => Directory_Entry_Type; > > Preelaborable_Initialization is not an aspect; it is view-specific > which doesn't map well to aspects. We tried and punted on it for Ada > 2012. Perhaps we should try again to make it an aspect (separate AI, > of course), but it certainly isn't one now. So you have to use the pragma. As in: > > type Directory_Listing is tagged limited private > with Constant_Indexing => Current_Entry, > Default_Iterator => Iterate, > Iterator_Element => Directory_Entry_Type; > pragma Preelaborable_Initialization (Directory_Listing); > > There's some other places like that. I forgot about that, but I note that the code did compile, so apparently GNAT supports this already. I think it seems warranted in cases like this to have it as an aspect, if for readability sake if nothing else. **************************************************************** From: Randy Brukardt Sent: Monday, June 6, 2016 11:16 PM > Thanks, I also realized I forgot to mention why I made > Name_Value_Pair_Type a tagged type, and thought it would be worth > mentioning. Would it be possible to have you attach the following to > replace the last sentence of the Discussion section, or should I > resubmit? I can put this wording in, but where does it go? It doesn't seem to fit with any of the existing discussion. ... > > type Directory_Listing is tagged limited private > > with Constant_Indexing => Current_Entry, > > Default_Iterator => Iterate, > > Iterator_Element => Directory_Entry_Type; > > pragma Preelaborable_Initialization (Directory_Listing); > > > > There's some other places like that. > > I forgot about that, but I note that the code did compile, so > apparently GNAT supports this already. Hopefully not in "pedantic" mode. :-) > I think it seems > warranted in cases like this to have it as an aspect, if for > readability sake if nothing else. Sure, that seems great. But it's not a (sub)type-related aspect semantically, as those have the same value for all views. (PI can be false for a partial view and true for the full view, and both have to work as expected for compatibility.) Abandoning that means that visibility begins to matter about (sub)type-related aspects, which means lots of new rules (and likely new bugs). One can imagine someone adding exceptions all over the Standard for PI (as the only view-specific type-related aspect), but I'm not volunteering for that. Or I suppose one could say something like "Notwithstanding what this Standard says elsewhere, the properties of aspect Preelaborable_Initialization are exactly as specified here; no other rules for type-related aspects apply." And then write everything that you need. I ain't signing up to do that, either. (Steve volunteered to try for Ada 2012, but the full group told him not to bother, given that it was too close to the final date. Maybe you can talk him into trying again??) Anyway, find someone to propose an AI (with wording!!!), and I'll be happy to tear it to shreds^H^H^H^H^Hput it on the agenda. In the meantime, use the pragma in any AIs proposed. **************************************************************** From: Brad Moore Sent: Tuesday, June 7, 2016 7:52 AM > I can put this wording in, but where does it go? It doesn't seem to > fit with any of the existing discussion. > I was thinking at the very end of the discussion, to replace the last sentence. But perhaps it should start off differently. Something like; "The overall approach taken for Ada.Environment_Variables is similar to the one taken for Ada.Directories except that..." ****************************************************************