[Proposal] Unified Search & Item Listing for Sub-Trees

[jonhealy1]

I'd like to propose expanding the scope of the Children extension to support Item Interaction and Recursive Search on nodes.

The Problem: The "Leaf Node" Disconnect

Currently, the Children extension focuses on traversing the Containers (Catalogs/Collections).

• Workflow: User browses Root -> Sub-Catalog -> Collection.
• The Break: Once the user reaches a Collection (or a Catalog containing Items), the children paradigm stops. To see or search the data, the client must switch logic to the STAC API - Features endpoints (/items or /search).

The Missing Capability: "Search This Branch"

If a Catalog contains 100 sub-collections (e.g., a "Project" catalog), there is currently no standard way to "Search all items within this Project".

• Current Workaround: The client must crawl /children, collect 100 Collection IDs, and send a massive POST /search query with collections: [id1, id2, ... id100]. This is inefficient and chatty.

Proposal: Child-Scoped Item Endpoints

We should define standard endpoints relative to any node (Catalog or Collection) that allow interacting with the Items contained within that branch of the tree.

1. Recursive Item Listing
GET .../{node}/children/items

• Behavior: Returns an ItemCollection of items contained in this node (and optionally its descendants).
• Use Case: "Show me a preview of data in this folder."

2. Recursive/Scoped Search
POST .../{node}/children/search

• Behavior: Performs a standard STAC Search, but implicitly scoped to the collections/items within the current node's hierarchy.
• Use Case: "Find all images with cloud_cover < 10 inside the 'Forestry' catalog."

Why this is necessary

This transforms the extension from a simple "Link Lister" into a powerful Hierarchical Search Interface. It allows clients to treat any node in the tree as a "Virtual Root" for searching, without needing to know the complex graph structure underneath.

[m-mohr]

The issue that we have with rel=child links (inefficient requests to many files without /children), exists for rel=item links as well if they appear in the tree. This applies for rel=item links, unless you link to a rel=items endpoint, which usually exists in STAC API implementations. I think within radiantearth/stac-spec#1374 it would make sense to use rel=items as that's usually available anyway and directly gives you filter options. So linking to rel=item within the proposed tree in radiantearth/stac-spec#1374 doesn't make a lot of sense to me.

Nevertheless, if you would encounter rel=item links, it could make sense to include them in /children as it would solve the same issue. But:

If a Catalog contains 100 sub-collections (e.g., a "Project" catalog), there is currently no standard way to "Search all items within this Project".

That seems to be bad metadata architecture. If searching a project is a usecase, add a metadata field project (or similar) to the collections (and/or items) and a queryable for projects. That works for both collection search and item search. This can easily be done via the /items endpoint as such, I think collections should be the leaf and point to the items endpoint. Implementations could opt in to also list rel=item links in collections vith pagination and then /items would serve the same purpose as /children.

Performs a standard STAC Search, but implicitly scoped to the collections/items within the current node's hierarchy.

This can also be solved similarly with a single /search and queryables. The /{node}/children/search is just a shortcut or as you'd say in programming: syntactic sugar.

[jonhealy1]

This issue is I guess, say you have 10 collections, all with items, as children or grandchildren of a catalog or theme. You would have to go to each collection, get the collection id and then copy paste them all into a traditional /search request. I would like to do /catalogs/forestry/children/items?datetime=... in one swoop - it would be a lot easier.

[m-mohr]

Sure, but in the end the following two requests would lead to the same result:

1. /catalogs/forestry/children/items?datetime=... and
2. /search?datetime=...&filter-lang=cql2-text&filter=catalog='forestry' (assuming catalog is your queryable for this kind of search).

With a proper design, you don't need to crawl for all the collection IDs. You could also provide such links in theory already in the collections, prepared for the user to call, if the verbosity of the second link is an issue.

Whether users do that, is the other question. The navigational path leads them to a list of collections. I'm wondering whether they would rather pick a collection and search a specific collection.

[jonhealy1]

I think if I am User exploring themes or sub-catalogs, I would probably be using the /childrens endpoint. I would want to see items at some point and it would feel more natural to just throw /items on the end of the url I am already using. I wouldn't want to go make a /search request and hope that using filter= gives me the data I am looking for.

[jonhealy1]

Also I am not sure how viable it will always be to store catalog information in Items. If we have virtual catalogs based on topics or concepts, I don't think we would add these ids to the Items themselves. So therefore, doing something like catalog_id= or catalog= wouldn't necessarily work on the normal /search endpoint. We would use the themes extension but I think with large collections, updating themes fields when these fields can change frequently may become too cumbersome. Querying themes, concepts is an option however, but I think the user-friendly option is what is being described in this issue.

[m-mohr]

1. Is that really a usecase outside of us power users/developers that are deep into the spec? I hear that from the spec editors at OGC as well, but I've yet to see a user concatenate URLs manually, I usually see the using clients (pystac variants, STAC Browser, etc.). It also only covers one specific filter query, all others still need to go to the search endpoint and so search. Generally, it can also be supported by clients to provide that queryable for search more prominently.
2. Fair enough. I don't see them updated that frequently, but this could be an issue in certain cases. In theory, this could be populated in the Items automatically by the server depending on assigned collection IDs, but I also understand that that's not super efficient.

[jonhealy1]

I always use urls. I understand what you're saying, but I think api design principles align with the idea that urls should be as easy to use as possible. There are probably data scientists who feel more comfortable writing their own code versus trying to dive into pystac. They expect urls to make sense and be easy to understand. As an app developer and a stac leader, it makes sense that you don't want a new endpoint route when you can theoretically do the same thing with /search, but it's worth thinking about this. Maybe we can let this issue sit for a while and see if there's interest once more developers start to explore using /children for hierarchical stac api catalog and/or collection browsing.

[ajgrowney]

So we have used this child extension to actually solve this problem at Vantor (previously Maxar, previously DigitalGlobe which you might see in this diagram haha)
We migrated to this Root / Sub-Catalog based implementation a couple years ago and has been great for users because we have such a deep catalog that has also become pretty wide across domains. It also allows us to serve a growing and dynamic group of internal data providers via transaction and collection-transaction extensions
Happy to provide additional details, but what it boils down to at the end of the day is users can quickly scope down search results to logical groups of collections via the Sub-Catalog ID.
Many of our users are only interested in just imagery products, so instead of trying to crawl around the catalog to see if new imagery collections need to be added to their request, they can just go to GET /catalogs/imagery/search instead of GET /search?collections=[...]

[ajgrowney]

We also resolve the hierarchy of collections from each catalog id passed in the "collections" query parameter to allow users to search across disjoint sub-catalogs. Ex: GET /search?collections=dg-archive,sar -> search(collections=["wv01","lg01", ..., "um01", ...])

[jonhealy1]

@ajgrowney What you have implemented is similar to this stac api extension I have recently created - https://github.com/Healy-Hyperspatial/multi-tenant-catalogs. Please have a look and let me know what you think.

I have also started adding the logic as an optional feature to stac-fastapi-elasticsearch-opensearch (SFEOS).

[jonhealy1]

@ajgrowney Presently in the stac-api spec, there is no /catalogs endpoint and this has created a bit of a debate lately where some people think there should be and others think that it isn't needed.

[ajgrowney]

Yeah you've got some additional endpoints that are pretty nice which we could explore adopting too.
We have a pretty robust need for catalog/collection "configuration" management (which configures things like access control and other backend properties), so we just let users link via a parent key in the configuration body.
We used to check for the parent link when a user would PUT /collections/{collectionId} or POST it, but it wasn't a great user experience. The POST /catalogs/{id}/catalogs does seem to be a bit better for the linking.
We don't support multi-parenting and don't forsee that in our future, but everything else seems to be pretty much in-line. I'll do a more careful conformance audit next week