Jim, no objections to trying to generalize PreFlightUnitsRow if it can be made useful elsewhere. I'm not sure I have that deep knowledge of all concepts, I'm mostly discovering things by looking at them from pre-flight side.
Can we refactor any more of the preflight logic into subset producers/consumers/filters? The post-query spatial filtering to remove disjoint spatial DataUnits seems like at least one candidate for a filter.
I'm sure things can be made more modular and reusable. Spatial filtering can probably be moved to a separate filter but I suspect that this would be a 100% obligatory filter, I can't imagine anyone would want to deal with regions that don't overlap (ideally all selection should be based on regions only, I consider SkyMap as just an optimization that most people don't need to worry about).
Is a Subset just a regular iterator over PreFlightUnitsRow, or does it need to be a special iterator with some kind of header that describes what's common to all rows? Regular iterators would be really nice, because then we could use all kinds of itertools operations an generator-expression syntax on them.
It is regular iterator, all structure is contained in the record itself (which may not be super-efficient if we have large volumes of those records).
would it ever be useful/possible to support heterogeneous Subsets, in which the set of links in the DataId change from row to row? Could we implement that by just treating missing links as null/None?
This could be useful for some cases, I guess. For pre-flight I don't think it matters as we still need to know which missing things we need to make.
BTW, I don't see your comments on PR, did you click on Submit Review?