Yusra AlSayyad, do you know if this was taken care of in your last round of changes to the PipelineTask version of AssembleCoadd et al?

Discussed this in slack, and then I took a closer look at the code.  We do still have work to do, because the selections we do in Gen2 can't just be written as data query expressions in Gen3.  In particular:

• AP uses BestSeeingWcsSelectImagesTask, which loads PSF models from calexps and computes sizes from them on the fly.  We might eventually start recording that as per-DatasetType metadata in the Registry, and then we might be able to use data query expressions, but not yet (this is in part DM-21774).
• DRP usually uses PsfWcsSelectImagesTask, which computes a rather complex PSF residual metric from src catalogs.  It's less obvious that this is something we should put in general-purpose calexp metadata, though that is a possibility.  Another option might be to convert this calculation into a true lsst.verify metric Measurement, and (perhaps on DM-21875) use a special Datatore to put those into the same database that holds the registry, and then add Registry support for including those metric external tables in queries.  That's worth doing for its own sake eventually, but it's a lot of work to get coaddition working as it does in Gen2.

More importantly, trying to use data query expressions to solve either of these has a much bigger problem: the information needed to make the selection fundamentally is not available until after calexps are created, but we may want to generate a QuantumGraph that starts from raws, and in that case we'd need to evaluate the data query expression before the fields it uses will exist (unless we base the selection on a different run, which seems both complex and dangerous).

The alternative is to continue to make selection for coaddition something that happens in Python.  The SelectImageTasks are (in Gen2) subtasks that nevertheless use the Butler, but this is problematic in Gen3, especially because they don't necessarily all use the same inputs (BestSeeingWcsSelectImagesTask uses only calexp, while PsfWcsSelectImagesTask also uses src).  We don't currently have any way for a PipelineTask to pull connections from a subtask, and I'm not sure we should invent one.

Instead, I think it'd be best to make the Gen3 equivalents of these tasks PipelineTasks in their own right that would take whatever inputs they wished and output an ExposureCatalog of selected images to be (perhaps optionally) consumed by AssembleCoadd and perhaps MakeWarpTask.

Finally, it's worth pointing out that the "Wcs" part of these selection tasks - which checks for overlap between the detector-level image and patch - should be unnecessary in Gen3 - the middleware will already perform a logically equivalent operation.  The calculation may be subtly different, thought, because the Gen2 version does the overlap in pixel coordinates while the Gen3 version operates in spherical coordinates.  Both are checked/refined later when warping, but we may want to retain the Gen2-style check in Gen3 during the middleware transition to make it easier to compare results, and then drop it after the Gen3 tasks are fully validated and the Gen2 ones are retired.

OK, ready for review.
Passed jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/33619/pipeline
Hits pipe_tasks and adds this to the HSC pipeline. Needed for the w_2020_10 monthly run. w_2020_10 is tagged in a week.

Meredith Rawls You came to mind first, but WARNING this isn't the BestSeeingWcsSelector that you've been waiting for.
Because a makeWarp quantum is per-visit, it can only select at the ccd-level. It operates on a list of ccds for one single visit.

I converted the BestSeeingWcsSelectImageTask that was in there, but it does not make any sense at all. Sorting the 20 ccds in a single visit that might overlap a patch doesn't give you the best seeing visits. That comes next.

I opened a new ticket for the visit-level selector that'll be run before coaddition: DM-28953.

I'm going to ask you to review that future ticket, but this one is yours to review too if you want it. Let's talk tomorrow on the diffim standup.

Meredith Rawls you're off the hook, Eric offered to review this during the diffim meeting. Thanks Eric!

Looks good to me, Yusra AlSayyad--just one tiny typo I caught.

And note to future self on testing:

Using bps, ran 5 bands 9615 and 9697:
/project/yusra/party/replicateRC2.yaml
logs in /project/yusra/party/submit
and confirmed that the directWarp counts for each of the tract/filter combos all matched the gen2 rerun converted to gen3. e.g.

dataId = {'skymap': 'hsc_rings_v1', 'tract': 9697, 'band': 'y'}

print(len(set(butler.registry.queryDatasets("deepCoadd_directWarp", collections=['u/yusra/DM-20695_NoEmpty',],

                                                     dataId=dataId

                                                    ))))

print(len(set(butler.registry.queryDatasets("deepCoadd_directWarp", collections=['HSC/runs/RC2/w_2021_02',],

                                                     dataId=dataId

                                                    ))))

813

813

Merged to master last week.