Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Story Points:4
-
Epic Link:
-
Sprint:DRP S21a (Dec Jan), DRP S21b
-
Team:Data Release Production
-
Urgent?:No
Description
To support Gen3 image differencing pipelines.
DM-20695 implemented the ccd-level selectors run at makeWarp. Runs on a list of ccds for one single visit.
This will implement the visit-level selectors done during coaddition. Operates on a list of deepCoadd_directWarps and other visit-level data products like sourceTable_visits
Attachments
Activity
I left comments on GitHub. Did you mean to put this ticket all the way in review, or are you still planning to do more things?
Thanks for the review. I think I addressed all your comments, including your not liking the parquet inputs. I swapped them out for the new ExposureCatalog `visitSummary`. (This was needed anyway when I add MJD bounds next ticket). With DM-29407 on master from last week, it’s super fast. My test run with 33 visits now takes <4s.
The idea was that BestSeeingSelectVisitTask, would be close to your original BestSeeingSelectImagesTask so that it’d look familiar and you could maintain along with the old Gen2 tasks. This was intended to be the task you maintained for AP, so I’m happy to make any edits you’d like to it. If you’re free to pair-code Tues/Wed, I’ll change all the variable names and default configs to whatever you want.
BestSeeeingQuantileSelectVisitsTask is the new one I’m planning going to use for DRP.
And now that visitSummary has the info, I also added a double check that the visits do indeed overlap the patch. This probably could be refined in the future to use getValidPolygon instead of the corners.
And whoops, need to "Ask for Review"
I really like the new `visitSummary`, thanks for using it here, and thanks also for your thoughtful replies to my initial comments.
The only thing these new tasks could use is some unit tests. If you'd like to get this functionality in the weekly for the release, I'm OK with a Jenkins run + merge today, and please file a ticket to add tests. If you don't think it's feasible to push it through today, please add tests on this ticket and I can take a final look when it's ready.
Overall I'm very happy with this, and I hope Jenkins is too!
I nominate alternating between myself and Eric Bellm; I could do this next one eventually if his backlog is overflowing more than mine.
Eric Bellm and Meredith Rawls Gen3 implementation of the your BestSeeingSelectVisitsTasK is ready for review. You're probably busy for the next 36 hours so it can wait until Thurs/Fri. If the class ends in "SelectImagesTask" it selects calexps for warping. If it ends in "SelectVisitsTask" (such as these new ones), it selects warps for coaddition. BestSeeingSelectVisitsTasks is the implementation of the existing gen2 BestSeeingSelectImages, just using catalogs as input. The pixelscale is slightly different than that computed from the per-ccd wcs. The PSF FWHM is spot on, however. I also included my own BestSeeingPercentileSelectVisitsTasks which we manually used this sprint and I plan on using as the default for DRP until we validate something better.
Took Nate Lust's suggestion to write out the selected visit lists as output and have assemble read them in as input. He pointed out the StructuredDataList isn't working yet and suggested I use the storage class StructuredDataDict. (If you're wondering why I'm storing them as a dict instead of a list).
Nate Lust, would you take a look at my pipeline file since I know you have a vision on how these should look:
{outputCoaddName}https://github.com/lsst/pipe_tasks/blob/865f4040131154d0034ded6e21f381f1d5cad218/pipelines/_DiffimDRP.yaml
I didn't know how to specify the template
= bestSeeing so I just specified the name of the three output connections. Please advise.