Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: pipe_tasks
-
Labels:None
-
Story Points:4
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
The current PropagateVisitFlagsTask at https://github.com/lsst/pipe_tasks/blob/c7c10ab8074954fee4cfab5bfd8db91f106dff96/python/lsst/pipe/tasks/propagateVisitFlags.py has several issues that will all be addressed on this ticket:
1) The inputs are afw src tables rather than (faster!) parquet tables.
2) The inputs do not have access to the finalized psf characterization flags from FinalizeCharacterizationTask.
3) As noted in the comments, the matching from coadd objects to flagged sources can be done more efficiently by reusing a kd-tree.
4) The code is an unholy mishmash of gen2 and gen3.
This ticket will create a new task (provisionally called NewPropagateVisitFlagsTask; yes, that's a terrible name) that will take per-visit dataframe inputs. It will use smatch as a convenient wrapper around scipy's fast kd-tree implementation for matching (while reusing the tree created from the coadd objects). And it will be gen3 only.
At the current time, the dataframe input will be the per-visit dataframes from FinalizeCharacterizationTask, which will be modified on this ticket to (a) include the initial photometry and astrometry flags that are also propagated here, and (b) include the isolated star id (as per request from Yusra AlSayyad).
In the future, after we have final-final source tables that have been rerun with the refined fgcmcal/jointcal photometry/astrometry, the default input dataframe can be changed to these new tables, and updated photometry and astrometry flags from fgcmcal and jointcal can be similarly propagated to the coadd catalogs (which is not done at the current time, and will also not be included in this ticket).
Attachments
Activity
Looks good for the most part, with a few comments in the pipe_tasks PR. Mainly I'm wondering about why it needs to be written for both a source table and finalized source table as opposed to running on a single table and allowing a user to run it twice, once on each table, if that's ever going to be the case. Feel free to take up this conversation here or in the PR.
Fred Moolekamp
added a comment - Looks good for the most part, with a few comments in the pipe_tasks PR. Mainly I'm wondering about why it needs to be written for both a source table and finalized source table as opposed to running on a single table and allowing a user to run it twice, once on each table, if that's ever going to be the case. Feel free to take up this conversation here or in the PR. Thanks for the explanations in slack and for making the requested changes. I would also have someone not familiar with this task read through the docs/comments and make sure they understand the two different catalogs used for propagating flags. I think that they're clear, but I also know how it works now so I might not be the best judge.
Fred Moolekamp
added a comment - Thanks for the explanations in slack and for making the requested changes. I would also have someone not familiar with this task read through the docs/comments and make sure they understand the two different catalogs used for propagating flags. I think that they're clear, but I also know how it works now so I might not be the best judge.
Thanks Fred Moolekamp! Note that Yusra AlSayyad has already taken a look last week and after a discussion I removed the final_psf_used names in favor of keeping the calib_psf_used names and just replacing the flags.