Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34174

Make fully gen3 replacement for PropagateVisitFlagsTask with dataframe inputs

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None

      Description

      The current PropagateVisitFlagsTask at https://github.com/lsst/pipe_tasks/blob/c7c10ab8074954fee4cfab5bfd8db91f106dff96/python/lsst/pipe/tasks/propagateVisitFlags.py has several issues that will all be addressed on this ticket:
      1) The inputs are afw src tables rather than (faster!) parquet tables.
      2) The inputs do not have access to the finalized psf characterization flags from FinalizeCharacterizationTask.
      3) As noted in the comments, the matching from coadd objects to flagged sources can be done more efficiently by reusing a kd-tree.
      4) The code is an unholy mishmash of gen2 and gen3.

      This ticket will create a new task (provisionally called NewPropagateVisitFlagsTask; yes, that's a terrible name) that will take per-visit dataframe inputs. It will use smatch as a convenient wrapper around scipy's fast kd-tree implementation for matching (while reusing the tree created from the coadd objects). And it will be gen3 only.

      At the current time, the dataframe input will be the per-visit dataframes from FinalizeCharacterizationTask, which will be modified on this ticket to (a) include the initial photometry and astrometry flags that are also propagated here, and (b) include the isolated star id (as per request from Yusra AlSayyad).

      In the future, after we have final-final source tables that have been rerun with the refined fgcmcal/jointcal photometry/astrometry, the default input dataframe can be changed to these new tables, and updated photometry and astrometry flags from fgcmcal and jointcal can be similarly propagated to the coadd catalogs (which is not done at the current time, and will also not be included in this ticket).

        Attachments

          Activity

          Hide
          erykoff Eli Rykoff added a comment -

          Thanks Fred Moolekamp! Note that Yusra AlSayyad has already taken a look last week and after a discussion I removed the final_psf_used names in favor of keeping the calib_psf_used names and just replacing the flags.

          Show
          erykoff Eli Rykoff added a comment - Thanks Fred Moolekamp ! Note that Yusra AlSayyad has already taken a look last week and after a discussion I removed the final_psf_used names in favor of keeping the calib_psf_used names and just replacing the flags.
          Hide
          fred3m Fred Moolekamp added a comment -

          Looks good for the most part, with a few comments in the pipe_tasks PR. Mainly I'm wondering about why it needs to be written for both a source table and finalized source table as opposed to running on a single table and allowing a user to run it twice, once on each table, if that's ever going to be the case. Feel free to take up this conversation here or in the PR.

          Show
          fred3m Fred Moolekamp added a comment - Looks good for the most part, with a few comments in the pipe_tasks PR. Mainly I'm wondering about why it needs to be written for both a source table and finalized source table as opposed to running on a single table and allowing a user to run it twice, once on each table, if that's ever going to be the case. Feel free to take up this conversation here or in the PR.
          Hide
          fred3m Fred Moolekamp added a comment -

          Thanks for the explanations in slack and for making the requested changes. I would also have someone not familiar with this task read through the docs/comments and make sure they understand the two different catalogs used for propagating flags. I think that they're clear, but I also know how it works now so I might not be the best judge.

          Show
          fred3m Fred Moolekamp added a comment - Thanks for the explanations in slack and for making the requested changes. I would also have someone not familiar with this task read through the docs/comments and make sure they understand the two different catalogs used for propagating flags. I think that they're clear, but I also know how it works now so I might not be the best judge.

            People

            Assignee:
            erykoff Eli Rykoff
            Reporter:
            erykoff Eli Rykoff
            Reviewers:
            Fred Moolekamp
            Watchers:
            Eli Rykoff, Fred Moolekamp, Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.