Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11866

Propagate astrometry and photometry visit calibration flags to coadds

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      6
    • Epic Link:
    • Sprint:
      DRP F18-4
    • Team:
      Data Release Production

      Description

      DM-4878 added a mechanism for propagating flags from the input catalogues to the coadd catalogue indicating stars that were used for measuring the PSF. Since then, DM-9050 added similar calibration flags for astrometry and photometry which should also be added to the list of flags that get propagated from visits to coadds.

      The default list of propagated flags is currently:

      flags = DictField(keytype=str, itemtype=float,
                            default={"calib_psf_candidate": 0.2, "calib_psf_used": 0.2, },
                            doc="Source catalog flags to propagate, with the threshold of relative occurrence.")
      

      Please add the following to this list:

      • calib_astrometry_used
      • calib_photometry_used
      • calib_photometry_reserved

        Attachments

          Issue Links

            Activity

            No builds found.
            lauren Lauren MacArthur created issue -
            lauren Lauren MacArthur made changes -
            Field Original Value New Value
            Link This issue relates to DM-4878 [ DM-4878 ]
            lauren Lauren MacArthur made changes -
            Link This issue relates to DM-9050 [ DM-9050 ]
            lauren Lauren MacArthur made changes -
            Link This issue blocks DM-11974 [ DM-11974 ]
            lauren Lauren MacArthur made changes -
            Link This issue is blocked by DM-14997 [ DM-14997 ]
            lauren Lauren MacArthur made changes -
            Watchers John Swinbank, Lauren MacArthur [ John Swinbank, Lauren MacArthur ] John Swinbank, Lauren MacArthur, Yusra AlSayyad [ John Swinbank, Lauren MacArthur, Yusra AlSayyad ]
            lauren Lauren MacArthur made changes -
            Risk Score 0
            lauren Lauren MacArthur made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            lauren Lauren MacArthur made changes -
            Description DM-4878 added a mechanism for propagating flags from the input catalogues to the coadd catalogue indicating stars that were used for measuring the PSF. Since then, DM-9050 added similar calibration flags for astrometry and photometry which should also be added to the list of flags that get propagated from visits to coadds.

            The default list of propagated flags is currently:
            {code}
            flags = DictField(keytype=str, itemtype=float,
                                  default={"calib_psfCandidate": 0.2, "calib_psfUsed": 0.2, },
                                  doc="Source catalog flags to propagate, with the threshold of relative occurrence.")
            {code}

            Please add the following to this list:
            - calib_astrometryUsed
            - calib_photometryUsed
            - calib_photometryReserved
            DM-4878 added a mechanism for propagating flags from the input catalogues to the coadd catalogue indicating stars that were used for measuring the PSF. Since then, DM-9050 added similar calibration flags for astrometry and photometry which should also be added to the list of flags that get propagated from visits to coadds.

            The default list of propagated flags is currently:
            {code}flags = DictField(keytype=str, itemtype=float,
                                  default={"calib_psf_candidate": 0.2, "calib_psf_used": 0.2, },
                                  doc="Source catalog flags to propagate, with the threshold of relative occurrence.")
            {code}
            Please add the following to this list:
             - calib_astrometry_used
             - calib_photometry_used
             - calib_photometry_reserved
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Would you mind giving this a look?  Seeing as you have been poking around the plots from pipe_analysis, you understand the motivation.

            Unit testing is not really feasible as we need a fully processed (through multiband) dataset, so I added a test in ci_hsc.

            The PR for pipe_tasks (which is not showing up in Jira these days) is: https://github.com/lsst/pipe_tasks/pull/217

            I ran a Jenkins run before the weekend which passed. Just to be on the safe side, I also kicked one off today [UPDATE: it also passed].

            Oh, and I also created a coadd and checked that the sources that get these flags set do indeed look sensible.

            Show
            lauren Lauren MacArthur added a comment - - edited Would you mind giving this a look?  Seeing as you have been poking around the plots from pipe_analysis, you understand the motivation. Unit testing is not really feasible as we need a fully processed (through multiband) dataset, so I added a test in ci_hsc . The PR for pipe_tasks (which is not showing up in Jira these days) is: https://github.com/lsst/pipe_tasks/pull/217 I ran a Jenkins run before the weekend which passed . Just to be on the safe side, I also kicked one off today [UPDATE: it also passed] . Oh, and I also created a coadd and checked that the sources that get these flags set do indeed look sensible.
            lauren Lauren MacArthur made changes -
            Reviewers Meredith Rawls [ mrawls ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            lauren Lauren MacArthur made changes -
            Story Points 2
            Hide
            mrawls Meredith Rawls added a comment -

            Looks good to me. The only thing I wasn't clear about is why the default values are all 0.2. If you could possibly clarify that in the docstring, that would be helpful.

            Show
            mrawls Meredith Rawls added a comment - Looks good to me. The only thing I wasn't clear about is why the default values are all 0.2. If you could possibly clarify that in the docstring, that would be helpful.
            mrawls Meredith Rawls made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            lauren Lauren MacArthur added a comment -

            Very fair question.  There is a lengthy discussion in the task description here: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/propagateVisitFlags.py#L64-L90

            This seemed like more than enough justification to me, and I felt the logic also applies to the flags added here.  Let me know if you disagree or think a note needs to be added elsewhere to make it more visible.

            Show
            lauren Lauren MacArthur added a comment - Very fair question.  There is a lengthy discussion in the task description here: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/propagateVisitFlags.py#L64-L90 This seemed like more than enough justification to me, and I felt the logic also applies to the flags added here.  Let me know if you disagree or think a note needs to be added elsewhere to make it more visible.
            Hide
            mrawls Meredith Rawls added a comment -

            Indeed, that provides plenty of justification, but is a rather odd place for it. I think I was further confused by the default for the following config field, matchRadius, which also happens to be 0.2... arcseconds.

            I'd prefer a second sentence added to the docstring along the lines of, "May be in the range 0-1; default 0.2 will flag the coadd if at least 20% of its inputs are flagged." I didn't fully understand that's what was happening here until you pointed me to the task description.

            Show
            mrawls Meredith Rawls added a comment - Indeed, that provides plenty of justification, but is a rather odd place for it. I think I was further confused by the default for the following config field, matchRadius, which also happens to be 0.2... arcseconds. I'd prefer a second sentence added to the docstring along the lines of, "May be in the range 0-1; default 0.2 will flag the coadd if at least 20% of its inputs are flagged." I didn't fully understand that's what was happening here until you pointed me to the task description.
            yusra Yusra AlSayyad made changes -
            Sprint DRP F18-4 [ 774 ]
            Hide
            lauren Lauren MacArthur added a comment -

            The config parameter doc now reads (with a --show config=propagateFlags* on a multiband.py run):

            # Source catalog flags to propagate, with the threshold of relative occurrence (valid range: [0-1], default is 0.2). Coadd object will have flag set if the fraction of input visits in which it is flagged is greater than the threshold.
            config.measureCoaddSources.propagateFlags.flags={'calib_psf_candidate': 0.2, 'calib_psf_used': 0.2, 'calib_psf_reserved': 0.2, 'calib_astrometry_used': 0.2, 'calib_photometry_used': 0.2, 'calib_photometry_reserved': 0.2}

            Show
            lauren Lauren MacArthur added a comment - The config parameter doc now reads (with a --show config=propagateFlags* on a multiband.py run): # Source catalog flags to propagate, with the threshold of relative occurrence (valid range: [0-1], default is 0.2). Coadd object will have flag set if the fraction of input visits in which it is flagged is greater than the threshold. config.measureCoaddSources.propagateFlags.flags={'calib_psf_candidate': 0.2, 'calib_psf_used': 0.2, 'calib_psf_reserved': 0.2, 'calib_astrometry_used': 0.2, 'calib_photometry_used': 0.2, 'calib_photometry_reserved': 0.2}
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks Meredith.  Docstring updated and merged to master.

            Show
            lauren Lauren MacArthur added a comment - Thanks Meredith.  Docstring updated and merged to master.
            lauren Lauren MacArthur made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            yusra Yusra AlSayyad made changes -
            Story Points 2 6
            yusra Yusra AlSayyad made changes -
            Epic Link DM-14402 [ 79665 ]

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Meredith Rawls
              Watchers:
              John Swinbank, Lauren MacArthur, Meredith Rawls, Yusra AlSayyad
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.