Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11866

Propagate astrometry and photometry visit calibration flags to coadds

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      6
    • Epic Link:
    • Sprint:
      DRP F18-4
    • Team:
      Data Release Production

      Description

      DM-4878 added a mechanism for propagating flags from the input catalogues to the coadd catalogue indicating stars that were used for measuring the PSF. Since then, DM-9050 added similar calibration flags for astrometry and photometry which should also be added to the list of flags that get propagated from visits to coadds.

      The default list of propagated flags is currently:

      flags = DictField(keytype=str, itemtype=float,
                            default={"calib_psf_candidate": 0.2, "calib_psf_used": 0.2, },
                            doc="Source catalog flags to propagate, with the threshold of relative occurrence.")
      

      Please add the following to this list:

      • calib_astrometry_used
      • calib_photometry_used
      • calib_photometry_reserved

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment - - edited

            Would you mind giving this a look?  Seeing as you have been poking around the plots from pipe_analysis, you understand the motivation.

            Unit testing is not really feasible as we need a fully processed (through multiband) dataset, so I added a test in ci_hsc.

            The PR for pipe_tasks (which is not showing up in Jira these days) is: https://github.com/lsst/pipe_tasks/pull/217

            I ran a Jenkins run before the weekend which passed. Just to be on the safe side, I also kicked one off today [UPDATE: it also passed].

            Oh, and I also created a coadd and checked that the sources that get these flags set do indeed look sensible.

            Show
            lauren Lauren MacArthur added a comment - - edited Would you mind giving this a look?  Seeing as you have been poking around the plots from pipe_analysis, you understand the motivation. Unit testing is not really feasible as we need a fully processed (through multiband) dataset, so I added a test in ci_hsc . The PR for pipe_tasks (which is not showing up in Jira these days) is: https://github.com/lsst/pipe_tasks/pull/217 I ran a Jenkins run before the weekend which passed . Just to be on the safe side, I also kicked one off today [UPDATE: it also passed] . Oh, and I also created a coadd and checked that the sources that get these flags set do indeed look sensible.
            Hide
            mrawls Meredith Rawls added a comment -

            Looks good to me. The only thing I wasn't clear about is why the default values are all 0.2. If you could possibly clarify that in the docstring, that would be helpful.

            Show
            mrawls Meredith Rawls added a comment - Looks good to me. The only thing I wasn't clear about is why the default values are all 0.2. If you could possibly clarify that in the docstring, that would be helpful.
            Hide
            lauren Lauren MacArthur added a comment -

            Very fair question.  There is a lengthy discussion in the task description here: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/propagateVisitFlags.py#L64-L90

            This seemed like more than enough justification to me, and I felt the logic also applies to the flags added here.  Let me know if you disagree or think a note needs to be added elsewhere to make it more visible.

            Show
            lauren Lauren MacArthur added a comment - Very fair question.  There is a lengthy discussion in the task description here: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/propagateVisitFlags.py#L64-L90 This seemed like more than enough justification to me, and I felt the logic also applies to the flags added here.  Let me know if you disagree or think a note needs to be added elsewhere to make it more visible.
            Hide
            mrawls Meredith Rawls added a comment -

            Indeed, that provides plenty of justification, but is a rather odd place for it. I think I was further confused by the default for the following config field, matchRadius, which also happens to be 0.2... arcseconds.

            I'd prefer a second sentence added to the docstring along the lines of, "May be in the range 0-1; default 0.2 will flag the coadd if at least 20% of its inputs are flagged." I didn't fully understand that's what was happening here until you pointed me to the task description.

            Show
            mrawls Meredith Rawls added a comment - Indeed, that provides plenty of justification, but is a rather odd place for it. I think I was further confused by the default for the following config field, matchRadius, which also happens to be 0.2... arcseconds. I'd prefer a second sentence added to the docstring along the lines of, "May be in the range 0-1; default 0.2 will flag the coadd if at least 20% of its inputs are flagged." I didn't fully understand that's what was happening here until you pointed me to the task description.
            Hide
            lauren Lauren MacArthur added a comment -

            The config parameter doc now reads (with a --show config=propagateFlags* on a multiband.py run):

            # Source catalog flags to propagate, with the threshold of relative occurrence (valid range: [0-1], default is 0.2). Coadd object will have flag set if the fraction of input visits in which it is flagged is greater than the threshold.
            config.measureCoaddSources.propagateFlags.flags={'calib_psf_candidate': 0.2, 'calib_psf_used': 0.2, 'calib_psf_reserved': 0.2, 'calib_astrometry_used': 0.2, 'calib_photometry_used': 0.2, 'calib_photometry_reserved': 0.2}

            Show
            lauren Lauren MacArthur added a comment - The config parameter doc now reads (with a --show config=propagateFlags* on a multiband.py run): # Source catalog flags to propagate, with the threshold of relative occurrence (valid range: [0-1], default is 0.2). Coadd object will have flag set if the fraction of input visits in which it is flagged is greater than the threshold. config.measureCoaddSources.propagateFlags.flags={'calib_psf_candidate': 0.2, 'calib_psf_used': 0.2, 'calib_psf_reserved': 0.2, 'calib_astrometry_used': 0.2, 'calib_photometry_used': 0.2, 'calib_photometry_reserved': 0.2}
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks Meredith.  Docstring updated and merged to master.

            Show
            lauren Lauren MacArthur added a comment - Thanks Meredith.  Docstring updated and merged to master.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Meredith Rawls
              Watchers:
              John Swinbank, Lauren MacArthur, Meredith Rawls, Yusra AlSayyad
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.