Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11866

Propagate astrometry and photometry visit calibration flags to coadds

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • pipe_tasks
    • None
    • 6
    • DRP F18-4
    • Data Release Production

    Description

      DM-4878 added a mechanism for propagating flags from the input catalogues to the coadd catalogue indicating stars that were used for measuring the PSF. Since then, DM-9050 added similar calibration flags for astrometry and photometry which should also be added to the list of flags that get propagated from visits to coadds.

      The default list of propagated flags is currently:

      flags = DictField(keytype=str, itemtype=float,
                            default={"calib_psf_candidate": 0.2, "calib_psf_used": 0.2, },
                            doc="Source catalog flags to propagate, with the threshold of relative occurrence.")
      

      Please add the following to this list:

      • calib_astrometry_used
      • calib_photometry_used
      • calib_photometry_reserved

      Attachments

        Issue Links

          Activity

            lauren Lauren MacArthur added a comment - - edited

            Would you mind giving this a look?  Seeing as you have been poking around the plots from pipe_analysis, you understand the motivation.

            Unit testing is not really feasible as we need a fully processed (through multiband) dataset, so I added a test in ci_hsc.

            The PR for pipe_tasks (which is not showing up in Jira these days) is: https://github.com/lsst/pipe_tasks/pull/217

            I ran a Jenkins run before the weekend which passed. Just to be on the safe side, I also kicked one off today [UPDATE: it also passed].

            Oh, and I also created a coadd and checked that the sources that get these flags set do indeed look sensible.

            lauren Lauren MacArthur added a comment - - edited Would you mind giving this a look?  Seeing as you have been poking around the plots from pipe_analysis, you understand the motivation. Unit testing is not really feasible as we need a fully processed (through multiband) dataset, so I added a test in ci_hsc . The PR for pipe_tasks (which is not showing up in Jira these days) is: https://github.com/lsst/pipe_tasks/pull/217 I ran a Jenkins run before the weekend which passed . Just to be on the safe side, I also kicked one off today [UPDATE: it also passed] . Oh, and I also created a coadd and checked that the sources that get these flags set do indeed look sensible.

            Looks good to me. The only thing I wasn't clear about is why the default values are all 0.2. If you could possibly clarify that in the docstring, that would be helpful.

            mrawls Meredith Rawls added a comment - Looks good to me. The only thing I wasn't clear about is why the default values are all 0.2. If you could possibly clarify that in the docstring, that would be helpful.

            Very fair question.  There is a lengthy discussion in the task description here: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/propagateVisitFlags.py#L64-L90

            This seemed like more than enough justification to me, and I felt the logic also applies to the flags added here.  Let me know if you disagree or think a note needs to be added elsewhere to make it more visible.

            lauren Lauren MacArthur added a comment - Very fair question.  There is a lengthy discussion in the task description here: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/propagateVisitFlags.py#L64-L90 This seemed like more than enough justification to me, and I felt the logic also applies to the flags added here.  Let me know if you disagree or think a note needs to be added elsewhere to make it more visible.

            Indeed, that provides plenty of justification, but is a rather odd place for it. I think I was further confused by the default for the following config field, matchRadius, which also happens to be 0.2... arcseconds.

            I'd prefer a second sentence added to the docstring along the lines of, "May be in the range 0-1; default 0.2 will flag the coadd if at least 20% of its inputs are flagged." I didn't fully understand that's what was happening here until you pointed me to the task description.

            mrawls Meredith Rawls added a comment - Indeed, that provides plenty of justification, but is a rather odd place for it. I think I was further confused by the default for the following config field, matchRadius, which also happens to be 0.2... arcseconds. I'd prefer a second sentence added to the docstring along the lines of, "May be in the range 0-1; default 0.2 will flag the coadd if at least 20% of its inputs are flagged." I didn't fully understand that's what was happening here until you pointed me to the task description.

            The config parameter doc now reads (with a --show config=propagateFlags* on a multiband.py run):

            # Source catalog flags to propagate, with the threshold of relative occurrence (valid range: [0-1], default is 0.2). Coadd object will have flag set if the fraction of input visits in which it is flagged is greater than the threshold.
            config.measureCoaddSources.propagateFlags.flags={'calib_psf_candidate': 0.2, 'calib_psf_used': 0.2, 'calib_psf_reserved': 0.2, 'calib_astrometry_used': 0.2, 'calib_photometry_used': 0.2, 'calib_photometry_reserved': 0.2}

            lauren Lauren MacArthur added a comment - The config parameter doc now reads (with a --show config=propagateFlags* on a multiband.py run): # Source catalog flags to propagate, with the threshold of relative occurrence (valid range: [0-1], default is 0.2). Coadd object will have flag set if the fraction of input visits in which it is flagged is greater than the threshold. config.measureCoaddSources.propagateFlags.flags={'calib_psf_candidate': 0.2, 'calib_psf_used': 0.2, 'calib_psf_reserved': 0.2, 'calib_astrometry_used': 0.2, 'calib_photometry_used': 0.2, 'calib_photometry_reserved': 0.2}

            Thanks Meredith.  Docstring updated and merged to master.

            lauren Lauren MacArthur added a comment - Thanks Meredith.  Docstring updated and merged to master.

            People

              lauren Lauren MacArthur
              lauren Lauren MacArthur
              Meredith Rawls
              John Swinbank, Lauren MacArthur, Meredith Rawls, Yusra AlSayyad
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.