Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25782

Add an "detect_isPrimary" flag to single frame processing schema

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
    • Story Points:
      9
    • Sprint:
      DRP F20-1 (June), DRP F20-2 (July)
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      The coadd level catalogs include a flag:

      "detect_isPrimary", type="Flag",
       doc="true if source has no children and is in the inner region of a coadd patch "
                "and is in the inner region of a coadd tract "
                "and is not \"detected\" in a pseudo-filter (see config.pseudoFilterList)",
      

      that is very useful for culling catalogs of duplicate (patch/tract overlap), non-deblended parent sources (deblend_nChild>0, and sky objects. No equivalent flag currently exists at the visit /single frame measurement level, so this ticket is to add such a flag. Currently, the only two conditions to be added to flag undesirable sources are:
      deblend_nChild == 0
      sky_source is False

      A community post describing this feature may be found here: https://community.lsst.org/t/primary-source-flag-for-single-frame-processing/4425

        Attachments

          Issue Links

            Activity

            Hide
            lskelvin Lee Kelvin added a comment -

            Thanks for taking a look at this ticket Meredith, much appreciated. We pushed it through Jenkins yesterday without issue, so hopefully it should be good to go.

            Show
            lskelvin Lee Kelvin added a comment - Thanks for taking a look at this ticket Meredith, much appreciated. We pushed it through Jenkins yesterday without issue, so hopefully it should be good to go.
            Hide
            mrawls Meredith Rawls added a comment -

            This adds a useful functionality, thanks for implementing it. Please take a look at my GitHub comments. The high-level question I have is, where do these flags ultimately land? In the source catalog as a new column alongside e.g. all the pixel flags? The definition of "primary" is a little funny to me since it really means "is a real source which has no child sources," so please take every opportunity in the code to remind us that is what is going on.

            Show
            mrawls Meredith Rawls added a comment - This adds a useful functionality, thanks for implementing it. Please take a look at my GitHub comments. The high-level question I have is, where do these flags ultimately land? In the source catalog as a new column alongside e.g. all the pixel flags? The definition of "primary" is a little funny to me since it really means "is a real source which has no child sources," so please take every opportunity in the code to remind us that is what is going on.
            Hide
            lskelvin Lee Kelvin added a comment -

            Thanks Meredith, these look great. To answer your high-level question, these flags are output in the main single-frame catalog (i.e., butler.get('src', ...)). The output column in the table schema looks like this: (Field['Flag'](name="detect_isPrimary", doc="true if source has no children and is not a sky source"), Key['Flag'](offset=624, bit=47)).

            I'll start taking a look at your comments on GitHub just now and respond further there. Wherever possible, I'll try to remind the reader that this new detect_isPrimary flag returns True for all sources which are not sky sources and which have no children. This flag is meant to allow the end-user to easily select a useful list of science sources, and so hopefully should come in handy.

            Show
            lskelvin Lee Kelvin added a comment - Thanks Meredith, these look great. To answer your high-level question, these flags are output in the main single-frame catalog (i.e., butler.get('src', ...)). The output column in the table schema looks like this: (Field ['Flag'] (name="detect_isPrimary", doc="true if source has no children and is not a sky source"), Key ['Flag'] (offset=624, bit=47)) . I'll start taking a look at your comments on GitHub just now and respond further there. Wherever possible, I'll try to remind the reader that this new detect_isPrimary flag returns True for all sources which are not sky sources and which have no children. This flag is meant to allow the end-user to easily select a useful list of science sources, and so hopefully should come in handy.
            Hide
            lskelvin Lee Kelvin added a comment - - edited

            Thanks again for all the comments, much appreciated. I've updated the PR with the changes suggested. Does that look okay? The branch has been rebased to master, and I've ran another Jenkins as a safety check.

            Show
            lskelvin Lee Kelvin added a comment - - edited Thanks again for all the comments, much appreciated. I've updated the PR with the changes suggested. Does that look okay? The branch has been rebased to master, and I've ran another Jenkins as a safety check.
            Hide
            mrawls Meredith Rawls added a comment -

            Looks great, thanks for addressing my comments. Good to merge assuming the rebasing and Jenkins are happy.

            Show
            mrawls Meredith Rawls added a comment - Looks great, thanks for addressing my comments. Good to merge assuming the rebasing and Jenkins are happy.
            Hide
            lskelvin Lee Kelvin added a comment -

            Thanks! Merged to master and removed the branch.

            Show
            lskelvin Lee Kelvin added a comment - Thanks! Merged to master and removed the branch.
            Hide
            yusra Yusra AlSayyad added a comment -

            Lee Kelvin This new column never made it into the parquet Source Table: https://github.com/lsst/obs_subaru/blob/master/policy/Source.yaml
            Please open a new ticket to add.

            Show
            yusra Yusra AlSayyad added a comment - Lee Kelvin This new column never made it into the parquet Source Table: https://github.com/lsst/obs_subaru/blob/master/policy/Source.yaml Please open a new ticket to add.
            Hide
            lskelvin Lee Kelvin added a comment -

            Thanks Yusra AlSayyad, ticket DM-26438 has been set up to address this.

            Show
            lskelvin Lee Kelvin added a comment - Thanks Yusra AlSayyad , ticket DM-26438 has been set up to address this.

              People

              Assignee:
              lskelvin Lee Kelvin
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Meredith Rawls
              Watchers:
              Jim Bosch, Lauren MacArthur, Lee Kelvin, Meredith Rawls, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.