# Add an "detect_isPrimary" flag to single frame processing schema

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
9
• Sprint:
DRP F20-1 (June), DRP F20-2 (July)
• Team:
Data Release Production
• Urgent?:
No

#### Description

The coadd level catalogs include a flag:

 "detect_isPrimary", type="Flag",  doc="true if source has no children and is in the inner region of a coadd patch "  "and is in the inner region of a coadd tract "  "and is not \"detected\" in a pseudo-filter (see config.pseudoFilterList)", 

that is very useful for culling catalogs of duplicate (patch/tract overlap), non-deblended parent sources (deblend_nChild>0, and sky objects. No equivalent flag currently exists at the visit /single frame measurement level, so this ticket is to add such a flag. Currently, the only two conditions to be added to flag undesirable sources are:
deblend_nChild == 0
sky_source is False

A community post describing this feature may be found here: https://community.lsst.org/t/primary-source-flag-for-single-frame-processing/4425

#### Activity

Lauren MacArthur created issue -
Field Original Value New Value
Epic Link DM-25271 [ 435631 ]
 Description The coadd level catalogs include a flag: {code:python} "detect_isPrimary", type="Flag",  doc="true if source has no children and is in the inner region of a coadd patch "           "and is in the inner region of a coadd tract "           "and is not \"detected\" in a pseudo-filter (see config.pseudoFilterList)", {code} that is very useful for culling catalogs of duplicate (patch/tract overlap), non-deblended parent sources ({{detect_nChild>0}}, and sky objects. No equivalent flag currently exists at the visit /single frame measurement level, so this ticket is to add such a flag. Currently, the only two conditions to be added to flag undesirable sources are: {{detect_nChild > 0}} {{sky_source is True}} The coadd level catalogs include a flag: {code:python}"detect_isPrimary", type="Flag",  doc="true if source has no children and is in the inner region of a coadd patch "           "and is in the inner region of a coadd tract "           "and is not \"detected\" in a pseudo-filter (see config.pseudoFilterList)", {code} that is very useful for culling catalogs of duplicate (patch/tract overlap), non-deblended parent sources ({{deblend_nChild>0}}, and sky objects. No equivalent flag currently exists at the visit /single frame measurement level, so this ticket is to add such a flag. Currently, the only two conditions to be added to flag undesirable sources are:  {{deblend_nChild > 0}}  {{sky_source is True}}
 Status To Do [ 10001 ] In Progress [ 3 ]
 Description The coadd level catalogs include a flag: {code:python}"detect_isPrimary", type="Flag",  doc="true if source has no children and is in the inner region of a coadd patch "           "and is in the inner region of a coadd tract "           "and is not \"detected\" in a pseudo-filter (see config.pseudoFilterList)", {code} that is very useful for culling catalogs of duplicate (patch/tract overlap), non-deblended parent sources ({{deblend_nChild>0}}, and sky objects. No equivalent flag currently exists at the visit /single frame measurement level, so this ticket is to add such a flag. Currently, the only two conditions to be added to flag undesirable sources are:  {{deblend_nChild > 0}}  {{sky_source is True}} The coadd level catalogs include a flag: {code:python}"detect_isPrimary", type="Flag",  doc="true if source has no children and is in the inner region of a coadd patch "           "and is in the inner region of a coadd tract "           "and is not \"detected\" in a pseudo-filter (see config.pseudoFilterList)", {code} that is very useful for culling catalogs of duplicate (patch/tract overlap), non-deblended parent sources ({{deblend_nChild>0}}, and sky objects. No equivalent flag currently exists at the visit /single frame measurement level, so this ticket is to add such a flag. Currently, the only two conditions to be added to flag undesirable sources are:  {{deblend_nChild == 0}}  {{sky_source is False}}
 Sprint DRP F20-1 (June) [ 1030 ] DRP F20-1 (June), DRP F20-2 (July) [ 1030, 1036 ]
Hide
Lee Kelvin added a comment -

Thanks for taking a look at this ticket Meredith, much appreciated. We pushed it through Jenkins yesterday without issue, so hopefully it should be good to go.

Show
Lee Kelvin added a comment - Thanks for taking a look at this ticket Meredith, much appreciated. We pushed it through Jenkins yesterday without issue, so hopefully it should be good to go.
 Reviewers Meredith Rawls [ mrawls ] Status In Progress [ 3 ] In Review [ 10004 ]
Hide
Meredith Rawls added a comment -

This adds a useful functionality, thanks for implementing it. Please take a look at my GitHub comments. The high-level question I have is, where do these flags ultimately land? In the source catalog as a new column alongside e.g. all the pixel flags? The definition of "primary" is a little funny to me since it really means "is a real source which has no child sources," so please take every opportunity in the code to remind us that is what is going on.

Show
Meredith Rawls added a comment - This adds a useful functionality, thanks for implementing it. Please take a look at my GitHub comments. The high-level question I have is, where do these flags ultimately land? In the source catalog as a new column alongside e.g. all the pixel flags? The definition of "primary" is a little funny to me since it really means "is a real source which has no child sources," so please take every opportunity in the code to remind us that is what is going on.
Hide
Lee Kelvin added a comment -

Thanks Meredith, these look great. To answer your high-level question, these flags are output in the main single-frame catalog (i.e., butler.get('src', ...)). The output column in the table schema looks like this: (Field['Flag'](name="detect_isPrimary", doc="true if source has no children and is not a sky source"), Key['Flag'](offset=624, bit=47)).

I'll start taking a look at your comments on GitHub just now and respond further there. Wherever possible, I'll try to remind the reader that this new detect_isPrimary flag returns True for all sources which are not sky sources and which have no children. This flag is meant to allow the end-user to easily select a useful list of science sources, and so hopefully should come in handy.

Show
Lee Kelvin added a comment - Thanks Meredith, these look great. To answer your high-level question, these flags are output in the main single-frame catalog (i.e., butler.get('src', ...)). The output column in the table schema looks like this: (Field ['Flag'] (name="detect_isPrimary", doc="true if source has no children and is not a sky source"), Key ['Flag'] (offset=624, bit=47)) . I'll start taking a look at your comments on GitHub just now and respond further there. Wherever possible, I'll try to remind the reader that this new detect_isPrimary flag returns True for all sources which are not sky sources and which have no children. This flag is meant to allow the end-user to easily select a useful list of science sources, and so hopefully should come in handy.
Hide
Lee Kelvin added a comment - - edited

Thanks again for all the comments, much appreciated. I've updated the PR with the changes suggested. Does that look okay? The branch has been rebased to master, and I've ran another Jenkins as a safety check.

Show
Lee Kelvin added a comment - - edited Thanks again for all the comments, much appreciated. I've updated the PR with the changes suggested. Does that look okay? The branch has been rebased to master, and I've ran another Jenkins as a safety check.
Hide
Meredith Rawls added a comment -

Looks great, thanks for addressing my comments. Good to merge assuming the rebasing and Jenkins are happy.

Show
Meredith Rawls added a comment - Looks great, thanks for addressing my comments. Good to merge assuming the rebasing and Jenkins are happy.
 Status In Review [ 10004 ] Reviewed [ 10101 ]
Hide
Lee Kelvin added a comment -

Thanks! Merged to master and removed the branch.

Show
Lee Kelvin added a comment - Thanks! Merged to master and removed the branch.
 Resolution Done [ 10000 ] Status Reviewed [ 10101 ] Done [ 10002 ]
 Story Points 12
Hide

Lee Kelvin This new column never made it into the parquet Source Table: https://github.com/lsst/obs_subaru/blob/master/policy/Source.yaml

Show
 Link This issue relates to DM-26438 [ DM-26438 ]
Hide
Lee Kelvin added a comment -

Thanks Yusra AlSayyad, ticket DM-26438 has been set up to address this.

Show
Lee Kelvin added a comment - Thanks Yusra AlSayyad , ticket DM-26438 has been set up to address this.
 Story Points 12 9
 Description The coadd level catalogs include a flag: {code:python}"detect_isPrimary", type="Flag",  doc="true if source has no children and is in the inner region of a coadd patch "           "and is in the inner region of a coadd tract "           "and is not \"detected\" in a pseudo-filter (see config.pseudoFilterList)", {code} that is very useful for culling catalogs of duplicate (patch/tract overlap), non-deblended parent sources ({{deblend_nChild>0}}, and sky objects. No equivalent flag currently exists at the visit /single frame measurement level, so this ticket is to add such a flag. Currently, the only two conditions to be added to flag undesirable sources are:  {{deblend_nChild == 0}}  {{sky_source is False}} The coadd level catalogs include a flag: {code:python}"detect_isPrimary", type="Flag",  doc="true if source has no children and is in the inner region of a coadd patch "           "and is in the inner region of a coadd tract "           "and is not \"detected\" in a pseudo-filter (see config.pseudoFilterList)", {code} that is very useful for culling catalogs of duplicate (patch/tract overlap), non-deblended parent sources ({{deblend_nChild>0}}, and sky objects. No equivalent flag currently exists at the visit /single frame measurement level, so this ticket is to add such a flag. Currently, the only two conditions to be added to flag undesirable sources are:  {{deblend_nChild == 0}}  {{sky_source is False}} A community post describing this feature may be found here: https://community.lsst.org/t/primary-source-flag-for-single-frame-processing/4425

#### People

Assignee:
Lee Kelvin
Reporter:
Lauren MacArthur
Reviewers:
Meredith Rawls
Watchers:
Jim Bosch, Lauren MacArthur, Lee Kelvin, Meredith Rawls, Yusra AlSayyad