Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16070

Implement RFC-534: Update naming of base_Blendedness fields

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: meas_base, meas_deblender
    • Labels:
      None
    • Story Points:
      3
    • Sprint:
      DRP F18-5
    • Team:
      Data Release Production

      Description

      The implementation of this RFC includes:

      1. strip "_instFlux" from the base_Blendedness_raw_instFlux and base_Blendedness_abs_instFlux names
      2. move "_instFlux" to the end of the name in the fields with actual flux units, thus
      3. base_Blendedness_raw_instFlux_child --> base_Blendedness_raw_child_instFlux
      4. base_Blendedness_raw_instFlux_parent --> base_Blendedness_raw_parent_instFlux
      5. base_Blendedness_abs_instFlux_child --> base_Blendedness_abs_child_instFlux
      6. base_Blendedness_abs_instFlux_parent --> base_Blendedness_abs_parent_instFlux
      7. an update to the doc strings to make a clear distinction/description between "raw" and "abs"

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            As per discussion on DM-16068, this will also include the following change for the field:

            name="deblend_psfFlux", doc="If deblended-as-psf, the PSF flux"
            

            deblend_psfFlux --> deblend_psf_instFlux

            Show
            lauren Lauren MacArthur added a comment - As per discussion on DM-16068 , this will also include the following change for the field: name = "deblend_psfFlux" , doc = "If deblended-as-psf, the PSF flux" deblend_psfFlux --> deblend_psf_instFlux
            Hide
            lauren Lauren MacArthur added a comment - - edited

            I'm thinking I should also change these:

            name="deblend_psfCenter_x", doc="If deblended-as-psf, the PSF centroid", units="pixel"
            name="deblend_psfCenter_y", doc="If deblended-as-psf, the PSF centroid", units="pixel"
            

            to

            name="deblend_PsfCentroid_x"
            name="deblend_PsfCentroid_y"
            

            to match, e.g.

            name="base_SdssCentroid_x"


            and that the flux name above should actually be

            name="deblend_PsfFlux_instFlux"

            to match, e.g.

            name="base_PsfFlux_instFlux"

            Jim Bosch do you agree?

            Show
            lauren Lauren MacArthur added a comment - - edited I'm thinking I should also change these: name = "deblend_psfCenter_x" , doc = "If deblended-as-psf, the PSF centroid" , units = "pixel" name = "deblend_psfCenter_y" , doc = "If deblended-as-psf, the PSF centroid" , units = "pixel" to name = "deblend_PsfCentroid_x" name = "deblend_PsfCentroid_y" to match, e.g. name = "base_SdssCentroid_x" and that the flux name above should actually be name = "deblend_PsfFlux_instFlux" to match, e.g. name = "base_PsfFlux_instFlux" Jim Bosch do you agree?
            Hide
            jbosch Jim Bosch added a comment -

            The convention we use for measurement plugin fields isn't as clearly applicable here, as the deblender isn't a measurement plugin.  In particular, while plugin names are supposed to be distinct and achieve this by embedding the class name in the field names (because they can all be run together at once), different deblenders are mutually exclusive in the same catalog (at least the way things are structured now), so it's more desirable that their fields have the same names.  I think it's probably best to minimize changes to those except when necessary to make sure:

            • instrumental fluxes end in _instFlux
            • centroids end in _x, _y
            • shapes end in _xx, _yy, _xy

            In other words, as long as the last bit of the suffix is ok, I think the names can mostly stay as-is.

            Show
            jbosch Jim Bosch added a comment - The convention we use for measurement plugin fields isn't as clearly applicable here, as the deblender isn't a measurement plugin.  In particular, while plugin names are supposed to be distinct and achieve this by embedding the class name in the field names (because they can all be run together at once), different deblenders are mutually exclusive in the same catalog (at least the way things are structured now), so it's more desirable that their fields have the same names.  I think it's probably best to minimize changes to those except when necessary to make sure: instrumental fluxes end in _instFlux centroids end in _x, _y shapes end in _xx, _yy, _xy In other words, as long as the last bit of the suffix is ok, I think the names can mostly stay as-is.
            Hide
            lauren Lauren MacArthur added a comment -

            Awesome, thanks...I will stick to what was on tap before my previous comment!

            Show
            lauren Lauren MacArthur added a comment - Awesome, thanks...I will stick to what was on tap before my previous comment!
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Jim, could you give this a quick look to make sure I've made the changes as proposed. I ran multiBandDriver.py (pointing at the outputs from the w_2018_41 processing at /datasets/hsc/repo/rerun/RC/w_2018_41/DM-16011/ on lsst-dev as input: output directory is `datasets/hsc/repo/rerun/private/lauren/DM-16070multi/`. The new fields in the deepCoadd_meas schema look like this:

                (Field['D'](name="base_Blendedness_raw", doc="Measure of how much instFlux is affected by neighbors: (1 - child_instFlux/parent_instFlux).  Operates on the "raw" pixel values."), Key<D>(offset=120, nElements=1)),
                (Field['D'](name="base_Blendedness_raw_child_instFlux", doc="instFlux of the child, measured with a Gaussian weight matched to the child.  Operates on the "raw" pixel values.", units="count"), Key<D>(offset=128, nElements=1)),
                (Field['D'](name="base_Blendedness_raw_parent_instFlux", doc="instFlux of the parent, measured with a Gaussian weight matched to the child.  Operates on the "raw" pixel values.", units="count"), Key<D>(offset=136, nElements=1)),
                (Field['D'](name="base_Blendedness_abs", doc="Measure of how much instFlux is affected by neighbors: (1 - child_instFlux/parent_instFlux).  Operates on the absolute value of the pixels to try to obtain a "de-noised" value.  See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details."), Key<D>(offset=144, nElements=1)),
                (Field['D'](name="base_Blendedness_abs_child_instFlux", doc="instFlux of the child, measured with a Gaussian weight matched to the child.  Operates on the absolute value of the pixels to try to obtain a "de-noised" value.  See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details.", units="count"), Key<D>(offset=152, nElements=1)),
                (Field['D'](name="base_Blendedness_abs_parent_instFlux", doc="instFlux of the parent, measured with a Gaussian weight matched to the child.  Operates on the absolute value of the pixels to try to obtain a "de-noised" value.  See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details.", units="count"), Key<D>(offset=160, nElements=1)),
                (Field['D'](name="base_Blendedness_raw_child_xx", doc="Shape of the child, measured with a Gaussian weight matched to the child.  Operates on the "raw" pixel values.", units="pixel^2"), Key<D>(offset=168, nElements=1)),
                (Field['D'](name="base_Blendedness_raw_child_yy", doc="Shape of the child, measured with a Gaussian weight matched to the child.  Operates on the "raw" pixel values.", units="pixel^2"), Key<D>(offset=176, nElements=1)),
                (Field['D'](name="base_Blendedness_raw_child_xy", doc="Shape of the child, measured with a Gaussian weight matched to the child.  Operates on the "raw" pixel values.", units="pixel^2"), Key<D>(offset=184, nElements=1)),
                (Field['D'](name="base_Blendedness_raw_parent_xx", doc="Shape of the parent, measured with a Gaussian weight matched to the child.  Operates on the "raw" pixel values.", units="pixel^2"), Key<D>(offset=192, nElements=1)),
                (Field['D'](name="base_Blendedness_raw_parent_yy", doc="Shape of the parent, measured with a Gaussian weight matched to the child.  Operates on the "raw" pixel values.", units="pixel^2"), Key<D>(offset=200, nElements=1)),
                (Field['D'](name="base_Blendedness_raw_parent_xy", doc="Shape of the parent, measured with a Gaussian weight matched to the child.  Operates on the "raw" pixel values.", units="pixel^2"), Key<D>(offset=208, nElements=1)),
                (Field['D'](name="base_Blendedness_abs_child_xx", doc="Shape of the child, measured with a Gaussian weight matched to the child.  Operates on the absolute value of the pixels to try to obtain a "de-noised" value.  See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details.", units="pixel^2"), Key<D>(offset=216, nElements=1)),
                (Field['D'](name="base_Blendedness_abs_child_yy", doc="Shape of the child, measured with a Gaussian weight matched to the child.  Operates on the absolute value of the pixels to try to obtain a "de-noised" value.  See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details.", units="pixel^2"), Key<D>(offset=224, nElements=1)),
                (Field['D'](name="base_Blendedness_abs_child_xy", doc="Shape of the child, measured with a Gaussian weight matched to the child.  Operates on the absolute value of the pixels to try to obtain a "de-noised" value.  See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details.", units="pixel^2"), Key<D>(offset=232, nElements=1)),
                (Field['D'](name="base_Blendedness_abs_parent_xx", doc="Shape of the parent, measured with a Gaussian weight matched to the child.  Operates on the absolute value of the pixels to try to obtain a "de-noised" value.  See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details.", units="pixel^2"), Key<D>(offset=240, nElements=1)),
                (Field['D'](name="base_Blendedness_abs_parent_yy", doc="Shape of the parent, measured with a Gaussian weight matched to the child.  Operates on the absolute value of the pixels to try to obtain a "de-noised" value.  See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details.", units="pixel^2"), Key<D>(offset=248, nElements=1)),
                (Field['D'](name="base_Blendedness_abs_parent_xy", doc="Shape of the parent, measured with a Gaussian weight matched to the child.  Operates on the absolute value of the pixels to try to obtain a "de-noised" value.  See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details.", units="pixel^2"), Key<D>(offset=256, nElements=1))
            

            I ran a successful lsst_distrib lsst_ci ci_hsc Jenkins over the weekend, but made a few updates today (and added COPYRIGHT & LICENSE files), so another one is running (it has PASSED).

            PRs:
            https://github.com/lsst/meas_base/pull/132
            https://github.com/lsst/meas_deblender/pull/62

            I will also write a brief community post outlining the changes once this gets merged.

            Show
            lauren Lauren MacArthur added a comment - - edited Jim, could you give this a quick look to make sure I've made the changes as proposed. I ran multiBandDriver.py (pointing at the outputs from the w_2018_41 processing at /datasets/hsc/repo/rerun/RC/w_2018_41/ DM-16011 / on lsst-dev as input: output directory is `datasets/hsc/repo/rerun/private/lauren/ DM-16070 multi/`. The new fields in the deepCoadd_meas schema look like this: (Field[ 'D' ](name = "base_Blendedness_raw" , doc = "Measure of how much instFlux is affected by neighbors: (1 - child_instFlux/parent_instFlux). Operates on the " raw " pixel values." ), Key<D>(offset = 120 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_raw_child_instFlux" , doc = "instFlux of the child, measured with a Gaussian weight matched to the child. Operates on the " raw " pixel values." , units = "count" ), Key<D>(offset = 128 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_raw_parent_instFlux" , doc = "instFlux of the parent, measured with a Gaussian weight matched to the child. Operates on the " raw " pixel values." , units = "count" ), Key<D>(offset = 136 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_abs" , doc = "Measure of how much instFlux is affected by neighbors: (1 - child_instFlux/parent_instFlux). Operates on the absolute value of the pixels to try to obtain a " de - noised " value. See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details." ), Key<D>(offset = 144 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_abs_child_instFlux" , doc = "instFlux of the child, measured with a Gaussian weight matched to the child. Operates on the absolute value of the pixels to try to obtain a " de - noised " value. See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details." , units = "count" ), Key<D>(offset = 152 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_abs_parent_instFlux" , doc = "instFlux of the parent, measured with a Gaussian weight matched to the child. Operates on the absolute value of the pixels to try to obtain a " de - noised " value. See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details." , units = "count" ), Key<D>(offset = 160 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_raw_child_xx" , doc = "Shape of the child, measured with a Gaussian weight matched to the child. Operates on the " raw " pixel values." , units = "pixel^2" ), Key<D>(offset = 168 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_raw_child_yy" , doc = "Shape of the child, measured with a Gaussian weight matched to the child. Operates on the " raw " pixel values." , units = "pixel^2" ), Key<D>(offset = 176 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_raw_child_xy" , doc = "Shape of the child, measured with a Gaussian weight matched to the child. Operates on the " raw " pixel values." , units = "pixel^2" ), Key<D>(offset = 184 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_raw_parent_xx" , doc = "Shape of the parent, measured with a Gaussian weight matched to the child. Operates on the " raw " pixel values." , units = "pixel^2" ), Key<D>(offset = 192 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_raw_parent_yy" , doc = "Shape of the parent, measured with a Gaussian weight matched to the child. Operates on the " raw " pixel values." , units = "pixel^2" ), Key<D>(offset = 200 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_raw_parent_xy" , doc = "Shape of the parent, measured with a Gaussian weight matched to the child. Operates on the " raw " pixel values." , units = "pixel^2" ), Key<D>(offset = 208 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_abs_child_xx" , doc = "Shape of the child, measured with a Gaussian weight matched to the child. Operates on the absolute value of the pixels to try to obtain a " de - noised " value. See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details." , units = "pixel^2" ), Key<D>(offset = 216 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_abs_child_yy" , doc = "Shape of the child, measured with a Gaussian weight matched to the child. Operates on the absolute value of the pixels to try to obtain a " de - noised " value. See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details." , units = "pixel^2" ), Key<D>(offset = 224 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_abs_child_xy" , doc = "Shape of the child, measured with a Gaussian weight matched to the child. Operates on the absolute value of the pixels to try to obtain a " de - noised " value. See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details." , units = "pixel^2" ), Key<D>(offset = 232 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_abs_parent_xx" , doc = "Shape of the parent, measured with a Gaussian weight matched to the child. Operates on the absolute value of the pixels to try to obtain a " de - noised " value. See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details." , units = "pixel^2" ), Key<D>(offset = 240 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_abs_parent_yy" , doc = "Shape of the parent, measured with a Gaussian weight matched to the child. Operates on the absolute value of the pixels to try to obtain a " de - noised " value. See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details." , units = "pixel^2" ), Key<D>(offset = 248 , nElements = 1 )), (Field[ 'D' ](name = "base_Blendedness_abs_parent_xy" , doc = "Shape of the parent, measured with a Gaussian weight matched to the child. Operates on the absolute value of the pixels to try to obtain a " de - noised " value. See section 4.9.11 of Bosch et al. 2018, PASJ, 70, S5 for details." , units = "pixel^2" ), Key<D>(offset = 256 , nElements = 1 )) I ran a successful lsst_distrib lsst_ci ci_hsc Jenkins over the weekend, but made a few updates today (and added COPYRIGHT & LICENSE files), so another one is running (it has PASSED ). PRs: https://github.com/lsst/meas_base/pull/132 https://github.com/lsst/meas_deblender/pull/62 I will also write a brief community post outlining the changes once this gets merged.
            Hide
            jbosch Jim Bosch added a comment -

            Made a few suggestions about docstrings on the PRs.  Feel free to merge after you think you've addressed them; no need to send it back to me.

            Show
            jbosch Jim Bosch added a comment - Made a few suggestions about docstrings on the PRs.  Feel free to merge after you think you've addressed them; no need to send it back to me.
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Thanks for getting to this so quickly.  I made all your suggested changes and am rerunning Jenkins just to be safe.

            Show
            lauren Lauren MacArthur added a comment - - edited Thanks for getting to this so quickly.  I made all your suggested changes and am rerunning Jenkins just to be safe.

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                lauren Lauren MacArthur
                Reviewers:
                Jim Bosch
                Watchers:
                Jim Bosch, John Parejko, Lauren MacArthur, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel