Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14075

Exclude bad mask plane in nImage for filtering artifact candidates

    Details

      Description

      Symptom: Artifact candidates are being clipped when there not sufficient epochs remaining to do so.

      Cause: A crude N-Image is passed to filterArtifacts to compute the maximum number of epochs a candidate can appear in and be clipped. If there are < 3 epochs it's zero. (i.e nothing should be clipped). This N-Image is too crude.

      The actual N-Image can be lower due to recorded chip defects (everything in the bad mask plane).

      NOTE: This only affects coadds with small numbers of epochs

        Attachments

          Activity

          Hide
          yusra Yusra AlSayyad added a comment -

          This fix passes an N-Image (RIGHT) to filter artifacts that better approximates the final N-Image (MIDDLE).

          Before:
          After:

          Note: The N-Image passed to the filterArtifacts still isn't exactly the same as the final nImage - CLIPPED because it's using the psfMatched masks rather than the direct masks. IMO it is worth the extra disk access to read in the direct warps for just to to make an NImage to filterArtifacts, because the filter method still needs to make a statistical decision on whether to filter the candidate or not (the candidate will not entirely lie in a 2-epoch region or a 3-epoch region).

          Show
          yusra Yusra AlSayyad added a comment - This fix passes an N-Image (RIGHT) to filter artifacts that better approximates the final N-Image (MIDDLE). Before: After: Note: The N-Image passed to the filterArtifacts still isn't exactly the same as the final nImage - CLIPPED because it's using the psfMatched masks rather than the direct masks. IMO it is worth the extra disk access to read in the direct warps for just to to make an NImage to filterArtifacts, because the filter method still needs to make a statistical decision on whether to filter the candidate or not (the candidate will not entirely lie in a 2-epoch region or a 3-epoch region).
          Hide
          yusra Yusra AlSayyad added a comment -

          Small change in how the approximate nImage is computed which is used to compute the temporal threshold of artifact rejection.

          Jenkins:
          https://ci.lsst.codes/job/stack-os-matrix/27816/
          https://ci.lsst.codes/job/stack-os-matrix/27817/

          I ran the whole RC2 dataset through to ensure there were no unintended consequences:
          /datasets/hsc/repo/rerun/private/yusra/RC2/DM-14075

          I visually inspsected them and compared coadds bitwise with /datasets/hsc/repo/rerun/RC/w_2018_14/DM-13890
          1) All changes have been improvements
          2) It doesn't only affect patches with low max number of visits because in any coadd, there will be regions (e.g.around bright stars) with local N < 3.

          Show
          yusra Yusra AlSayyad added a comment - Small change in how the approximate nImage is computed which is used to compute the temporal threshold of artifact rejection. Jenkins: https://ci.lsst.codes/job/stack-os-matrix/27816/ https://ci.lsst.codes/job/stack-os-matrix/27817/ I ran the whole RC2 dataset through to ensure there were no unintended consequences: /datasets/hsc/repo/rerun/private/yusra/RC2/ DM-14075 I visually inspsected them and compared coadds bitwise with /datasets/hsc/repo/rerun/RC/w_2018_14/ DM-13890 1) All changes have been improvements 2) It doesn't only affect patches with low max number of visits because in any coadd, there will be regions (e.g.around bright stars) with local N < 3.
          Hide
          nlust Nate Lust added a comment -

          I am a bit concerned about dropping the checks for nans, though other parts of the code don't seem to check for it either. I will leave up to your best judgment to decide if it is worth checking for nans or not. I would hate for them to get propagated to the coadd if a mask plane was not set somewhere. However we may be confident that mask planes will always be set, or that we would want to pass the nans through so we could catch mask planes not being set.

          Show
          nlust Nate Lust added a comment - I am a bit concerned about dropping the checks for nans, though other parts of the code don't seem to check for it either. I will leave up to your best judgment to decide if it is worth checking for nans or not. I would hate for them to get propagated to the coadd if a mask plane was not set somewhere. However we may be confident that mask planes will always be set, or that we would want to pass the nans through so we could catch mask planes not being set.
          Hide
          yusra Yusra AlSayyad added a comment -

          This line of code has absolutely nothing to do with data that gets stacked in the coadd. No NaNs get propagated or passed to the coadds. That's the job of statisticsStack and afwMath.StatisticsControl().

          This line of code defines the approximate nImage the artifact filter uses to compute the max number of epochs an artifact can appear in before being called persistent rather than ephemeral. It is counting. The line was using the fact that chip-gaps were populated with NaNs to find the chip gaps. Note: chip gaps also have the "NO_DATA" bit set. I want to extend the definition beyond NO_DATA to include every bit plane specified in the badMaskPlanes config parameter. If we don't use the NaNs to define the zero-contribution here to the approximate nImage, it will have no effect on the stacked image.

          Weird edge case This afternoon I was thinking about scenarios where accumulating by excluding NO_DATA would yield a different approximate-nImage than excluding NO_DATA and NaN's. I could think of a scenario where someone accidentally removes NO_DATA from badMaskPlanes, which I tried... In this case: The final N-Image produced by assembleSubregion:
          https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/assembleCoadd.py#L646
          does not take into account the NaNs (the chip-gaps and edges), and over estimates the number of epochs contributing. For this weird edge case, it probably would be best to exclude NaNs from the count too.

          For the final nImage computed here: (https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/assembleCoadd.py#L646)

          LEFT (normal with badMaskPlane includes NO_DATA) RIGHT(weird, with badMaskPlane that does not include no data)

          You'll see on the right the number of visits are overestimated in the final nImage.
          Note the coadds are identical.

          Show
          yusra Yusra AlSayyad added a comment - This line of code has absolutely nothing to do with data that gets stacked in the coadd. No NaNs get propagated or passed to the coadds. That's the job of statisticsStack and afwMath.StatisticsControl() . This line of code defines the approximate nImage the artifact filter uses to compute the max number of epochs an artifact can appear in before being called persistent rather than ephemeral. It is counting. The line was using the fact that chip-gaps were populated with NaNs to find the chip gaps. Note: chip gaps also have the "NO_DATA" bit set. I want to extend the definition beyond NO_DATA to include every bit plane specified in the badMaskPlanes config parameter. If we don't use the NaNs to define the zero-contribution here to the approximate nImage, it will have no effect on the stacked image. Weird edge case This afternoon I was thinking about scenarios where accumulating by excluding NO_DATA would yield a different approximate-nImage than excluding NO_DATA and NaN's. I could think of a scenario where someone accidentally removes NO_DATA from badMaskPlanes, which I tried... In this case: The final N-Image produced by assembleSubregion: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/assembleCoadd.py#L646 does not take into account the NaNs (the chip-gaps and edges), and over estimates the number of epochs contributing. For this weird edge case, it probably would be best to exclude NaNs from the count too. For the final nImage computed here: ( https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/assembleCoadd.py#L646 ) LEFT (normal with badMaskPlane includes NO_DATA) RIGHT(weird, with badMaskPlane that does not include no data) You'll see on the right the number of visits are overestimated in the final nImage. Note the coadds are identical.

            People

            • Assignee:
              yusra Yusra AlSayyad
              Reporter:
              yusra Yusra AlSayyad
              Reviewers:
              Nate Lust
              Watchers:
              Nate Lust, Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel