Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28677

Avoid numerical warnings, update docstrings, add option to calculate only the factors in ScaleVarianceTask

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      2
    • Sprint:
      AP S21-3 (February)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      • Modify the application of the mask based pixel filtering in the code to avoid invalid numerical operations and numpy warnings everywhere
      • Add method to calculate and return both factors without applying to the input image
      • Update docstrings of the task to make clear that the input image is modified in-place in the default case

        Attachments

          Issue Links

            Activity

            Hide
            gkovacs Gabor Kovacs [X] (Inactive) added a comment - - edited

            Would you please review these changes ?

             

            The warning producing bug was to perform the sqrt first then filter for positive values after that.

             

            Besides, the behavior was so far that both pixelBased and imageBased returned 1. if no unmasked pixels remained (based on mask bits & variance > 0.) but if there remained a single nan unmasked in the image plane that caused the correction factor to be nan.(Both np.percentile and np.median give nan for any nan inputs.) Similarly, if there were too many inf-s that could result in the interquartile region to be inf - inf == nan.

            I expect that this was unintentional behavior and I changed it to ignore these pixels as well.

            Show
            gkovacs Gabor Kovacs [X] (Inactive) added a comment - - edited Would you please review these changes ?   The warning producing bug was to perform the sqrt first then filter for positive values after that.   Besides, the behavior was so far that both pixelBased and imageBased returned 1. if no unmasked pixels remained (based on mask bits & variance > 0.) but if there remained a single nan unmasked in the image plane that caused the correction factor to be nan. (Both np.percentile and np.median give nan for any nan inputs.) Similarly, if there were too many inf -s that could result in the interquartile region to be inf - inf == nan . I expect that this was unintentional behavior and I changed it to ignore these pixels as well.
            Hide
            yusra Yusra AlSayyad added a comment -

            Look OK, but take a look through our dev guide's guidelines for commit messages:
            https://developer.lsst.io/work/flow.html#appendix-commit-message-best-practices
            Best to split into summary line with bullets.

            I ran a couple tract/filter combos on RC2. Nice to see the warnings disappear. (Just in time to be replaced with a barrage of filter warnings). You can find the results here:
            /datasets/hsc/repo/rerun/private/yusra/RC2/DM-28677
            Max difference in variance scaling (as scraped from the logs) is -6.9e-5:
            1.2518487948031607 (1.251849 printed) vs
            1.2517795822884965 (1.251780)
            It's a little weird the scale factors printed in the logs are always larger on the ticket branch, but diff is always < 1e-4.

            Show
            yusra Yusra AlSayyad added a comment - Look OK, but take a look through our dev guide's guidelines for commit messages: https://developer.lsst.io/work/flow.html#appendix-commit-message-best-practices Best to split into summary line with bullets. I ran a couple tract/filter combos on RC2. Nice to see the warnings disappear. (Just in time to be replaced with a barrage of filter warnings). You can find the results here: /datasets/hsc/repo/rerun/private/yusra/RC2/ DM-28677 Max difference in variance scaling (as scraped from the logs) is -6.9e-5: 1.2518487948031607 (1.251849 printed) vs 1.2517795822884965 (1.251780) It's a little weird the scale factors printed in the logs are always larger on the ticket branch, but diff is always < 1e-4.
            Hide
            yusra Yusra AlSayyad added a comment -

            Forgot to mention one more thing:
            the method name `calculateBothFactors()` ties you to computing 2 and exactly 2 factors. If you named it something like `computeScaleFactors()` you could add more in the future without changing any APIs.

            Show
            yusra Yusra AlSayyad added a comment - Forgot to mention one more thing: the method name `calculateBothFactors()` ties you to computing 2 and exactly 2 factors. If you named it something like `computeScaleFactors()` you could add more in the future without changing any APIs.
            Hide
            gkovacs Gabor Kovacs [X] (Inactive) added a comment - - edited

            I could not easily find the calexp pairs in your run to test this hypothesis directly, but I assume that there were a few unmasked +inf variance values that satisfied >0 and went into the estimator.  Reciprocating these values they become additional zeroes in the middle of the distribution in the estimator. By removing them, the q3-q1 interquartile range will be a little larger; both tails will be a little shorter in its number of samples, so q1 value becomes smaller, q3 value becomes larger, q3-q1 thus will be larger.

            Show
            gkovacs Gabor Kovacs [X] (Inactive) added a comment - - edited I could not easily find the calexp pairs in your run to test this hypothesis directly, but I assume that there were a few unmasked + inf variance values that satisfied >0 and went into the estimator.  Reciprocating these values they become additional zeroes in the middle of the distribution in the estimator. By removing them, the q3-q1 interquartile range will be a little larger; both tails will be a little shorter in its number of samples, so q1 value becomes smaller, q3 value becomes larger, q3-q1 thus will be larger.

              People

              Assignee:
              gkovacs Gabor Kovacs [X] (Inactive)
              Reporter:
              gkovacs Gabor Kovacs [X] (Inactive)
              Reviewers:
              Yusra AlSayyad
              Watchers:
              Gabor Kovacs [X] (Inactive), Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.