Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21704

Check whether the template scaling has a connection to the number of detected souorces

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      4
    • Epic Link:
    • Sprint:
      AP F19-5 (October), AP F19-6 (November)
    • Team:
      Alert Production

      Description

      The goal in this ticket is to check whether the high number of detection is in any connection with the signal level of the science or the template images i.e. whether any signal level or threshold is "inherited" from the original images. They are on different scales at the moment and image differencing is fitting a flux scaling as well, see DM-21700.

        Attachments

          Issue Links

          is child task of

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21702 Investigate the cause of the high number of false sources in convolveTemplate=False image differencing

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-21847 Deep coadd images contain a grid pattern in the variance plane

          • Won't Fix

          Bug - A problem which impairs or prevents the functions of the product. DM-21845 Investigate whether the wcs information of difference images are properly set

          • Invalid

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            gkovacs Gabor Kovacs [X] (Inactive) added a comment - - edited

            The development branch u/gkovacs/DM-21704_convsci_scaling contains some science/template scaling tweaking commits:

            • Git tag DM-21704noRescale: The difference exposure is _not scaled back to the science image level after subtraction in the science image convolution case. (I.e. it is fitted to the template level and remains there). The attached images are a result of running imageDifference.py at this commit.
            • Commit "Rescale template": Rescaling of the template so that its absolute median level is at the science image absolute median level. The median level belongs to the background, so matching the backgrounds is actually an overcorrection for the template due to its lower background noise.
            • Commit "Rescale template with hard wired factor of 10". For the test run this roughly levels the flux signals between the template and the science image (v411355).
            Show
            gkovacs Gabor Kovacs [X] (Inactive) added a comment - - edited The development branch u/gkovacs/ DM-21704 _convsci_scaling contains some science/template scaling tweaking commits: Git tag DM-21704 noRescale : The difference exposure is _not scaled back to the science image level after subtraction in the science image convolution case. (I.e. it is fitted to the template level and remains there). The attached images are a result of running imageDifference.py at this commit. Commit "Rescale template": Rescaling of the template so that its absolute median level is at the science image absolute median level. The median level belongs to the background, so matching the backgrounds is actually an overcorrection for the template due to its lower background noise. Commit "Rescale template with hard wired factor of 10". For the test run this roughly levels the flux signals between the template and the science image (v411355).
            Hide
            Permalink
            gkovacs Gabor Kovacs [X] (Inactive) added a comment - - edited
            • Screenshots are shown of the HiTS2015 v411035 ccd=25 differenced in science convolution mode. This was selected due to having a good overlap with one sky patch (23,10) and not being a close PSF match. (I.e. without the science convolution mode, this would strongly fall into the deconvolution regime.)
            • Note the visual "grid" pattern in the template variance. Is this a resampling side effect?
            • One set shows the calexp + diffexp, the other one the template + diffexp. (ds9 is unable to align the diffexp based on WCS, is it properly inherited from the calexp DM-21845)
            • scr_411035_25_templ_calexp_nReScale1018_focus_template_multiZscales.png show zscale ranges of the template, calexp, and calexpVar.

            Conclusions:

            • The higher the template level, the lower the number of detected sources. I am quite sure that this is due to the fact that the difference image inherits its variance mostly from the template in this case. The grid pattern was also well visible inherited in the difference image variance plane, though the detection seems to be quite insensitive to it.
            • DM-21809 With any factor up or downscaling, a MaskedImage's variance plane is scaled accordingly.
            • Source detection only considers the difference image planes on their own. The number of sources detected in this experiment is solely due to "inheriting different amount" of variance from the science and template images.
            Show
            gkovacs Gabor Kovacs [X] (Inactive) added a comment - - edited Screenshots are shown of the HiTS2015 v411035 ccd=25 differenced in science convolution mode. This was selected due to having a good overlap with one sky patch (23,10) and not being a close PSF match. (I.e. without the science convolution mode, this would strongly fall into the deconvolution regime.) Note the visual "grid" pattern in the template variance. Is this a resampling side effect? One set shows the calexp + diffexp, the other one the template + diffexp. (ds9 is unable to align the diffexp based on WCS, is it properly inherited from the calexp DM-21845 ) scr_411035_25_templ_calexp_nReScale1018_focus_template_multiZscales.png show zscale ranges of the template, calexp, and calexpVar. Conclusions: The higher the template level, the lower the number of detected sources. I am quite sure that this is due to the fact that the difference image inherits its variance mostly from the template in this case. The grid pattern was also well visible inherited in the difference image variance plane, though the detection seems to be quite insensitive to it. DM-21809 With any factor up or downscaling, a MaskedImage's variance plane is scaled accordingly. Source detection only considers the difference image planes on their own. The number of sources detected in this experiment is solely due to "inheriting different amount" of variance from the science and template images.
            Hide
            Permalink
            gkovacs Gabor Kovacs [X] (Inactive) added a comment -

            Could you please have a look at this ticket ? It's comment/attachments only and the goal is to close it perhaps at the sprint planning.

            Show
            gkovacs Gabor Kovacs [X] (Inactive) added a comment - Could you please have a look at this ticket ? It's comment/attachments only and the goal is to close it perhaps at the sprint planning.

              People

              Assignee:
              gkovacs Gabor Kovacs [X] (Inactive)
              Reporter:
              gkovacs Gabor Kovacs [X] (Inactive)
              Reviewers:
              Eric Bellm
              Watchers:
              Eric Bellm, Gabor Kovacs [X] (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.