Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13081

Examine bad-column masking in HiTS

    Details

    • Story Points:
      6
    • Epic Link:
    • Sprint:
      AP F18-6
    • Team:
      Alert Production

      Description

      The HiTS processing for the DM-503-3 test report (DM-12534) identified some unmasked bad columns. This ticket is to investigate their cause and fix the masking.

        Attachments

          Issue Links

            Activity

            Hide
            mrawls Meredith Rawls added a comment -

            I ran the script Eric Bellm pointed me to and inspected the 10 images used for AG-00-10 as reported in DMTR-53. The bad columns in both the calexps appear to be masked as both BAD and NO_DATA (red and orange by default in ds9, respectively). However, there are some un-masked features in the diffim which look kind of like bad columns. I'm not sure how to determine which coadd template was used to create each diffim, but I suspect these features may originate in the template there since they are not present in the calexp. I also noticed that the detected sources in the diffim all seem to be dipoles and many have messy ringed morphologies.

            In order to make further progress, I need to know which coadd template (patch) was used for which calexp (visit and ccd).

            Show
            mrawls Meredith Rawls added a comment - I ran the script Eric Bellm pointed me to and inspected the 10 images used for AG-00-10 as reported in DMTR-53. The bad columns in both the calexps appear to be masked as both BAD and NO_DATA (red and orange by default in ds9, respectively). However, there are some un-masked features in the diffim which look kind of like bad columns. I'm not sure how to determine which coadd template was used to create each diffim, but I suspect these features may originate in the template there since they are not present in the calexp. I also noticed that the detected sources in the diffim all seem to be dipoles and many have messy ringed morphologies. In order to make further progress, I need to know which coadd template (patch) was used for which calexp (visit and ccd).
            Hide
            mrawls Meredith Rawls added a comment -

            This ds9 screenshot shows a small portion of ccd 32 for two visits, 411673 (top) and 410915 (bottom). The left column is the calexps and the right column is the diffims. The mask color scheme is red=bad, orange=no_data (note red + orange almost always coincident make a lovely brown), yellow=suspect, green=saturated and/or interpolated, blue=detected, cyan=detected_negative. There is one example of crosstalk in the top visit (lower left) that clearly differs in appearance from the other image artifacts.

            Show
            mrawls Meredith Rawls added a comment - This ds9 screenshot shows a small portion of ccd 32 for two visits, 411673 (top) and 410915 (bottom). The left column is the calexps and the right column is the diffims. The mask color scheme is red=bad, orange=no_data (note red + orange almost always coincident make a lovely brown), yellow=suspect, green=saturated and/or interpolated, blue=detected, cyan=detected_negative. There is one example of crosstalk in the top visit (lower left) that clearly differs in appearance from the other image artifacts.
            Hide
            ebellm Eric Bellm added a comment - - edited

            As reported in DM-13085, determining the coadd from the diffim requires a new call to getCoaddAsTemplateTask; there's no lightweight way to identify the tract,patch used. 

            For the output directory at /scratch/ebellm/ap_verify_HiTS, the input coadds are at /project/ebellm/coadd_2014.

            Show
            ebellm Eric Bellm added a comment - - edited As reported in DM-13085 , determining the coadd from the diffim requires a new call to getCoaddAsTemplateTask ; there's no lightweight way to identify the tract,patch used.  For the output directory at /scratch/ebellm/ap_verify_HiTS , the input coadds are at /project/ebellm/coadd_2014 .
            Hide
            swinbank John Swinbank added a comment -

            First step in resolving this is to pull up recent data and check that it's really still a problem. Meredith Rawls will do that during November 2018.

            Show
            swinbank John Swinbank added a comment - First step in resolving this is to pull up recent data and check that it's really still a problem. Meredith Rawls will do that during November 2018.
            Hide
            mrawls Meredith Rawls added a comment -

            I've been working on this again, and made some plots of one of the original "problem CCDs" in a notebook. Within the scope of this ticket, for an ap_pipe run with the old template, it appears that bad columns are masked as BAD in both the calexps and diffims as one would expect. So, that may be sufficient to close this ticket, since I can't reproduce the issue.

            However, there is some really weird stuff happening with diffim masks with the new templates. I have been hoping that if we can just filter objects and/or sources by whether or not they are in a masked region, we can get rid of most of the spurious sources. In principle this is correct, but I think something is not working correctly in the present diffim mask assignment, because there are huge swaths of INTRP, BAD, CR, REJECTED, INEXACT_PSF, etc. Maybe this is the result of the coadd template inheriting all/most of the flags when it didn't use to? I'm not sure, but I think it's probably out of scope for this ticket.

            I'm going to put this in review, and request Eric Bellm take a look at my notebook at /project/mrawls/DM-13081-Bad-Columns-Take2.ipynb so we can work together to open new mask-related tickets as needed.

            Show
            mrawls Meredith Rawls added a comment - I've been working on this again, and made some plots of one of the original "problem CCDs" in a notebook. Within the scope of this ticket, for an ap_pipe run with the old template, it appears that bad columns are masked as BAD in both the calexps and diffims as one would expect. So, that may be sufficient to close this ticket, since I can't reproduce the issue. However, there is some really weird stuff happening with diffim masks with the new templates. I have been hoping that if we can just filter objects and/or sources by whether or not they are in a masked region, we can get rid of most of the spurious sources. In principle this is correct, but I think something is not working correctly in the present diffim mask assignment, because there are huge swaths of INTRP, BAD, CR, REJECTED, INEXACT_PSF, etc. Maybe this is the result of the coadd template inheriting all/most of the flags when it didn't use to? I'm not sure, but I think it's probably out of scope for this ticket. I'm going to put this in review, and request  Eric Bellm take a look at my notebook at /project/mrawls/ DM-13081 -Bad-Columns-Take2.ipynb so we can work together to open new mask-related tickets as needed.
            Hide
            mrawls Meredith Rawls added a comment -

            I went back and manually backed out which coadd template corresponds to a particular ccd (I used ccdnum=15 because it showed some clear bad columns in the diffims). I can now definitively say the unmasked bad columns are due to the template. (I did look at calexp/diffim combos with a few different visitIDs; the ones pictured below are visit=411673.) I am still moderately concerned about the mask footprints growing more than we would like during difference imaging, but I think the new templates are properly masked while the old ones suffered from incomplete masking from an unknown cause.

            Show
            mrawls Meredith Rawls added a comment - I went back and manually backed out which coadd template corresponds to a particular ccd (I used ccdnum=15 because it showed some clear bad columns in the diffims). I can now definitively say the unmasked bad columns are due to the template. (I did look at calexp/diffim combos with a few different visitIDs; the ones pictured below are visit=411673.) I am still moderately concerned about the mask footprints growing more than we would like during difference imaging, but I think the new templates are properly masked while the old ones suffered from incomplete masking from an unknown cause.
            Hide
            ebellm Eric Bellm added a comment -

            We have now established that the source of the unmasked bad columns in the fall 2017 processing was due to the coadds, and that coadds constructed with the current pipeline are appropriately masked. We identified two new issues that have been spun out into independent tickets (DM-16543 and DM-16544).

            Show
            ebellm Eric Bellm added a comment - We have now established that the source of the unmasked bad columns in the fall 2017 processing was due to the coadds, and that coadds constructed with the current pipeline are appropriately masked. We identified two new issues that have been spun out into independent tickets ( DM-16543 and DM-16544 ).

              People

              • Assignee:
                mrawls Meredith Rawls
                Reporter:
                ebellm Eric Bellm
                Reviewers:
                Eric Bellm
                Watchers:
                Eric Bellm, John Swinbank, Meredith Rawls
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: