Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-532

Make CompareWarp the Default Coaddition Algorithm

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      We developed an artifact rejection algorithm that compares the time series of PSF-matched Warps to find artifacts like asteroids, cosmic rays, ghosts and satellite trails, and mask them before coadding. It was used in production in the most recent HSC data release (S18A), and it is the default in the HSC biweekly processing and ci_hsc. I'm requesting comments on making it the default algorithm stack-wide.

      Details about its performance compared with SafeClip, the current default, are presented in https://dmtn-080.lsst.io.

      This algorithm depends on the availability of PSF-matched Warps. Because of the variance in cameras' pixel scales and seeing at telescope sites, no PSF-matching configs will be perfect for all cameras.  The config parameters that we adapted for obs_subaru are the target model FWHM for the PSF-matched Warps and the width of the basis functions for the matching kernel. Both were increased for obs_subaru:

      config.warpAndPsfMatch.psfMatch.kernel['AL'].alardSigGauss = [1.0, 2.0, 4.5]
      config.modelPsf.defaultFwhm = 7.7
      

      Before making CompareWarp default, I would like to take a look at least decam data and DC2 data to see if we need to update the defaults in order to produce sufficient results for most camera/surveys, but defaultFwhm is survey dependent (it should be set to something close to the maximum seeing in the dataset being processed).  In the future, we may be able to programmatically extract the best target FWHM from the processed single frame visits. 

        Attachments

          Issue Links

            Activity

            Hide
            zivezic Zeljko Ivezic added a comment -

            I am curious to know if the pipeline produces any metadata about the rejects,

            that could be later analyzed (counts, is there persistence in pixel coordinates, etc). 

            Show
            zivezic Zeljko Ivezic added a comment - I am curious to know if the pipeline produces any metadata about the rejects, that could be later analyzed (counts, is there persistence in pixel coordinates, etc). 
            Hide
            erykoff Eli Rykoff added a comment -

            Can't the defaults be set specifically in each obs package?  Is this a problem?

            Show
            erykoff Eli Rykoff added a comment - Can't the defaults be set specifically in each obs package?  Is this a problem?
            Hide
            yusra Yusra AlSayyad added a comment -

            Zeljko Ivezic, Whether the pipeline produces and saves the count images of the coadds,  https://github.com/lsst/obs_base/blob/master/policy/exposures.yaml#L8 is controlled by the config parameter doNImage. This is False by default. Both obs_subaru and obs_lsstCam turn this on. Given that the data product takes minimal space on disk, compared to the additional PSF-matched Warps, I support setting doNImage=True by default as well.

            Show
            yusra Yusra AlSayyad added a comment - Zeljko Ivezic , Whether the pipeline produces and saves the count images of the coadds,   https://github.com/lsst/obs_base/blob/master/policy/exposures.yaml#L8 is controlled by the config parameter doNImage . This is False by default. Both obs_subaru and obs_lsstCam turn this on. Given that the data product takes minimal space on disk, compared to the additional PSF-matched Warps, I support setting doNImage=True by default as well.
            Hide
            yusra Yusra AlSayyad added a comment -

            Eli Rykoff, Yes, and they will be. 

            I'm saying that I will do the due diligence to prevent the scenario where most of the obs_packages have identical overrides.  But I can't look at future obs_packages or even test PSF-matching configs for all existing data, so this serves as a heads up that when you are writing a new obs_package, the target model PSF will be something you might have to think about when you're making coadds. 

            Show
            yusra Yusra AlSayyad added a comment - Eli Rykoff , Yes, and they will be.  I'm saying that I will do the due diligence to prevent the scenario where most of the obs_packages have identical overrides.  But I can't look at future obs_packages or even test PSF-matching configs for all existing data, so this serves as a heads up that when you are writing a new obs_package, the target model PSF will be something you might have to think about when you're making coadds. 
            Hide
            tjenness Tim Jenness added a comment -

            Yusra AlSayyad are you happy to adopt this RFC?

            Show
            tjenness Tim Jenness added a comment - Yusra AlSayyad are you happy to adopt this RFC?
            Hide
            yusra Yusra AlSayyad added a comment -

            I am. See implementation ticket DM-16042 for future updates.

            Show
            yusra Yusra AlSayyad added a comment - I am. See implementation ticket DM-16042 for future updates.
            Hide
            yusra Yusra AlSayyad added a comment -

            This was quietly implemented in the Gen3 default DRP.yaml pipeline. 

            Show
            yusra Yusra AlSayyad added a comment - This was quietly implemented in the Gen3 default DRP.yaml pipeline. 

              People

              Assignee:
              yusra Yusra AlSayyad
              Reporter:
              yusra Yusra AlSayyad
              Watchers:
              Eli Rykoff, John Swinbank, Tim Jenness, Yusra AlSayyad, Zeljko Ivezic
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.