Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30048

Investigate differences in gen2 vs. gen3 SFP products for DC2 data

    XMLWordPrintable

Details

    • 2
    • DRP S21b
    • Data Release Production
    • No

    Description

      Differences between the gen2 and gen3 processed calexp images for DC2 data have been noted on DM-29889. Look into what the root cause is and, if possible, resolve it.

      Attachments

        Issue Links

          Activity

            lauren Lauren MacArthur added a comment - - edited

            Well, this has been a "fun" rabbit hole.  I first had a look at the difference of the gen2/gen2 calexp images (as the pattern seen there usually provides a clue as to the root cause).  Here is what it looks like for visit=257768 filter="r" detector=161 (I also reran with doWrtie=True so I could get the icExp for gen2):
            icExp:

            calexp:

            Looking very "background"-related, so I went into isr and charImage stages to output backgrounds (and other stages) to look for differences...none...what???!!!  I then had a closer look at the logs and the first difference appeared in the number of CRs detected in the first round of repair, so thinking I should look into the configs I found some pretty significant differences:
            gen2:

            config.charImage.measurePsf.reserve.fraction=0.2
            config.charImage.repair.cosmicray.nCrPixelMax=1000000
            config.charImage.repair.cosmicray.cond3_fac2=0.4
            config.charImage.detection.isotropicGrow=True
            config.charImage.detection.includeThresholdMultiplier=1.0
            config.charImage.detection.thresholdType='pixel_stdev'
            config.charImage.detection.doTempLocalBackground=True
            

            gen3 (charImage config):

            config.measurePsf.reserve.fraction=0.0
            config.repair.cosmicray.nCrPixelMax=10000
            config.repair.cosmicray.cond3_fac2=0.6
            config.detection.isotropicGrow=False
            config.detection.includeThresholdMultiplier=10.0
            config.detection.thresholdType='stdev'
            config.detection.doTempLocalBackground=False
            

            there may be others...

            Running a gen3 pipeline including the gen2 config overrides above gets us much closer to parity...I may have missed a few overrides (but am not going to explore this further based on my conclusion below!): (note scales)
            icExp:

            calexp:

            Conclusion: gen3 is not picking up any of the config overrides...I'm not actually sure how this is supposed to happen, but parity is certainly not going to get anywhere close until this is resolved such that both middlewares are using the same configs!

            lauren Lauren MacArthur added a comment - - edited Well, this has been a "fun" rabbit hole.  I first had a look at the difference of the gen2/gen2  calexp  images (as the pattern seen there usually provides a clue as to the root cause).  Here is what it looks like for visit=257768 filter="r" detector=161 (I also reran with doWrtie=True so I could get the icExp for gen2): icExp: calexp: Looking very "background"-related, so I went into isr and charImage stages to output backgrounds (and other stages) to look for differences...none...what???!!!  I then had a closer look at the logs and the first difference appeared in the number of CRs detected in the first round of repair , so thinking I should look into the configs I found some pretty significant differences: gen2: config.charImage.measurePsf.reserve.fraction = 0.2 config.charImage.repair.cosmicray.nCrPixelMax = 1000000 config.charImage.repair.cosmicray.cond3_fac2 = 0.4 config.charImage.detection.isotropicGrow = True config.charImage.detection.includeThresholdMultiplier = 1.0 config.charImage.detection.thresholdType = 'pixel_stdev' config.charImage.detection.doTempLocalBackground = True gen3 (charImage config): config.measurePsf.reserve.fraction = 0.0 config.repair.cosmicray.nCrPixelMax = 10000 config.repair.cosmicray.cond3_fac2 = 0.6 config.detection.isotropicGrow = False config.detection.includeThresholdMultiplier = 10.0 config.detection.thresholdType = 'stdev' config.detection.doTempLocalBackground = False there may be others... Running a gen3 pipeline including the gen2 config overrides above gets us much closer to parity...I may have missed a few overrides (but am not going to explore this further based on my conclusion below!): (note scales) icExp: calexp: Conclusion: gen3 is not picking up any of the config overrides...I'm not actually sure how this is supposed to happen, but parity is certainly not going to get anywhere close until this is resolved such that both middlewares are using the same configs!

            For your consideration!

            lauren Lauren MacArthur added a comment - For your consideration!
            dtaranu Dan Taranu added a comment - - edited

            Thanks, Lauren. I should have thought to check the configs. Actually, it's as simple as adding the same characterizeImage.py that's in obs_subaru (https://github.com/lsst/obs_subaru/blob/master/config/characterizeImage.py) to obs_lsst. PipelineTasks look for {_DefaultName}.py and CharacterizeImageTask's _DefaultName is characterizeImage, not charImage.

            I'll file a new ticket (DM-30076) to fix that, as there might be other similar fixes that people made in obs_subaru but not obs_lsst.

            dtaranu Dan Taranu added a comment - - edited Thanks, Lauren. I should have thought to check the configs. Actually, it's as simple as adding the same characterizeImage.py that's in obs_subaru ( https://github.com/lsst/obs_subaru/blob/master/config/characterizeImage.py ) to obs_lsst. PipelineTasks look for {_DefaultName}.py and CharacterizeImageTask's _DefaultName is characterizeImage, not charImage. I'll file a new ticket ( DM-30076 ) to fix that, as there might be other similar fixes that people made in obs_subaru but not obs_lsst.

            Sounds good, thanks.

            lauren Lauren MacArthur added a comment - Sounds good, thanks.

            People

              lauren Lauren MacArthur
              lauren Lauren MacArthur
              Dan Taranu
              Dan Taranu, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.