Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30048

Investigate differences in gen2 vs. gen3 SFP products for DC2 data

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Epic Link:
    • Sprint:
      DRP S21b
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Differences between the gen2 and gen3 processed calexp images for DC2 data have been noted on DM-29889. Look into what the root cause is and, if possible, resolve it.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29899 DC2 Reprocessing with w_2021_16 (Gen3)

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            lauren Lauren MacArthur added a comment - - edited

            Well, this has been a "fun" rabbit hole.  I first had a look at the difference of the gen2/gen2 calexp images (as the pattern seen there usually provides a clue as to the root cause).  Here is what it looks like for visit=257768 filter="r" detector=161 (I also reran with doWrtie=True so I could get the icExp for gen2):
            icExp:

            calexp:

            Looking very "background"-related, so I went into isr and charImage stages to output backgrounds (and other stages) to look for differences...none...what???!!!  I then had a closer look at the logs and the first difference appeared in the number of CRs detected in the first round of repair, so thinking I should look into the configs I found some pretty significant differences:
            gen2:

            config.charImage.measurePsf.reserve.fraction=0.2
            		 
            config.charImage.repair.cosmicray.nCrPixelMax=1000000
            		 
            config.charImage.repair.cosmicray.cond3_fac2=0.4
            		 
            config.charImage.detection.isotropicGrow=True
            		 
            config.charImage.detection.includeThresholdMultiplier=1.0
            		 
            config.charImage.detection.thresholdType='pixel_stdev'
            		 
            config.charImage.detection.doTempLocalBackground=True

            gen3 (charImage config):

            config.measurePsf.reserve.fraction=0.0
            		 
            config.repair.cosmicray.nCrPixelMax=10000
            		 
            config.repair.cosmicray.cond3_fac2=0.6
            		 
            config.detection.isotropicGrow=False
            		 
            config.detection.includeThresholdMultiplier=10.0
            		 
            config.detection.thresholdType='stdev'
            		 
            config.detection.doTempLocalBackground=False

            there may be others...

            Running a gen3 pipeline including the gen2 config overrides above gets us much closer to parity...I may have missed a few overrides (but am not going to explore this further based on my conclusion below!): (note scales)
            icExp:

            calexp:

            Conclusion: gen3 is not picking up any of the config overrides...I'm not actually sure how this is supposed to happen, but parity is certainly not going to get anywhere close until this is resolved such that both middlewares are using the same configs!

            Show
            lauren Lauren MacArthur added a comment - - edited Well, this has been a "fun" rabbit hole.  I first had a look at the difference of the gen2/gen2  calexp  images (as the pattern seen there usually provides a clue as to the root cause).  Here is what it looks like for visit=257768 filter="r" detector=161 (I also reran with doWrtie=True so I could get the icExp for gen2): icExp: calexp: Looking very "background"-related, so I went into isr and charImage stages to output backgrounds (and other stages) to look for differences...none...what???!!!  I then had a closer look at the logs and the first difference appeared in the number of CRs detected in the first round of repair , so thinking I should look into the configs I found some pretty significant differences: gen2: config.charImage.measurePsf.reserve.fraction = 0.2 config.charImage.repair.cosmicray.nCrPixelMax = 1000000 config.charImage.repair.cosmicray.cond3_fac2 = 0.4 config.charImage.detection.isotropicGrow = True config.charImage.detection.includeThresholdMultiplier = 1.0 config.charImage.detection.thresholdType = 'pixel_stdev' config.charImage.detection.doTempLocalBackground = True gen3 (charImage config): config.measurePsf.reserve.fraction = 0.0 config.repair.cosmicray.nCrPixelMax = 10000 config.repair.cosmicray.cond3_fac2 = 0.6 config.detection.isotropicGrow = False config.detection.includeThresholdMultiplier = 10.0 config.detection.thresholdType = 'stdev' config.detection.doTempLocalBackground = False there may be others... Running a gen3 pipeline including the gen2 config overrides above gets us much closer to parity...I may have missed a few overrides (but am not going to explore this further based on my conclusion below!): (note scales) icExp: calexp: Conclusion: gen3 is not picking up any of the config overrides...I'm not actually sure how this is supposed to happen, but parity is certainly not going to get anywhere close until this is resolved such that both middlewares are using the same configs!
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            For your consideration!

            Show
            lauren Lauren MacArthur added a comment - For your consideration!
            Hide
            Permalink
            dtaranu Dan Taranu added a comment - - edited

            Thanks, Lauren. I should have thought to check the configs. Actually, it's as simple as adding the same characterizeImage.py that's in obs_subaru (https://github.com/lsst/obs_subaru/blob/master/config/characterizeImage.py) to obs_lsst. PipelineTasks look for {_DefaultName}.py and CharacterizeImageTask's _DefaultName is characterizeImage, not charImage.

            I'll file a new ticket (DM-30076) to fix that, as there might be other similar fixes that people made in obs_subaru but not obs_lsst.

            Show
            dtaranu Dan Taranu added a comment - - edited Thanks, Lauren. I should have thought to check the configs. Actually, it's as simple as adding the same characterizeImage.py that's in obs_subaru ( https://github.com/lsst/obs_subaru/blob/master/config/characterizeImage.py ) to obs_lsst. PipelineTasks look for {_DefaultName}.py and CharacterizeImageTask's _DefaultName is characterizeImage, not charImage. I'll file a new ticket ( DM-30076 ) to fix that, as there might be other similar fixes that people made in obs_subaru but not obs_lsst.
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            Sounds good, thanks.

            Show
            lauren Lauren MacArthur added a comment - Sounds good, thanks.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Dan Taranu
              Watchers:
              Dan Taranu, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.