# Investigate differences in gen2 vs. gen3 SFP products for DC2 data

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
• Story Points:
2
• Sprint:
DRP S21b
• Team:
Data Release Production
• Urgent?:
No

#### Description

Differences between the gen2 and gen3 processed calexp images for DC2 data have been noted on DM-29889. Look into what the root cause is and, if possible, resolve it.

#### Attachments

1. DC2_calexp_gen2gen3_newConfigs.png
172 kB
2. DC2_calexp_gen2gen3.png
165 kB
3. DC2_icExp_gen2gen3_newConfigs.png
207 kB
4. DC2_icExp_gen2gen3.png
168 kB

#### Activity

Hide
Lauren MacArthur added a comment - - edited

Well, this has been a "fun" rabbit hole.  I first had a look at the difference of the gen2/gen2 calexp images (as the pattern seen there usually provides a clue as to the root cause).  Here is what it looks like for visit=257768 filter="r" detector=161 (I also reran with doWrtie=True so I could get the icExp for gen2):
icExp:

calexp:

Looking very "background"-related, so I went into isr and charImage stages to output backgrounds (and other stages) to look for differences...none...what???!!!  I then had a closer look at the logs and the first difference appeared in the number of CRs detected in the first round of repair, so thinking I should look into the configs I found some pretty significant differences:
gen2:

 config.charImage.measurePsf.reserve.fraction=0.2 config.charImage.repair.cosmicray.nCrPixelMax=1000000 config.charImage.repair.cosmicray.cond3_fac2=0.4 config.charImage.detection.isotropicGrow=True config.charImage.detection.includeThresholdMultiplier=1.0 config.charImage.detection.thresholdType='pixel_stdev' config.charImage.detection.doTempLocalBackground=True 

gen3 (charImage config):

 config.measurePsf.reserve.fraction=0.0 config.repair.cosmicray.nCrPixelMax=10000 config.repair.cosmicray.cond3_fac2=0.6 config.detection.isotropicGrow=False config.detection.includeThresholdMultiplier=10.0 config.detection.thresholdType='stdev' config.detection.doTempLocalBackground=False 

there may be others...

Running a gen3 pipeline including the gen2 config overrides above gets us much closer to parity...I may have missed a few overrides (but am not going to explore this further based on my conclusion below!): (note scales)
icExp:

calexp:

Conclusion: gen3 is not picking up any of the config overrides...I'm not actually sure how this is supposed to happen, but parity is certainly not going to get anywhere close until this is resolved such that both middlewares are using the same configs!

Show
Lauren MacArthur added a comment - - edited Well, this has been a "fun" rabbit hole.  I first had a look at the difference of the gen2/gen2  calexp  images (as the pattern seen there usually provides a clue as to the root cause).  Here is what it looks like for visit=257768 filter="r" detector=161 (I also reran with doWrtie=True so I could get the icExp for gen2): icExp: calexp: Looking very "background"-related, so I went into isr and charImage stages to output backgrounds (and other stages) to look for differences...none...what???!!!  I then had a closer look at the logs and the first difference appeared in the number of CRs detected in the first round of repair , so thinking I should look into the configs I found some pretty significant differences: gen2: config.charImage.measurePsf.reserve.fraction = 0.2 config.charImage.repair.cosmicray.nCrPixelMax = 1000000 config.charImage.repair.cosmicray.cond3_fac2 = 0.4 config.charImage.detection.isotropicGrow = True config.charImage.detection.includeThresholdMultiplier = 1.0 config.charImage.detection.thresholdType = 'pixel_stdev' config.charImage.detection.doTempLocalBackground = True gen3 (charImage config): config.measurePsf.reserve.fraction = 0.0 config.repair.cosmicray.nCrPixelMax = 10000 config.repair.cosmicray.cond3_fac2 = 0.6 config.detection.isotropicGrow = False config.detection.includeThresholdMultiplier = 10.0 config.detection.thresholdType = 'stdev' config.detection.doTempLocalBackground = False there may be others... Running a gen3 pipeline including the gen2 config overrides above gets us much closer to parity...I may have missed a few overrides (but am not going to explore this further based on my conclusion below!): (note scales) icExp: calexp: Conclusion: gen3 is not picking up any of the config overrides...I'm not actually sure how this is supposed to happen, but parity is certainly not going to get anywhere close until this is resolved such that both middlewares are using the same configs!
Hide
Lauren MacArthur added a comment -

Show
Hide
Dan Taranu added a comment - - edited

Thanks, Lauren. I should have thought to check the configs. Actually, it's as simple as adding the same characterizeImage.py that's in obs_subaru (https://github.com/lsst/obs_subaru/blob/master/config/characterizeImage.py) to obs_lsst. PipelineTasks look for {_DefaultName}.py and CharacterizeImageTask's _DefaultName is characterizeImage, not charImage.

I'll file a new ticket (DM-30076) to fix that, as there might be other similar fixes that people made in obs_subaru but not obs_lsst.

Show
Dan Taranu added a comment - - edited Thanks, Lauren. I should have thought to check the configs. Actually, it's as simple as adding the same characterizeImage.py that's in obs_subaru ( https://github.com/lsst/obs_subaru/blob/master/config/characterizeImage.py ) to obs_lsst. PipelineTasks look for {_DefaultName}.py and CharacterizeImageTask's _DefaultName is characterizeImage, not charImage. I'll file a new ticket ( DM-30076 ) to fix that, as there might be other similar fixes that people made in obs_subaru but not obs_lsst.
Hide
Lauren MacArthur added a comment -

Sounds good, thanks.

Show
Lauren MacArthur added a comment - Sounds good, thanks.

#### People

Assignee:
Lauren MacArthur
Reporter:
Lauren MacArthur
Reviewers:
Dan Taranu
Watchers:
Dan Taranu, Lauren MacArthur