# HSC warp making is broken with doApplyUberCal=True

XMLWordPrintable

## Details

• Type: Bug
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
1
• Sprint:
AP S19-5
• Team:

## Description

makeCoaddTempExp.py with doApplyUberCal=True to use meas_mosaic outputs fail to build warps despite the input data exist. A small example to reproduce on lsst-dev is:

 makeCoaddTempExp.py /datasets/hsc/repo/ --rerun RC/w_2019_14/DM-18300:private/user/name --id tract=9615 filter=HSC-G patch=4,8 --selectId ccd=0..8^10..103 visit=26050^26036^26060^26032  

This doesn't really "fail" (and no logs in ERROR or FATAL level) but instead gives many confusing/misleading warnings and produce no outputs:

 WARN 2019-04-08T01:32:59.059 makeCoaddTempExp (DataId(initialdata={'tract': 9615, 'filter': 'HSC-G', 'patch': '4,8'}, tag=set()))(makeCoaddTempExp.py:341)- Calexp DataId(initialdata={'ccd': 103, 'visit': 26032, 'pointing': 1179, 'filter': 'HSC-G', 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'taiObs': '2015-03-25', 'expTime': 150.0, 'tract': 9615} , tag=set()) not found; skipping it: 'NoneType' object has no attribute'getInstFluxAtZeroMagnitude' 

The log is misleading; this is DM-16537 about the generic exceptions. I troubleshoot a bit and trace to this line
https://github.com/lsst/meas_mosaic/blob/6e395ac11a625c877374f99e2bed771b427835b6/python/lsst/meas/mosaic/updateExposure.py#L103
where a non-empty header is obtained but no photoCalib is returned (that photCalib is None there). Feels like a bug in afw or meas_mosaic files; I paused there to file this ticket. Besides the bug, it'd be nice if the None calib is caught earlier.

## Activity

Hide
John Parejko added a comment - - edited

Yusra AlSayyad: I'm guessing the problem in getFluxFitParams is somehow due to meas_mosaic's fcr metadata not being readable by PhotoCalib.makePhotoCalibFromMetadata? Would you or someone in DRP be able to look into that please? I don't know anything about those files.

Separately, can we have a telecon sometime to formally deal with DM-16537? We need to fix how we handle exceptions in the coadd code (as evidenced above).

Show
John Parejko added a comment - - edited Yusra AlSayyad : I'm guessing the problem in getFluxFitParams is somehow due to meas_mosaic's fcr metadata not being readable by PhotoCalib.makePhotoCalibFromMetadata ? Would you or someone in DRP be able to look into that please? I don't know anything about those files. Separately, can we have a telecon sometime to formally deal with DM-16537 ? We need to fix how we handle exceptions in the coadd code (as evidenced above).
Hide
John Parejko added a comment -

I think I have a fix for this. But I'm not sure how to ensure that my fix also works with old meas_mosaic files too. Do we have a way to test that?

Show
John Parejko added a comment - I think I have a fix for this. But I'm not sure how to ensure that my fix also works with old meas_mosaic files too. Do we have a way to test that?
Hide
John Parejko added a comment -

Hsin-Fang Chiang: can you please review the one line change here and confirm that it works when making warps?

Do we have any way to test this on old repositories? I think that my fix should maintain backwards compatibility, but I'm not positive and don't know how to test it in any case.

I still think we need to have a telecon about how to deal with DM-16537.

Show
John Parejko added a comment - Hsin-Fang Chiang : can you please review the one line change here and confirm that it works when making warps? Do we have any way to test this on old repositories? I think that my fix should maintain backwards compatibility, but I'm not positive and don't know how to test it in any case. I still think we need to have a telecon about how to deal with DM-16537 .
Hide

I tried building this branch to give it a spin on the RC2 data and test_fluxFitBoundedField.py doesn't pass.

Show
Yusra AlSayyad added a comment - - edited I tried building this branch to give it a spin on the RC2 data and test_fluxFitBoundedField.py doesn't pass.
Hide

But, I can confirm that makeCoaddTempExp/coaddDriver works with your branch with both w_2019_14 AND w_2019_10 meas_mosaic outputs.

Show
Hide
Hsin-Fang Chiang added a comment -

Thanks Yusra AlSayyad for testing this and also with an old repo.

John Parejko Looks fine, I only tested one case and it worked fine. Please fix the test failure and remember to run Jenkins before merging.

Show
Hsin-Fang Chiang added a comment - Thanks Yusra AlSayyad for testing this and also with an old repo. John Parejko Looks fine, I only tested one case and it worked fine. Please fix the test failure and remember to run Jenkins before merging.
Hide
John Parejko added a comment -

Thanks for the note on the failed test. I swore that passed for me, but it clearly couldn't have. I've fixed the test code: can someone please give it a quick sign off?

Show
John Parejko added a comment - Thanks for the note on the failed test. I swore that passed for me, but it clearly couldn't have. I've fixed the test code: can someone please give it a quick sign off? Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29671/pipeline
Hide
John Parejko added a comment -

Merged and done.

Show
John Parejko added a comment - Merged and done.

## People

• Assignee:
John Parejko
Reporter:
Hsin-Fang Chiang
Reviewers:
Hsin-Fang Chiang
Watchers:
Hsin-Fang Chiang, John Parejko, John Swinbank, Yusra AlSayyad