Details
-
Type:
Bug
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: meas_mosaic
-
Labels:
-
Story Points:1
-
Epic Link:
-
Sprint:AP S19-5
-
Team:Alert Production
Description
makeCoaddTempExp.py with doApplyUberCal=True to use meas_mosaic outputs fail to build warps despite the input data exist. A small example to reproduce on lsst-dev is:
makeCoaddTempExp.py /datasets/hsc/repo/ --rerun RC/w_2019_14/DM-18300:private/user/name --id tract=9615 filter=HSC-G patch=4,8 --selectId ccd=0..8^10..103 visit=26050^26036^26060^26032 |
This doesn't really "fail" (and no logs in ERROR or FATAL level) but instead gives many confusing/misleading warnings and produce no outputs:
WARN 2019-04-08T01:32:59.059 makeCoaddTempExp (DataId(initialdata={'tract': 9615, 'filter': 'HSC-G', 'patch': '4,8'}, tag=set()))(makeCoaddTempExp.py:341)- Calexp DataId(initialdata={'ccd': 103, 'visit': 26032, 'pointing': 1179, 'filter': 'HSC-G', 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'taiObs': '2015-03-25', 'expTime': 150.0, 'tract': 9615}
|
, tag=set()) not found; skipping it: 'NoneType' object has no attribute'getInstFluxAtZeroMagnitude'
|
The log is misleading; this is DM-16537 about the generic exceptions. I troubleshoot a bit and trace to this line
https://github.com/lsst/meas_mosaic/blob/6e395ac11a625c877374f99e2bed771b427835b6/python/lsst/meas/mosaic/updateExposure.py#L103
where a non-empty header is obtained but no photoCalib is returned (that photCalib is None there). Feels like a bug in afw or meas_mosaic files; I paused there to file this ticket. Besides the bug, it'd be nice if the None calib is caught earlier.
Attachments
Issue Links
Activity
I think I have a fix for this. But I'm not sure how to ensure that my fix also works with old meas_mosaic files too. Do we have a way to test that?
Hsin-Fang Chiang: can you please review the one line change here and confirm that it works when making warps?
Do we have any way to test this on old repositories? I think that my fix should maintain backwards compatibility, but I'm not positive and don't know how to test it in any case.
I still think we need to have a telecon about how to deal with DM-16537.
I tried building this branch to give it a spin on the RC2 data and test_fluxFitBoundedField.py doesn't pass.
But, I can confirm that makeCoaddTempExp/coaddDriver works with your branch with both w_2019_14 AND w_2019_10 meas_mosaic outputs.
Thanks Yusra AlSayyad for testing this and also with an old repo.
John Parejko Looks fine, I only tested one case and it worked fine. Please fix the test failure and remember to run Jenkins before merging.
Hsin-Fang Chiang
added a comment - Thanks Yusra AlSayyad for testing this and also with an old repo.
John Parejko Looks fine, I only tested one case and it worked fine. Please fix the test failure and remember to run Jenkins before merging. Thanks for the note on the failed test. I swore that passed for me, but it clearly couldn't have. I've fixed the test code: can someone please give it a quick sign off?
Merged and done.
Yusra AlSayyad: I'm guessing the problem in getFluxFitParams is somehow due to meas_mosaic's fcr metadata not being readable by PhotoCalib.makePhotoCalibFromMetadata? Would you or someone in DRP be able to look into that please? I don't know anything about those files.
Separately, can we have a telecon sometime to formally deal with DM-16537? We need to fix how we handle exceptions in the coadd code (as evidenced above).