# Update testdata_jointcal refcats to new Indexed format

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
2
• Sprint:
AP S19-2
• Team:

#### Description

Many of the reference catalogs provided with testdata_jointcal are in the old astrometry.net format, which is now deprecated. We should update those provided refcats to the new style format, which will also simplify the test configuration.

#### Activity

Hide
John Parejko added a comment -

John Swinbank: would you be willing to review this change, since we've recently discussed it? It's not as bad as it looks: The +1000 is because I added a couple of small fits files directly to the jointcal/tests/data directory: it's more like ~200 lines of real changes.

I'm going to do a bit more staring at the numbers tomorrow to ensure I really believe them, but it should be in fine shape to review it as it stands. The commits to jointcal should be atomic, so you can review them separately to make it easier on yourself.

Show
John Parejko added a comment - John Swinbank : would you be willing to review this change, since we've recently discussed it? It's not as bad as it looks: The +1000 is because I added a couple of small fits files directly to the jointcal/tests/data directory: it's more like ~200 lines of real changes. I'm going to do a bit more staring at the numbers tomorrow to ensure I really believe them, but it should be in fine shape to review it as it stands. The commits to jointcal should be atomic, so you can review them separately to make it easier on yourself. Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29372/pipeline
Hide
John Swinbank added a comment - - edited

Alarming things to see when running (successful!) tests:

 jointcal FATAL: Failed processing tract 0, RuntimeError: No images in the ccdImageList! root ERROR: 1 dataRefs failed; not exiting as --noExit was set 

However, it looks as though this is intentional. Phew.

Show
John Swinbank added a comment - - edited Alarming things to see when running (successful!) tests: jointcal FATAL: Failed processing tract 0, RuntimeError: No images in the ccdImageList! root ERROR: 1 dataRefs failed; not exiting as --noExit was set However, it looks as though this is intentional. Phew.
Hide
John Swinbank added a comment -

Spent some time trying to understand where

   ====================================================================================================================================================================================================================================== warnings summary ====================================================================================================================================================================================================================================== tests/test_jointcal_cfht_minimal.py::JointcalTestCFHTMinimal::test_jointcalTask_fails_raise  /software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__  return f(*args, **kwds)   tests/test_jointcal_cfht_minimal.py::JointcalTestCFHTMinimal::test_jointcalTask_2_visits_photometry  /software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__  return f(*args, **kwds)   tests/test_jointcal_cfht_minimal.py::JointcalTestCFHTMinimal::test_jointcalTask_fails_no_raise  /software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__  return f(*args, **kwds)   -- Docs: http://doc.pytest.org/en/latest/warnings.html ==================================================================================================================================================================================================================== 158 passed, 47 skipped, 3 warnings in 286.51 seconds ==================================================================================================================================================================================================================== 

is coming from. Looks like it's actually Astropy, coming in via lsst.verify (and maybe elsewhere; that's the first entry-point I found). I guess there's nothing to be done about this for now, except hope that Astropy sort themselves out. That's half an hour of my life I won't get back...

Otherwise, good to go here. Minor comments on the PR, but nothing that should give you much pause for thought.

Show
John Swinbank added a comment - Spent some time trying to understand where ====================================================================================================================================================================================================================================== warnings summary ====================================================================================================================================================================================================================================== tests/test_jointcal_cfht_minimal.py::JointcalTestCFHTMinimal::test_jointcalTask_fails_raise /software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ return f(*args, **kwds)   tests/test_jointcal_cfht_minimal.py::JointcalTestCFHTMinimal::test_jointcalTask_2_visits_photometry /software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ return f(*args, **kwds)   tests/test_jointcal_cfht_minimal.py::JointcalTestCFHTMinimal::test_jointcalTask_fails_no_raise /software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ return f(*args, **kwds)   -- Docs: http://doc.pytest.org/en/latest/warnings.html ==================================================================================================================================================================================================================== 158 passed, 47 skipped, 3 warnings in 286.51 seconds ==================================================================================================================================================================================================================== is coming from. Looks like it's actually Astropy , coming in via lsst.verify (and maybe elsewhere; that's the first entry-point I found). I guess there's nothing to be done about this for now, except hope that Astropy sort themselves out. That's half an hour of my life I won't get back... Otherwise, good to go here. Minor comments on the PR, but nothing that should give you much pause for thought.
Hide
John Parejko added a comment -

Show
Hide
John Parejko added a comment - - edited
Show
John Parejko added a comment - - edited New Jenkins run, now that I've updated fgcmcal (review requested from Eli Rykoff ): https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29378/pipeline
Hide
Tim Jenness added a comment -

John Swinbank I've also spent time trying to sort out where those warnings come from (it's partly when I started investigating enabling fatal warnings). It's not helped that _bootstrap.py doesn't seem to be a real file. Did you learn what astropy need to fix?

Show
Tim Jenness added a comment - John Swinbank I've also spent time trying to sort out where those warnings come from (it's partly when I started investigating enabling fatal warnings). It's not helped that _bootstrap.py doesn't seem to be a real file. Did you learn what astropy need to fix?
Hide
John Parejko added a comment -

Thanks for the quick review, John Swinbank.

When investigating the jenkins failures, I discovered one decam constrained astrometry test that produced different results on mac and linux. I've disabled that test and added notes about it to DM-14439: my guess is that these decam catalogs are just pathological in some way, so I'm putting off investigation until we have a newer processing run on them in DM-17597.

I also updated fgcmcal to use these new refcats, which resulted in a small change to the zpOffset in one of its tests. Eli Rykoff claims that's acceptable.

Merged and done!

Show
John Parejko added a comment - Thanks for the quick review, John Swinbank . When investigating the jenkins failures, I discovered one decam constrained astrometry test that produced different results on mac and linux. I've disabled that test and added notes about it to DM-14439 : my guess is that these decam catalogs are just pathological in some way, so I'm putting off investigation until we have a newer processing run on them in DM-17597 . I also updated fgcmcal to use these new refcats, which resulted in a small change to the zpOffset in one of its tests. Eli Rykoff claims that's acceptable. Merged and done!

#### People

Assignee:
John Parejko
Reporter:
John Parejko
Reviewers:
John Swinbank
Watchers:
John Parejko, John Swinbank, Simon Krughoff, Tim Jenness