Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11395

Update testdata_jointcal refcats to new Indexed format

    XMLWordPrintable

    Details

      Description

      Many of the reference catalogs provided with testdata_jointcal are in the old astrometry.net format, which is now deprecated. We should update those provided refcats to the new style format, which will also simplify the test configuration.

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            John Swinbank: would you be willing to review this change, since we've recently discussed it? It's not as bad as it looks: The +1000 is because I added a couple of small fits files directly to the jointcal/tests/data directory: it's more like ~200 lines of real changes.

            I'm going to do a bit more staring at the numbers tomorrow to ensure I really believe them, but it should be in fine shape to review it as it stands. The commits to jointcal should be atomic, so you can review them separately to make it easier on yourself.

            Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29372/pipeline

            Show
            Parejkoj John Parejko added a comment - John Swinbank : would you be willing to review this change, since we've recently discussed it? It's not as bad as it looks: The +1000 is because I added a couple of small fits files directly to the jointcal/tests/data directory: it's more like ~200 lines of real changes. I'm going to do a bit more staring at the numbers tomorrow to ensure I really believe them, but it should be in fine shape to review it as it stands. The commits to jointcal should be atomic, so you can review them separately to make it easier on yourself. Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29372/pipeline
            Hide
            swinbank John Swinbank added a comment - - edited

            Alarming things to see when running (successful!) tests:

            jointcal FATAL: Failed processing tract 0, RuntimeError: No images in the ccdImageList!
            root ERROR: 1 dataRefs failed; not exiting as --noExit was set
            

            However, it looks as though this is intentional. Phew.

            Show
            swinbank John Swinbank added a comment - - edited Alarming things to see when running (successful!) tests: jointcal FATAL: Failed processing tract 0, RuntimeError: No images in the ccdImageList! root ERROR: 1 dataRefs failed; not exiting as --noExit was set However, it looks as though this is intentional. Phew.
            Hide
            swinbank John Swinbank added a comment -

            Spent some time trying to understand where

             
            ====================================================================================================================================================================================================================================== warnings summary ======================================================================================================================================================================================================================================
            tests/test_jointcal_cfht_minimal.py::JointcalTestCFHTMinimal::test_jointcalTask_fails_raise
              /software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
                return f(*args, **kwds)
             
            tests/test_jointcal_cfht_minimal.py::JointcalTestCFHTMinimal::test_jointcalTask_2_visits_photometry
              /software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
                return f(*args, **kwds)
             
            tests/test_jointcal_cfht_minimal.py::JointcalTestCFHTMinimal::test_jointcalTask_fails_no_raise
              /software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
                return f(*args, **kwds)
             
            -- Docs: http://doc.pytest.org/en/latest/warnings.html
            ==================================================================================================================================================================================================================== 158 passed, 47 skipped, 3 warnings in 286.51 seconds ====================================================================================================================================================================================================================
            

            is coming from. Looks like it's actually Astropy, coming in via lsst.verify (and maybe elsewhere; that's the first entry-point I found). I guess there's nothing to be done about this for now, except hope that Astropy sort themselves out. That's half an hour of my life I won't get back...

            Otherwise, good to go here. Minor comments on the PR, but nothing that should give you much pause for thought.

            Show
            swinbank John Swinbank added a comment - Spent some time trying to understand where ====================================================================================================================================================================================================================================== warnings summary ====================================================================================================================================================================================================================================== tests/test_jointcal_cfht_minimal.py::JointcalTestCFHTMinimal::test_jointcalTask_fails_raise /software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ return f(*args, **kwds)   tests/test_jointcal_cfht_minimal.py::JointcalTestCFHTMinimal::test_jointcalTask_2_visits_photometry /software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ return f(*args, **kwds)   tests/test_jointcal_cfht_minimal.py::JointcalTestCFHTMinimal::test_jointcalTask_fails_no_raise /software/lsstsw/stack_20181012/python/miniconda3-4.5.4/envs/lsst-scipipe/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__ return f(*args, **kwds)   -- Docs: http://doc.pytest.org/en/latest/warnings.html ==================================================================================================================================================================================================================== 158 passed, 47 skipped, 3 warnings in 286.51 seconds ==================================================================================================================================================================================================================== is coming from. Looks like it's actually Astropy , coming in via lsst.verify (and maybe elsewhere; that's the first entry-point I found). I guess there's nothing to be done about this for now, except hope that Astropy sort themselves out. That's half an hour of my life I won't get back... Otherwise, good to go here. Minor comments on the PR, but nothing that should give you much pause for thought.
            Hide
            Parejkoj John Parejko added a comment -

            Per your warning search above, we've already had a ticket on that: DM-14626

            Show
            Parejkoj John Parejko added a comment - Per your warning search above, we've already had a ticket on that: DM-14626
            Hide
            Parejkoj John Parejko added a comment - - edited
            Show
            Parejkoj John Parejko added a comment - - edited New Jenkins run, now that I've updated fgcmcal (review requested from Eli Rykoff ): https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29378/pipeline
            Hide
            tjenness Tim Jenness added a comment -

            John Swinbank I've also spent time trying to sort out where those warnings come from (it's partly when I started investigating enabling fatal warnings). It's not helped that _bootstrap.py doesn't seem to be a real file. Did you learn what astropy need to fix?

            Show
            tjenness Tim Jenness added a comment - John Swinbank I've also spent time trying to sort out where those warnings come from (it's partly when I started investigating enabling fatal warnings). It's not helped that _bootstrap.py doesn't seem to be a real file. Did you learn what astropy need to fix?
            Hide
            Parejkoj John Parejko added a comment -

            Thanks for the quick review, John Swinbank.

            When investigating the jenkins failures, I discovered one decam constrained astrometry test that produced different results on mac and linux. I've disabled that test and added notes about it to DM-14439: my guess is that these decam catalogs are just pathological in some way, so I'm putting off investigation until we have a newer processing run on them in DM-17597.

            I also updated fgcmcal to use these new refcats, which resulted in a small change to the zpOffset in one of its tests. Eli Rykoff claims that's acceptable.

            Merged and done!

            Show
            Parejkoj John Parejko added a comment - Thanks for the quick review, John Swinbank . When investigating the jenkins failures, I discovered one decam constrained astrometry test that produced different results on mac and linux. I've disabled that test and added notes about it to DM-14439 : my guess is that these decam catalogs are just pathological in some way, so I'm putting off investigation until we have a newer processing run on them in DM-17597 . I also updated fgcmcal to use these new refcats, which resulted in a small change to the zpOffset in one of its tests. Eli Rykoff claims that's acceptable. Merged and done!

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              John Swinbank
              Watchers:
              John Parejko, John Swinbank, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: