Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17843

A new set of processCcd failure in HSC-RC2 reprocessing

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Comparing HSC-RC2 reprocessing by stack w_2019_02 and w_2019_06, we've got a new set of processCcd errors. They fail at calibrate.astrometry.matcher with

        File "/software/lsstsw/stack_20181012/stack/miniconda3-4.5.4-fcd27eb/Linux64/meas_astrom/16.0-24-gfa57b64+1/python/lsst/meas/astrom/matchPessimisticB.py", line 278, in matchObjectsToSources
          raise RuntimeError("Unable to match sources")
      

      The 60 data ids of the new failure set is as follows

      --id visit=34338 ccd=75
      --id visit=34362 ccd=38
      --id visit=34382 ccd=43
      --id visit=34384 ccd=88
      --id visit=34402 ccd=61
      --id visit=34422 ccd=42
      --id visit=34424 ccd=2
      --id visit=34448 ccd=79
      --id visit=34450 ccd=25
      --id visit=34478 ccd=21
      --id visit=34482 ccd=50
      --id visit=34644 ccd=72
      --id visit=34690 ccd=39
      --id visit=36140 ccd=30
      --id visit=36170 ccd=18
      --id visit=36182 ccd=80
      --id visit=36212 ccd=34
      --id visit=36234 ccd=82
      --id visit=36240 ccd=0
      --id visit=36258 ccd=73
      --id visit=36428 ccd=61
      --id visit=36432 ccd=23
      --id visit=36492 ccd=50
      --id visit=34942 ccd=44
      --id visit=36762 ccd=18
      --id visit=36774 ccd=24
      --id visit=36792 ccd=34
      --id visit=36808 ccd=26
      --id visit=36828 ccd=16
      --id visit=36828 ccd=73
      --id visit=26046 ccd=32
      --id visit=26050 ccd=74
      --id visit=26058 ccd=34
      --id visit=26060 ccd=70
      -id visit=26072 ccd=12
      --id visit=26080 ccd=66
      --id visit=26084 ccd=64
      --id visit=23864 ccd=12
      --id visit=1308 ccd=40
      --id visit=23224 ccd=59
      --id visit=23232 ccd=32
      --id visit=27106 ccd=33
      --id visit=27106 ccd=41
      --id visit=27128 ccd=26
      --id visit=27134 ccd=28
      --id visit=440 ccd=92
      --id visit=452 ccd=35
      --id visit=452 ccd=42
      --id visit=472 ccd=30
      --id visit=1246 ccd=19
      --id visit=19696 ccd=20
      --id visit=30500 ccd=20
      --id visit=1180 ccd=89
      --id visit=1184 ccd=26
      --id visit=1194 ccd=26
      --id visit=360 ccd=26
      --id visit=22628 ccd=18
      --id visit=22664 ccd=26
      --id visit=23046 ccd=26
      --id visit=23050 ccd=84
      

        Attachments

          Issue Links

            Activity

            No builds found.
            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Link This issue relates to DM-17400 [ DM-17400 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Summary A new set of processCcd errors in HSC-RC2 reprocessing A new set of processCcd failure in HSC-RC2 reprocessing
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The change looks expected by DM-14857 (Switch the default matcher to PessimisticB) which merged before w_2019_04. I'm not sure if further action is necessary; Yusra AlSayyad please close this if you think it's fine to close.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The change looks expected by DM-14857 (Switch the default matcher to PessimisticB) which merged before w_2019_04. I'm not sure if further action is necessary; Yusra AlSayyad please close this if you think it's fine to close.
            Hide
            swinbank John Swinbank added a comment -

            Regressions in the SQuaSH metrics are expected, and should be addressed by DM-17552 — we should, of course, confirm they really are fixed when that closes.

            It's not written here, but I understand from Slack that we have jumped from 9 CCDs failing to 60 failing with the new matcher. That tells me that this issue is not fine to close without further investigation.

            Chris Morrison [X], any thoughts about what's happening here?

            Show
            swinbank John Swinbank added a comment - Regressions in the SQuaSH metrics are expected, and should be addressed by DM-17552 — we should, of course, confirm they really are fixed when that closes. It's not written here, but I understand from Slack that we have jumped from 9 CCDs failing to 60 failing with the new matcher. That tells me that this issue is not fine to close without further investigation. Chris Morrison [X] , any thoughts about what's happening here?
            swinbank John Swinbank made changes -
            Watchers Hsin-Fang Chiang, John Swinbank [ Hsin-Fang Chiang, John Swinbank ] Chris Morrison, Hsin-Fang Chiang, John Swinbank [ Chris Morrison, Hsin-Fang Chiang, John Swinbank ]
            Hide
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            The regression is already fixed in DM-17552.

            As for what is happening with the new set of failures, I wouldn't know without looking. My guess is some of the default obs_subaru/HSC settings that are hard coded and were designed for the old matcher don't apply anymore and should be tweaked. What reference catalog is being used?

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - The regression is already fixed in DM-17552 . As for what is happening with the new set of failures, I wouldn't know without looking. My guess is some of the default obs_subaru/HSC settings that are hard coded and were designed for the old matcher don't apply anymore and should be tweaked. What reference catalog is being used?
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            From the config file

            config.processCcd.charImage.refObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110'
            config.processCcd.calibrate.astromRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110'
            config.processCcd.calibrate.photoRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110'
            config.processCcd.calibrate.photoCal.photoCatName='ps1_pv3_3pi_20170110'
            

            Show
            hchiang2 Hsin-Fang Chiang added a comment - From the config file config.processCcd.charImage.refObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110' config.processCcd.calibrate.astromRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110' config.processCcd.calibrate.photoRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110' config.processCcd.calibrate.photoCal.photoCatName='ps1_pv3_3pi_20170110'
            Hide
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            Okay, my other wonder is if these failures are happening more in one given band than another. For instance, are all the failures in Y or Z say where fewer source detentions are available?

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - Okay, my other wonder is if these failures are happening more in one given band than another. For instance, are all the failures in Y or Z say where fewer source detentions are available?
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            It happens in non-Y/Z bands too. Their full data IDs are:

            dataId={'visit': 1246, 'ccd': 19, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-I', 'taiObs': '2014-03-28', 'expTime': 270.0}
            dataId={'visit': 19696, 'ccd': 20, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-01-21', 'pointing': 1116, 'filter': 'HSC-I', 'taiObs': '2015-01-21', 'expTime': 300.0}
            dataId={'visit': 30500, 'ccd': 20, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-05-21', 'pointing': 1236, 'filter': 'HSC-I', 'taiObs': '2015-05-21', 'expTime': 300.0}
            dataId={'visit': 23046, 'ccd': 26, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-03-16', 'pointing': 1170, 'filter': 'NB0921', 'taiObs': '2015-03-16', 'expTime': 900.0}
            dataId={'visit': 23050, 'ccd': 84, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-03-16', 'pointing': 1170, 'filter': 'NB0921', 'taiObs': '2015-03-16', 'expTime': 900.0}
            dataId={'visit': 360, 'ccd': 26, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-25', 'pointing': 814, 'filter': 'HSC-Y', 'taiObs': '2014-03-25', 'expTime': 240.0}
            dataId={'visit': 22628, 'ccd': 18, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-01-27', 'pointing': 1122, 'filter': 'HSC-Y', 'taiObs': '2015-01-27', 'expTime': 300.0}
            dataId={'visit': 22664, 'ccd': 26, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-01-27', 'pointing': 1122, 'filter': 'HSC-Y', 'taiObs': '2015-01-27', 'expTime': 300.0}
            dataId={'visit': 1180, 'ccd': 89, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-Z', 'taiObs': '2014-03-28', 'expTime': 270.0}
            dataId={'visit': 1184, 'ccd': 26, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-Z', 'taiObs': '2014-03-28', 'expTime': 270.0}
            dataId={'visit': 1194, 'ccd': 26, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-Z', 'taiObs': '2014-03-28', 'expTime': 270.0}
            dataId={'visit': 26046, 'ccd': 32, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0}
            dataId={'visit': 26050, 'ccd': 74, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0}
            dataId={'visit': 26058, 'ccd': 34, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0}
            dataId={'visit': 26060, 'ccd': 70, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0}
            dataId={'visit': 26072, 'ccd': 12, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0}
            dataId={'visit': 26080, 'ccd': 66, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0}
            dataId={'visit': 26084, 'ccd': 64, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0}
            dataId={'visit': 1308, 'ccd': 40, 'field': 'SSP_WIDE', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-I', 'taiObs': '2014-03-28', 'expTime': 200.0}
            dataId={'visit': 23864, 'ccd': 12, 'field': 'SSP_WIDE', 'dateObs': '2015-03-18', 'pointing': 1172, 'filter': 'HSC-R', 'taiObs': '2015-03-18', 'expTime': 150.0}
            dataId={'visit': 440, 'ccd': 92, 'field': 'SSP_WIDE', 'dateObs': '2014-03-25', 'pointing': 814, 'filter': 'HSC-Y', 'taiObs': '2014-03-25', 'expTime': 200.0}
            dataId={'visit': 452, 'ccd': 35, 'field': 'SSP_WIDE', 'dateObs': '2014-03-25', 'pointing': 814, 'filter': 'HSC-Y', 'taiObs': '2014-03-25', 'expTime': 200.0}
            dataId={'visit': 452, 'ccd': 42, 'field': 'SSP_WIDE', 'dateObs': '2014-03-25', 'pointing': 814, 'filter': 'HSC-Y', 'taiObs': '2014-03-25', 'expTime': 200.0}
            dataId={'visit': 472, 'ccd': 30, 'field': 'SSP_WIDE', 'dateObs': '2014-03-25', 'pointing': 814, 'filter': 'HSC-Y', 'taiObs': '2014-03-25', 'expTime': 200.0}
            dataId={'visit': 23224, 'ccd': 59, 'field': 'SSP_WIDE', 'dateObs': '2015-03-16', 'pointing': 1170, 'filter': 'HSC-Z', 'taiObs': '2015-03-16', 'expTime': 200.0}
            dataId={'visit': 23232, 'ccd': 32, 'field': 'SSP_WIDE', 'dateObs': '2015-03-16', 'pointing': 1170, 'filter': 'HSC-Z', 'taiObs': '2015-03-16', 'expTime': 200.0}
            dataId={'visit': 27106, 'ccd': 33, 'field': 'SSP_WIDE', 'dateObs': '2015-03-29', 'pointing': 1183, 'filter': 'HSC-Z', 'taiObs': '2015-03-29', 'expTime': 200.0}
            dataId={'visit': 27106, 'ccd': 41, 'field': 'SSP_WIDE', 'dateObs': '2015-03-29', 'pointing': 1183, 'filter': 'HSC-Z', 'taiObs': '2015-03-29', 'expTime': 200.0}
            dataId={'visit': 27128, 'ccd': 26, 'field': 'SSP_WIDE', 'dateObs': '2015-03-29', 'pointing': 1183, 'filter': 'HSC-Z', 'taiObs': '2015-03-29', 'expTime': 200.0}
            dataId={'visit': 27134, 'ccd': 28, 'field': 'SSP_WIDE', 'dateObs': '2015-03-29', 'pointing': 1183, 'filter': 'HSC-Z', 'taiObs': '2015-03-29', 'expTime': 200.0}
            dataId={'visit': 34338, 'ccd': 75, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0}
            dataId={'visit': 34362, 'ccd': 38, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0}
            dataId={'visit': 34382, 'ccd': 43, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0}
            dataId={'visit': 34384, 'ccd': 88, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0}
            dataId={'visit': 34402, 'ccd': 61, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0}
            dataId={'visit': 34422, 'ccd': 42, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0}
            dataId={'visit': 34424, 'ccd': 2, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0}
            dataId={'visit': 34448, 'ccd': 79, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0}
            dataId={'visit': 34450, 'ccd': 25, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0}
            dataId={'visit': 34478, 'ccd': 21, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0}
            dataId={'visit': 34482, 'ccd': 50, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0}
            dataId={'visit': 36140, 'ccd': 30, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0}
            dataId={'visit': 36170, 'ccd': 18, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0}
            dataId={'visit': 36182, 'ccd': 80, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0}
            dataId={'visit': 36212, 'ccd': 34, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0}
            dataId={'visit': 36234, 'ccd': 82, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0}
            dataId={'visit': 36240, 'ccd': 0, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0}
            dataId={'visit': 36258, 'ccd': 73, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0}
            dataId={'visit': 34644, 'ccd': 72, 'field': 'SSP_WIDE', 'dateObs': '2015-07-15', 'pointing': 1291, 'filter': 'HSC-R', 'taiObs': '2015-07-15', 'expTime': 150.0}
            dataId={'visit': 34690, 'ccd': 39, 'field': 'SSP_WIDE', 'dateObs': '2015-07-15', 'pointing': 1291, 'filter': 'HSC-R', 'taiObs': '2015-07-15', 'expTime': 150.0}
            dataId={'visit': 34942, 'ccd': 44, 'field': 'SSP_WIDE', 'dateObs': '2015-07-16', 'pointing': 1292, 'filter': 'HSC-Y', 'taiObs': '2015-07-16', 'expTime': 200.0}
            dataId={'visit': 36762, 'ccd': 18, 'field': 'SSP_WIDE', 'dateObs': '2015-07-23', 'pointing': 1299, 'filter': 'HSC-Y', 'taiObs': '2015-07-23', 'expTime': 200.0}
            dataId={'visit': 36774, 'ccd': 24, 'field': 'SSP_WIDE', 'dateObs': '2015-07-23', 'pointing': 1299, 'filter': 'HSC-Y', 'taiObs': '2015-07-23', 'expTime': 200.0}
            dataId={'visit': 36792, 'ccd': 34, 'field': 'SSP_WIDE', 'dateObs': '2015-07-23', 'pointing': 1299, 'filter': 'HSC-Y', 'taiObs': '2015-07-23', 'expTime': 200.0}
            dataId={'visit': 36808, 'ccd': 26, 'field': 'SSP_WIDE', 'dateObs': '2015-07-23', 'pointing': 1299, 'filter': 'HSC-Y', 'taiObs': '2015-07-23', 'expTime': 200.0}
            dataId={'visit': 36828, 'ccd': 16, 'field': 'SSP_WIDE', 'dateObs': '2015-07-23', 'pointing': 1299, 'filter': 'HSC-Y', 'taiObs': '2015-07-23', 'expTime': 200.0}
            dataId={'visit': 36828, 'ccd': 73, 'field': 'SSP_WIDE', 'dateObs': '2015-07-23', 'pointing': 1299, 'filter': 'HSC-Y', 'taiObs': '2015-07-23', 'expTime': 200.0}
            dataId={'visit': 36428, 'ccd': 61, 'field': 'SSP_WIDE', 'dateObs': '2015-07-22', 'pointing': 1298, 'filter': 'HSC-Z', 'taiObs': '2015-07-22', 'expTime': 200.0}
            dataId={'visit': 36432, 'ccd': 23, 'field': 'SSP_WIDE', 'dateObs': '2015-07-22', 'pointing': 1298, 'filter': 'HSC-Z', 'taiObs': '2015-07-22', 'expTime': 200.0}
            dataId={'visit': 36492, 'ccd': 50, 'field': 'SSP_WIDE', 'dateObs': '2015-07-22', 'pointing': 1298, 'filter': 'HSC-Z', 'taiObs': '2015-07-22', 'expTime': 200.0}
            

            Show
            hchiang2 Hsin-Fang Chiang added a comment - It happens in non-Y/Z bands too. Their full data IDs are: dataId={'visit': 1246, 'ccd': 19, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-I', 'taiObs': '2014-03-28', 'expTime': 270.0} dataId={'visit': 19696, 'ccd': 20, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-01-21', 'pointing': 1116, 'filter': 'HSC-I', 'taiObs': '2015-01-21', 'expTime': 300.0} dataId={'visit': 30500, 'ccd': 20, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-05-21', 'pointing': 1236, 'filter': 'HSC-I', 'taiObs': '2015-05-21', 'expTime': 300.0} dataId={'visit': 23046, 'ccd': 26, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-03-16', 'pointing': 1170, 'filter': 'NB0921', 'taiObs': '2015-03-16', 'expTime': 900.0} dataId={'visit': 23050, 'ccd': 84, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-03-16', 'pointing': 1170, 'filter': 'NB0921', 'taiObs': '2015-03-16', 'expTime': 900.0} dataId={'visit': 360, 'ccd': 26, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-25', 'pointing': 814, 'filter': 'HSC-Y', 'taiObs': '2014-03-25', 'expTime': 240.0} dataId={'visit': 22628, 'ccd': 18, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-01-27', 'pointing': 1122, 'filter': 'HSC-Y', 'taiObs': '2015-01-27', 'expTime': 300.0} dataId={'visit': 22664, 'ccd': 26, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-01-27', 'pointing': 1122, 'filter': 'HSC-Y', 'taiObs': '2015-01-27', 'expTime': 300.0} dataId={'visit': 1180, 'ccd': 89, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-Z', 'taiObs': '2014-03-28', 'expTime': 270.0} dataId={'visit': 1184, 'ccd': 26, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-Z', 'taiObs': '2014-03-28', 'expTime': 270.0} dataId={'visit': 1194, 'ccd': 26, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-Z', 'taiObs': '2014-03-28', 'expTime': 270.0} dataId={'visit': 26046, 'ccd': 32, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0} dataId={'visit': 26050, 'ccd': 74, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0} dataId={'visit': 26058, 'ccd': 34, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0} dataId={'visit': 26060, 'ccd': 70, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0} dataId={'visit': 26072, 'ccd': 12, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0} dataId={'visit': 26080, 'ccd': 66, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0} dataId={'visit': 26084, 'ccd': 64, 'field': 'SSP_WIDE', 'dateObs': '2015-03-25', 'pointing': 1179, 'filter': 'HSC-G', 'taiObs': '2015-03-25', 'expTime': 150.0} dataId={'visit': 1308, 'ccd': 40, 'field': 'SSP_WIDE', 'dateObs': '2014-03-28', 'pointing': 817, 'filter': 'HSC-I', 'taiObs': '2014-03-28', 'expTime': 200.0} dataId={'visit': 23864, 'ccd': 12, 'field': 'SSP_WIDE', 'dateObs': '2015-03-18', 'pointing': 1172, 'filter': 'HSC-R', 'taiObs': '2015-03-18', 'expTime': 150.0} dataId={'visit': 440, 'ccd': 92, 'field': 'SSP_WIDE', 'dateObs': '2014-03-25', 'pointing': 814, 'filter': 'HSC-Y', 'taiObs': '2014-03-25', 'expTime': 200.0} dataId={'visit': 452, 'ccd': 35, 'field': 'SSP_WIDE', 'dateObs': '2014-03-25', 'pointing': 814, 'filter': 'HSC-Y', 'taiObs': '2014-03-25', 'expTime': 200.0} dataId={'visit': 452, 'ccd': 42, 'field': 'SSP_WIDE', 'dateObs': '2014-03-25', 'pointing': 814, 'filter': 'HSC-Y', 'taiObs': '2014-03-25', 'expTime': 200.0} dataId={'visit': 472, 'ccd': 30, 'field': 'SSP_WIDE', 'dateObs': '2014-03-25', 'pointing': 814, 'filter': 'HSC-Y', 'taiObs': '2014-03-25', 'expTime': 200.0} dataId={'visit': 23224, 'ccd': 59, 'field': 'SSP_WIDE', 'dateObs': '2015-03-16', 'pointing': 1170, 'filter': 'HSC-Z', 'taiObs': '2015-03-16', 'expTime': 200.0} dataId={'visit': 23232, 'ccd': 32, 'field': 'SSP_WIDE', 'dateObs': '2015-03-16', 'pointing': 1170, 'filter': 'HSC-Z', 'taiObs': '2015-03-16', 'expTime': 200.0} dataId={'visit': 27106, 'ccd': 33, 'field': 'SSP_WIDE', 'dateObs': '2015-03-29', 'pointing': 1183, 'filter': 'HSC-Z', 'taiObs': '2015-03-29', 'expTime': 200.0} dataId={'visit': 27106, 'ccd': 41, 'field': 'SSP_WIDE', 'dateObs': '2015-03-29', 'pointing': 1183, 'filter': 'HSC-Z', 'taiObs': '2015-03-29', 'expTime': 200.0} dataId={'visit': 27128, 'ccd': 26, 'field': 'SSP_WIDE', 'dateObs': '2015-03-29', 'pointing': 1183, 'filter': 'HSC-Z', 'taiObs': '2015-03-29', 'expTime': 200.0} dataId={'visit': 27134, 'ccd': 28, 'field': 'SSP_WIDE', 'dateObs': '2015-03-29', 'pointing': 1183, 'filter': 'HSC-Z', 'taiObs': '2015-03-29', 'expTime': 200.0} dataId={'visit': 34338, 'ccd': 75, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0} dataId={'visit': 34362, 'ccd': 38, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0} dataId={'visit': 34382, 'ccd': 43, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0} dataId={'visit': 34384, 'ccd': 88, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0} dataId={'visit': 34402, 'ccd': 61, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0} dataId={'visit': 34422, 'ccd': 42, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0} dataId={'visit': 34424, 'ccd': 2, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0} dataId={'visit': 34448, 'ccd': 79, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0} dataId={'visit': 34450, 'ccd': 25, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0} dataId={'visit': 34478, 'ccd': 21, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0} dataId={'visit': 34482, 'ccd': 50, 'field': 'SSP_WIDE', 'dateObs': '2015-07-14', 'pointing': 1290, 'filter': 'HSC-G', 'taiObs': '2015-07-14', 'expTime': 150.0} dataId={'visit': 36140, 'ccd': 30, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0} dataId={'visit': 36170, 'ccd': 18, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0} dataId={'visit': 36182, 'ccd': 80, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0} dataId={'visit': 36212, 'ccd': 34, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0} dataId={'visit': 36234, 'ccd': 82, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0} dataId={'visit': 36240, 'ccd': 0, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0} dataId={'visit': 36258, 'ccd': 73, 'field': 'SSP_WIDE', 'dateObs': '2015-07-21', 'pointing': 1297, 'filter': 'HSC-I', 'taiObs': '2015-07-21', 'expTime': 200.0} dataId={'visit': 34644, 'ccd': 72, 'field': 'SSP_WIDE', 'dateObs': '2015-07-15', 'pointing': 1291, 'filter': 'HSC-R', 'taiObs': '2015-07-15', 'expTime': 150.0} dataId={'visit': 34690, 'ccd': 39, 'field': 'SSP_WIDE', 'dateObs': '2015-07-15', 'pointing': 1291, 'filter': 'HSC-R', 'taiObs': '2015-07-15', 'expTime': 150.0} dataId={'visit': 34942, 'ccd': 44, 'field': 'SSP_WIDE', 'dateObs': '2015-07-16', 'pointing': 1292, 'filter': 'HSC-Y', 'taiObs': '2015-07-16', 'expTime': 200.0} dataId={'visit': 36762, 'ccd': 18, 'field': 'SSP_WIDE', 'dateObs': '2015-07-23', 'pointing': 1299, 'filter': 'HSC-Y', 'taiObs': '2015-07-23', 'expTime': 200.0} dataId={'visit': 36774, 'ccd': 24, 'field': 'SSP_WIDE', 'dateObs': '2015-07-23', 'pointing': 1299, 'filter': 'HSC-Y', 'taiObs': '2015-07-23', 'expTime': 200.0} dataId={'visit': 36792, 'ccd': 34, 'field': 'SSP_WIDE', 'dateObs': '2015-07-23', 'pointing': 1299, 'filter': 'HSC-Y', 'taiObs': '2015-07-23', 'expTime': 200.0} dataId={'visit': 36808, 'ccd': 26, 'field': 'SSP_WIDE', 'dateObs': '2015-07-23', 'pointing': 1299, 'filter': 'HSC-Y', 'taiObs': '2015-07-23', 'expTime': 200.0} dataId={'visit': 36828, 'ccd': 16, 'field': 'SSP_WIDE', 'dateObs': '2015-07-23', 'pointing': 1299, 'filter': 'HSC-Y', 'taiObs': '2015-07-23', 'expTime': 200.0} dataId={'visit': 36828, 'ccd': 73, 'field': 'SSP_WIDE', 'dateObs': '2015-07-23', 'pointing': 1299, 'filter': 'HSC-Y', 'taiObs': '2015-07-23', 'expTime': 200.0} dataId={'visit': 36428, 'ccd': 61, 'field': 'SSP_WIDE', 'dateObs': '2015-07-22', 'pointing': 1298, 'filter': 'HSC-Z', 'taiObs': '2015-07-22', 'expTime': 200.0} dataId={'visit': 36432, 'ccd': 23, 'field': 'SSP_WIDE', 'dateObs': '2015-07-22', 'pointing': 1298, 'filter': 'HSC-Z', 'taiObs': '2015-07-22', 'expTime': 200.0} dataId={'visit': 36492, 'ccd': 50, 'field': 'SSP_WIDE', 'dateObs': '2015-07-22', 'pointing': 1298, 'filter': 'HSC-Z', 'taiObs': '2015-07-22', 'expTime': 200.0}
            Hide
            swinbank John Swinbank added a comment -

            It seems to me that this probably blocks release v17. Chris Morrison [X], Eric Bellm, Jim Bosch, Yusra AlSayyad — opinions?

            Show
            swinbank John Swinbank added a comment - It seems to me that this probably blocks release v17. Chris Morrison [X] , Eric Bellm , Jim Bosch , Yusra AlSayyad — opinions?
            swinbank John Swinbank made changes -
            Link This issue blocks DM-16397 [ DM-16397 ]
            Hide
            jbosch Jim Bosch added a comment -

            Agree that this is a release blocker.  I'd be open to reverting to OptimisticB for just HSC to avoid impacting the v17 timeline, though a real fix would of course be even better.

            Show
            jbosch Jim Bosch added a comment - Agree that this is a release blocker.  I'd be open to reverting to OptimisticB for just HSC to avoid impacting the v17 timeline, though a real fix would of course be even better.
            swinbank John Swinbank made changes -
            Assignee Chris Morrison [ cmorrison ]
            swinbank John Swinbank made changes -
            Component/s Science Pipelines [ 10706 ]
            swinbank John Swinbank made changes -
            Epic Link DM-16722 [ 235355 ]
            swinbank John Swinbank made changes -
            Sprint AP S19-3 [ 831 ]
            Story Points 4
            Team Alert Production [ 10300 ]
            cmorrison Chris Morrison [X] (Inactive) made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            Hey, I believe I've locked in on the problem after rerunning the 60 failed ccds.

            The pessimistic matcher computes an automated match tolerance. Basically this automated tolerance gets set very low in some unlucky cases and the tolerance is allowed to soften enough to get a match. By my estimates it would never be allowed larger than 1 arcsec. I also found a bug in this tolerance that let's it be set to very small, unphysical values.

            I'll fix the bug and see if I can't tweak things slightly to get all 60 to succeed.

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - Hey, I believe I've locked in on the problem after rerunning the 60 failed ccds. The pessimistic matcher computes an automated match tolerance. Basically this automated tolerance gets set very low in some unlucky cases and the tolerance is allowed to soften enough to get a match. By my estimates it would never be allowed larger than 1 arcsec. I also found a bug in this tolerance that let's it be set to very small, unphysical values. I'll fix the bug and see if I can't tweak things slightly to get all 60 to succeed.
            swinbank John Swinbank made changes -
            Link This issue relates to DM-17918 [ DM-17918 ]
            Hide
            cmorrison Chris Morrison [X] (Inactive) added a comment -

            Okay, so tweak I made to the distance tolerance softening iterations and was able to get all 60 ccds to succeed. As I said previously, the problem was that the match distances was never softened with enough iterations to produce a match given the distortions in HSC. I've run the same settings through validate_drp and everything checks out as fine. I'll clean up the ticket once I head into the office. John Swinbank could you find me a reviewer in the man time please?

            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - Okay, so tweak I made to the distance tolerance softening iterations and was able to get all 60 ccds to succeed. As I said previously, the problem was that the match distances was never softened with enough iterations to produce a match given the distortions in HSC. I've run the same settings through validate_drp and everything checks out as fine. I'll clean up the ticket once I head into the office. John Swinbank could you find me a reviewer in the man time please?
            cmorrison Chris Morrison [X] (Inactive) made changes -
            Reviewers Colin Slater [ ctslater ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            ebellm Eric Bellm made changes -
            Watchers Chris Morrison, Colin Slater, Hsin-Fang Chiang, Jim Bosch, John Swinbank [ Chris Morrison, Colin Slater, Hsin-Fang Chiang, Jim Bosch, John Swinbank ] Chris Morrison, Colin Slater, Eric Bellm, Hsin-Fang Chiang, Jim Bosch, John Swinbank [ Chris Morrison, Colin Slater, Eric Bellm, Hsin-Fang Chiang, Jim Bosch, John Swinbank ]
            Show
            cmorrison Chris Morrison [X] (Inactive) added a comment - - edited Jenkins run:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29442/pipeline
            Hide
            ctslater Colin Slater added a comment -

            Fixes look good to me. The transition to multiple-pattern consensus mode above 2000 stars in the refcat seems reasonable, but it might be worth pulling that number out into a config option; that would at least give it some visibility and a doc string, etc (though it seems unlikely to need to be changed much).

            To record some offline discussions here: because the main change here is to allow more iterations of softening the match tolerances before giving up, we think this is unlikely to break any other CCDs, and the testing on validation_data_* hasn't shown any other breakage or degradation of the metrics. 

            Show
            ctslater Colin Slater added a comment - Fixes look good to me. The transition to multiple-pattern consensus mode above 2000 stars in the refcat seems reasonable, but it might be worth pulling that number out into a config option; that would at least give it some visibility and a doc string, etc (though it seems unlikely to need to be changed much). To record some offline discussions here: because the main change here is to allow more iterations of softening the match tolerances before giving up, we think this is unlikely to break any other CCDs, and the testing on validation_data_* hasn't shown any other breakage or degradation of the metrics. 
            ctslater Colin Slater made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            cmorrison Chris Morrison [X] (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            swinbank John Swinbank made changes -
            Epic Link DM-16722 [ 235355 ] DM-17995 [ 240551 ]

              People

              Assignee:
              cmorrison Chris Morrison [X] (Inactive)
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Colin Slater
              Watchers:
              Chris Morrison [X] (Inactive), Colin Slater, Eric Bellm, Hsin-Fang Chiang, Jim Bosch, John Swinbank
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.