Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11189

Reprocess RC with w_2017_36

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC dataset using w_2017_36 following the same steps as in DM-10129/DM-11020

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11800 Update meas_mosaic to new transform-based API for afw::cameraGeom

          • Done
          is triggering

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11927 Set maxOffsetPix consistently

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11861 SqlRegistry raising exception due to missing super.__del__

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11916 Warnings "Extent2I object has no attribute getWidth" when running meas_mosaic, jointcal

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-11917 Error in coaddDriver following compression changes

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11356 Confusion of refObjLoader.pixelMargin and matcher.maxOffsetPix

          • Done
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            meas_mosaic fails to build with w_2017_36: this will likely be solved by DM-11800

            Show
            hchiang2 Hsin-Fang Chiang added a comment - meas_mosaic fails to build with w_2017_36 : this will likely be solved by DM-11800
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            I started running singleFrameDriver and makeSkyMap; the rest is blocked by DM-11800.

            There are many error messages like this: 

            Exception AttributeError: "'super' object has no attribute '__del__'" in <bound method SqliteRegistry.__del__ of <lsst.daf.persistence.registries.SqliteRegistry object at 0x7f52e7b16390>> ignored

            This has been reported in DM-11861

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I started running singleFrameDriver and makeSkyMap ; the rest is blocked by DM-11800 . There are many error messages like this: Exception AttributeError: "'super' object has no attribute '__del__'" in <bound method SqliteRegistry.__del__ of <lsst.daf.persistence.registries.SqliteRegistry object at 0x7f52e7b16390 >> ignored This has been reported in DM-11861
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            The DM-11861 warnings are non-fatal. They happen during garbage collection so don't affect process execution.

            Show
            tjenness Tim Jenness added a comment - The DM-11861 warnings are non-fatal. They happen during garbage collection so don't affect process execution.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            SingleFrameDriver/prcessCcd failed in 11 CCDs at WIDE, and 37 CCDs at COSMOS. Their data IDs are:

            --id visit=278 ccd=95 --id visit=280 ccd=22 --id visit=280 ccd=69 --id visit=284 ccd=61 --id visit=24304 ccd=70 --id visit=24308 ccd=29 --id visit=28374 ccd=101 --id visit=28376 ccd=69 --id visit=28380 ccd=0 --id visit=28382 ccd=101 --id visit=28386 ccd=103 --id visit=28392 ccd=101 --id visit=28392 ccd=102 --id visit=28394 ccd=93 --id visit=28396 ccd=102 --id visit=28398 ccd=95 --id visit=28398 ccd=101 --id visit=28400 ccd=5 --id visit=28400 ccd=10 --id visit=28400 ccd=15 --id visit=28400 ccd=23 --id visit=28400 ccd=26 --id visit=28400 ccd=40 --id visit=28400 ccd=53 --id visit=28400 ccd=55 --id visit=28400 ccd=61 --id visit=28400 ccd=68 --id visit=28400 ccd=77 --id visit=28400 ccd=84 --id visit=28400 ccd=89 --id visit=28400 ccd=93 --id visit=28400 ccd=94 --id visit=28400 ccd=95 --id visit=28400 ccd=99 --id visit=28400 ccd=100 --id visit=28400 ccd=101 --id visit=28400 ccd=102 --id visit=9868 ccd=76 --id visit=9868 ccd=77 --id visit=11582 ccd=76 --id visit=7344 ccd=67 --id visit=19468 ccd=69 --id visit=6478 ccd=99 --id visit=6528 ccd=24 --id visit=6528 ccd=67 --id visit=9736 ccd=67 --id visit=17738 ccd=69 --id visit=17750 ccd=58

            Out of the 48 singleFrameDriver/prcessCcd fatal failures:

            • 31 failed with "Unable to match sources"
            • 2 failed with "No sources remaining in match list after magnitude limit cuts"
            • 4 failed with "No objects passed our cuts for consideration as psf stars"
            • 11 failed with "PSF star selector found [123] candidates"

            They are different from the results of w_2017_34, with which there were only 46 failures in total.

            These 2 failed in w_2017_34 but passed in w_2017_36:

             --id visit=1206 ccd=77
            		 
             --id visit=29324 ccd=99

            These 4 failed in w_2017_36 but passed in w_2017_34:

             --id visit=24304 ccd=70
            		 
             --id visit=28374 ccd=101
            		 
             --id visit=28386 ccd=103
            		 
             --id visit=9868 ccd=77

            Show
            hchiang2 Hsin-Fang Chiang added a comment - SingleFrameDriver/prcessCcd failed in 11 CCDs at WIDE, and 37 CCDs at COSMOS. Their data IDs are: --id visit= 278 ccd= 95 --id visit= 280 ccd= 22 --id visit= 280 ccd= 69 --id visit= 284 ccd= 61 --id visit= 24304 ccd= 70 --id visit= 24308 ccd= 29 --id visit= 28374 ccd= 101 --id visit= 28376 ccd= 69 --id visit= 28380 ccd= 0 --id visit= 28382 ccd= 101 --id visit= 28386 ccd= 103 --id visit= 28392 ccd= 101 --id visit= 28392 ccd= 102 --id visit= 28394 ccd= 93 --id visit= 28396 ccd= 102 --id visit= 28398 ccd= 95 --id visit= 28398 ccd= 101 --id visit= 28400 ccd= 5 --id visit= 28400 ccd= 10 --id visit= 28400 ccd= 15 --id visit= 28400 ccd= 23 --id visit= 28400 ccd= 26 --id visit= 28400 ccd= 40 --id visit= 28400 ccd= 53 --id visit= 28400 ccd= 55 --id visit= 28400 ccd= 61 --id visit= 28400 ccd= 68 --id visit= 28400 ccd= 77 --id visit= 28400 ccd= 84 --id visit= 28400 ccd= 89 --id visit= 28400 ccd= 93 --id visit= 28400 ccd= 94 --id visit= 28400 ccd= 95 --id visit= 28400 ccd= 99 --id visit= 28400 ccd= 100 --id visit= 28400 ccd= 101 --id visit= 28400 ccd= 102 --id visit= 9868 ccd= 76 --id visit= 9868 ccd= 77 --id visit= 11582 ccd= 76 --id visit= 7344 ccd= 67 --id visit= 19468 ccd= 69 --id visit= 6478 ccd= 99 --id visit= 6528 ccd= 24 --id visit= 6528 ccd= 67 --id visit= 9736 ccd= 67 --id visit= 17738 ccd= 69 --id visit= 17750 ccd= 58 Out of the 48 singleFrameDriver/prcessCcd fatal failures: 31 failed with "Unable to match sources" 2 failed with "No sources remaining in match list after magnitude limit cuts" 4 failed with "No objects passed our cuts for consideration as psf stars" 11 failed with "PSF star selector found [123] candidates" They are different from the results of w_2017_34 , with which there were only 46 failures in total. These 2 failed in w_2017_34 but passed in w_2017_36: --id visit=1206 ccd=77 --id visit=29324 ccd=99 These 4 failed in w_2017_36 but passed in w_2017_34: --id visit=24304 ccd=70 --id visit=28374 ccd=101 --id visit=28386 ccd=103 --id visit=9868 ccd=77
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            A log with the 48 processCcd failures is attached: retryFailedSfm_w36.log

            Show
            hchiang2 Hsin-Fang Chiang added a comment - A log with the 48 processCcd failures is attached: retryFailedSfm_w36.log
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The ProcessCcd outcome changes were related to DM-11356. DM-11927 has fixed it and improved the configs to make matchers consistent. Expect to see fewer failures in the next weekly.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The ProcessCcd outcome changes were related to DM-11356 . DM-11927 has fixed it and improved the configs to make matchers consistent. Expect to see fewer failures in the next weekly.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment - - edited
            mosaic

            The version of meas_mosaic is after DM-11916 merged.

            No longer see warnings like afw.table WARN: Persisted match record with ID 110531509187609976 not found in catalog 1 in the logs, thanks to the fix of DM-11356.

            coadd/multiband

            coaddDriver was run with the ticket branch of DM-11917

            The numbers of products and the empty patches are the same as w_2017_28 (DM-11184 , this comment). No fatal errors were seen in the logs; but there were lots of warnings.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - - edited mosaic The version of meas_mosaic is after DM-11916 merged. No longer see warnings like afw.table WARN: Persisted match record with ID 110531509187609976 not found in catalog 1 in the logs , thanks to the fix of DM-11356 . coadd/multiband coaddDriver was run with the ticket branch of DM-11917 The numbers of products and the empty patches are the same as w_2017_28 ( DM-11184 , this comment ). No fatal errors were seen in the logs; but there were lots of warnings.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Logs have been copied to /datasets/hsc/repo/rerun/RC/w_2017_36/DM-11189/logs
            The output repo is at /datasets/hsc/repo/rerun/RC/w_2017_36/DM-11189/

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Logs have been copied to /datasets/hsc/repo/rerun/RC/w_2017_36/ DM-11189 /logs The output repo is at /datasets/hsc/repo/rerun/RC/w_2017_36/ DM-11189 /

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.