Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18300

Reprocess RC2 with w_2019_14

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC2 dataset, as defined in DM-11345, using Stack w_2019_14. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, skyCorrection.py, coaddDriver.py, and multiBandDriver.py. 

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-19150 Update /datasets/refcats/htm/* using meas_algorithms/bin/convert_refcat_to_nJy.py

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-19015 HSC warp making is broken with doApplyUberCal=True

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-19105 Run jointcal with HSC-RC2 and w_2019_14

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-19128 Run pipe_analysis & validate_drp with w_2019_14 HSC RC2

          • Done

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Epic Link DM-16962 [ 238433 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Link DM-16962 [ 238433 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Link DM-18762 [ 251570 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-19015 [ DM-19015 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-19105 [ DM-19105 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-19128 [ DM-19128 ]
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The new BrightObjectMasks should be in place (DM-18147). Look out for "Unable to read brightObjectMask for" DataId warn messages in assembleCoadd.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The new BrightObjectMasks should be in place ( DM-18147 ). Look out for "Unable to read brightObjectMask for" DataId warn messages in assembleCoadd.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            processCcd/singleFrameDriver for all visits/ccds finished running.

            There were two non-fatal errors seen in w_2019_10 (DM-17940). The LengthError from --id visit=38950 ccd=64 is still seen (DM-15121). DM-18400 has fixed the other LengthError with Number of matches is smaller than request. There are many more warnings in the logs.

            Lots of WARN of Found version 0 reference catalog with old style units in schema. appear in the log, expected by the implementation of RFC-575 as the reference catalogs on /datasets/refcats/ have not been updated.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - processCcd/singleFrameDriver for all visits/ccds finished running. There were two non-fatal errors seen in w_2019_10 ( DM-17940 ). The LengthError from --id visit=38950 ccd=64 is still seen ( DM-15121 ). DM-18400 has fixed the other LengthError with Number of matches is smaller than request . There are many more warnings in the logs. Lots of WARN of Found version 0 reference catalog with old style units in schema. appear in the log, expected by the implementation of RFC-575 as the reference catalogs on /datasets/refcats/ have not been updated.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Attempting to use meas_mosaic results to make coadd products (the HSC default) failed, due to a bug likely in meas_mosaic or afw from DM-10156; the bug is reported at DM-19015.

            Meanwhile, continue the coaddDriver runs by using the jointcal outputs instead. The output butler repo will be /datasets/hsc/repo/rerun/RC/w_2019_14/DM-18300-jointcal/ (different from the usual path pattern!)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Attempting to use meas_mosaic results to make coadd products (the HSC default) failed, due to a bug likely in meas_mosaic or afw from DM-10156 ; the bug is reported at DM-19015 . Meanwhile, continue the coaddDriver runs by using the jointcal outputs instead. The output butler repo will be /datasets/hsc/repo/rerun/RC/w_2019_14/ DM-18300 -jointcal/ (different from the usual path pattern!)
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue blocks DM-19150 [ DM-19150 ]
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Had to break down the patches to finish multiband run for the COSMOS tract.

            coadd/multiband: the number of data products and empty patches are the same as w_2018_15/DM-14123 (this comment)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Had to break down the patches to finish multiband run for the COSMOS tract. coadd/multiband: the number of data products and empty patches are the same as w_2018_15/ DM-14123 ( this comment )
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2019_14/DM-18300-jointcal/logs/

            The coadd and multiband products are based on the jointcal outputs (not meas_mosaic).

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2019_14/ DM-18300 -jointcal/logs/ The coadd and multiband products are based on the jointcal outputs (not meas_mosaic).
            hchiang2 Hsin-Fang Chiang made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.