Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11796

Run ctrl_pool/pipe_driver tasks with HSC RC data with a Python 3 stack

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Run the HSC RC reprocessing using a Python 3 stack.

        Attachments

          Activity

          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Started this last week, using the Python 3 stack version w_2017_33 installed by Mikolaj Kowalik.

          I've run singleFrameDriver with the HSC RC data. There was one more failure seen in the Python 3 stack compared with the Python 2 stack of the same version; this has been reported in DM-11779.

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Started this last week, using the Python 3 stack version w_2017_33 installed by Mikolaj Kowalik . I've run singleFrameDriver with the HSC RC data. There was one more failure seen in the Python 3 stack compared with the Python 2 stack of the same version; this has been reported in DM-11779 .
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Different from the usual biweekly RC reprocessing, this does not include running MosaicTask, which is not yet Python 3 compatible and not a ctrl_pool task.

          Following singleFrameDriver, I've run coaddDriver with -c makeCoaddTempExp.doApplyUberCal=False assembleCoadd.doApplyUberCal=False and then multiBandDriver.

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Different from the usual biweekly RC reprocessing, this does not include running MosaicTask , which is not yet Python 3 compatible and not a ctrl_pool task. Following singleFrameDriver , I've run coaddDriver with -c makeCoaddTempExp.doApplyUberCal=False assembleCoadd.doApplyUberCal=False and then multiBandDriver .
          Hide
          tjenness Tim Jenness added a comment -

          meas_mosaic might work now. I pushed some fixes last week. It has not been tested on python3 though so YMMV.

          Show
          tjenness Tim Jenness added a comment - meas_mosaic might work now. I pushed some fixes last week. It has not been tested on python3 though so YMMV.
          Hide
          lauren Lauren MacArthur added a comment -

          meas_mosaic still needs DM-11800 to get merged...

          Show
          lauren Lauren MacArthur added a comment - meas_mosaic still needs DM-11800 to get merged...
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          Thanks for the info.

          For testing DESDM we have decided not to include meas_mosaic in the current cycle. We are aware it deviates from the usual biweekly RC runs, but (1) meas_mosaic will soon be replaced (2) science results are not the main focus in this cycle. The original purpose of this ticket is to generate an output repo using ctrl_pool that can be compared with the outputs using DESDM (that's why it's in DM-11840 but not in DM-11839)

          The output repo of this Python 3 run is linked at
          /project/hsc_rc/w_2017_33/DM-11796/py3/
          A same run using the Python 2 Stack is linked at
          /project/hsc_rc/w_2017_33/DM-11796/py2/

          Show
          hchiang2 Hsin-Fang Chiang added a comment - Thanks for the info. For testing DESDM we have decided not to include meas_mosaic in the current cycle. We are aware it deviates from the usual biweekly RC runs, but (1) meas_mosaic will soon be replaced (2) science results are not the main focus in this cycle. The original purpose of this ticket is to generate an output repo using ctrl_pool that can be compared with the outputs using DESDM (that's why it's in DM-11840 but not in DM-11839 ) The output repo of this Python 3 run is linked at /project/hsc_rc/w_2017_33/ DM-11796 /py3/ A same run using the Python 2 Stack is linked at /project/hsc_rc/w_2017_33/ DM-11796 /py2/

            People

            • Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, Lauren MacArthur, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel