Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16234

Move qa_explorer/postprocess.py and writeObjectTable to pipe_tasks

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks, qa_explorer
    • Labels:
      None
    • Story Points:
      12
    • Epic Link:
    • Sprint:
      DRP F19-5, DRP F19-6 (Nov)
    • Team:
      Data Release Production

      Attachments

        Issue Links

          Activity

          Hide
          yusra Yusra AlSayyad added a comment - - edited

          Review time: pipe_tasks, obs_subaru, obs_base.
          I asked Tim on github to look at qa_explorer

          Proposed to do list:
          This ticket:

          • Rename all `transform` to `standardize`. Rename module `postprocess` something with `transform` or `standardize` in the name.
          • Add pyarrow as third party package

          Future tickets

          • There's a hardcoded HSC pixelscale in functors.py
          • Test an Object.yaml in obs_lsst
          • Move testPostprocess.py over pipe_tasks. Original wasn't passing and I didn't want to hold this ticket up.
          • If one accidentally includes a filter in the --id for `transformObjectCatalogTask` it'll run the task #filter times. (Same results, so I won't bother for now)

          Test on 3 patches w/:
          writeObjectTable.py /datasets/hsc/repo --rerun RC/w_2019_34/DM-21091:private/<user>/sdm_output --id tract=9697 patch=3,3^3,4^3,5 filter=HSC-G^HSC-R^HSC-I^HSC-Z^HSC-Y -j 3
          transformObjectCatalog.py /datasets/hsc/repo --rerun private/<user>/sdm_output --id tract=9697 patch=3,3^3,4^3,5
          consolidateObjectTable.py /datasets/hsc/repo --rerun private/<user>/sdm_output --id tract=9697 patch=3,3^3,4^3,5

          Jenkins: won't pass until adding pyarrow

          Show
          yusra Yusra AlSayyad added a comment - - edited Review time: pipe_tasks, obs_subaru, obs_base. I asked Tim on github to look at qa_explorer Proposed to do list: This ticket: Rename all `transform` to `standardize`. Rename module `postprocess` something with `transform` or `standardize` in the name. Add pyarrow as third party package Future tickets There's a hardcoded HSC pixelscale in functors.py Test an Object.yaml in obs_lsst Move testPostprocess.py over pipe_tasks. Original wasn't passing and I didn't want to hold this ticket up. If one accidentally includes a filter in the --id for `transformObjectCatalogTask` it'll run the task #filter times. (Same results, so I won't bother for now) Test on 3 patches w/: writeObjectTable.py /datasets/hsc/repo --rerun RC/w_2019_34/ DM-21091 :private/<user>/sdm_output --id tract=9697 patch=3,3^3,4^3,5 filter=HSC-G^HSC-R^HSC-I^HSC-Z^HSC-Y -j 3 transformObjectCatalog.py /datasets/hsc/repo --rerun private/<user>/sdm_output --id tract=9697 patch=3,3^3,4^3,5 consolidateObjectTable.py /datasets/hsc/repo --rerun private/<user>/sdm_output --id tract=9697 patch=3,3^3,4^3,5 Jenkins: won't pass until adding pyarrow
          Hide
          yusra Yusra AlSayyad added a comment -

          A couple other notes, instead of a multilevelPostprocessTask this ticket proposes a {transformCatalogBaseTask}} which not multiband. transformObjectCatalogTask inherits straight from transformCatalogBaseTask and adds the multiband aspect. I didn't want to preemptively guess which would be needed by AP, but when needed we can add a transformMultilevelcatalogBaseTask in between.

          Show
          yusra Yusra AlSayyad added a comment - A couple other notes, instead of a multilevelPostprocessTask this ticket proposes a {transformCatalogBaseTask}} which not multiband. transformObjectCatalogTask inherits straight from transformCatalogBaseTask and adds the multiband aspect. I didn't want to preemptively guess which would be needed by AP, but when needed we can add a transformMultilevelcatalogBaseTask in between.
          Hide
          cmorrison Chris Morrison added a comment -

          Think I got everything. scipipe_conda_env seems to have a ticket branch but no pull request.

          Show
          cmorrison Chris Morrison added a comment - Think I got everything. scipipe_conda_env seems to have a ticket branch but no pull request.
          Hide
          yusra Yusra AlSayyad added a comment -

          I got tired of waiting for pyarrow, so I put the import pyarrow into a try block and skip the unit tests that need it unless it's been imported. Test coverage will come from the linux nodes. Jenkins running now. Well merge tomorrow morning if everything goes well.

          Show
          yusra Yusra AlSayyad added a comment - I got tired of waiting for pyarrow, so I put the import pyarrow into a try block and skip the unit tests that need it unless it's been imported. Test coverage will come from the linux nodes. Jenkins running now. Well merge tomorrow morning if everything goes well.

            People

            • Assignee:
              yusra Yusra AlSayyad
              Reporter:
              yusra Yusra AlSayyad
              Reviewers:
              Chris Morrison
              Watchers:
              Chris Morrison, Hsin-Fang Chiang, Tim Morton, Yusra AlSayyad
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel