Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32029

Add faro steps to DRP.yaml in obs_subaru

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: faro
    • Labels:

      Description

      In order to integrate `faro` metric calculation into the RC2 workflow, we need to update the DRP.yaml file in obs_subaru. This should integrate `faro` steps into the various steps that produce the required inputs for them (e.g., metrics that only require singleFrame processing can be calculated before moving on to coadds). 

        Attachments

          Issue Links

            Activity

            Hide
            jcarlin Jeffrey Carlin added a comment -

            DM-32059 moves the tasks that were causing failures in obs_subaru to separate pipelines in `faro` so they won't affect obs_subaru.

            Show
            jcarlin Jeffrey Carlin added a comment - DM-32059 moves the tasks that were causing failures in obs_subaru to separate pipelines in `faro` so they won't affect obs_subaru.
            Hide
            dtaranu Dan Taranu added a comment - - edited

            FYI, if you want to get the list of tasks from a hierarchical pipeline, you can do something like:

            pipetask build -p $FARO_DIR/pipelines/metrics_pipeline_jointcal_fgcm.yaml --show pipeline | awk '{if((length($0) > 2) && (substr($0, 3, 1) != " ")) print $0}'

             

            (The awk bit is ugly and depends on the yaml formatting so you could handle that better if needed).

            Show
            dtaranu Dan Taranu added a comment - - edited FYI, if you want to get the list of tasks from a hierarchical pipeline, you can do something like: pipetask build -p $FARO_DIR/pipelines/metrics_pipeline_jointcal_fgcm.yaml --show pipeline | awk '{if((length($0) > 2) && (substr($0, 3, 1) != " ")) print $0}'   (The awk bit is ugly and depends on the yaml formatting so you could handle that better if needed).
            Hide
            jcarlin Jeffrey Carlin added a comment -

            Sorry, Arun - I forgot to mention that there are two PRs for this ticket. The ci_hsc_gen3 PR is just to exclude faro from being run during that CI job.

            Show
            jcarlin Jeffrey Carlin added a comment - Sorry, Arun - I forgot to mention that there are two PRs for this ticket. The ci_hsc_gen3 PR is just to exclude faro from being run during that CI job.
            Hide
            kannawad Arun Kannawadi added a comment -

            Looks good. Left a comment about a minor reorganization at a later point.

            Show
            kannawad Arun Kannawadi added a comment - Looks good. Left a comment about a minor reorganization at a later point.
            Hide
            jcarlin Jeffrey Carlin added a comment -

            Did a little reorganization of the subsets. I was going to simply call "faro_step2" in "step2" of the pipeline, but Nate pointed out that subsets of subsets don't work. Nonetheless, at least now there are faro_step2, faro_step3, and faro_step5 subsets.

            Jenkins is green: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35188/pipeline

            Show
            jcarlin Jeffrey Carlin added a comment - Did a little reorganization of the subsets. I was going to simply call "faro_step2" in "step2" of the pipeline, but Nate pointed out that subsets of subsets don't work. Nonetheless, at least now there are faro_step2, faro_step3, and faro_step5 subsets. Jenkins is green:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35188/pipeline

              People

              Assignee:
              jcarlin Jeffrey Carlin
              Reporter:
              jcarlin Jeffrey Carlin
              Reviewers:
              Arun Kannawadi
              Watchers:
              Arun Kannawadi, Dan Taranu, Jeffrey Carlin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.