Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32029

Add faro steps to DRP.yaml in obs_subaru

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • faro

    Description

      In order to integrate `faro` metric calculation into the RC2 workflow, we need to update the DRP.yaml file in obs_subaru. This should integrate `faro` steps into the various steps that produce the required inputs for them (e.g., metrics that only require singleFrame processing can be calculated before moving on to coadds). 

      Attachments

        Issue Links

          Activity

            DM-32059 moves the tasks that were causing failures in obs_subaru to separate pipelines in `faro` so they won't affect obs_subaru.

            jcarlin Jeffrey Carlin added a comment - DM-32059 moves the tasks that were causing failures in obs_subaru to separate pipelines in `faro` so they won't affect obs_subaru.
            dtaranu Dan Taranu added a comment - - edited

            FYI, if you want to get the list of tasks from a hierarchical pipeline, you can do something like:

            pipetask build -p $FARO_DIR/pipelines/metrics_pipeline_jointcal_fgcm.yaml --show pipeline | awk '{if((length($0) > 2) && (substr($0, 3, 1) != " ")) print $0}'

             

            (The awk bit is ugly and depends on the yaml formatting so you could handle that better if needed).

            dtaranu Dan Taranu added a comment - - edited FYI, if you want to get the list of tasks from a hierarchical pipeline, you can do something like: pipetask build -p $FARO_DIR/pipelines/metrics_pipeline_jointcal_fgcm.yaml --show pipeline | awk '{if((length($0) > 2) && (substr($0, 3, 1) != " ")) print $0}'   (The awk bit is ugly and depends on the yaml formatting so you could handle that better if needed).

            Sorry, Arun - I forgot to mention that there are two PRs for this ticket. The ci_hsc_gen3 PR is just to exclude faro from being run during that CI job.

            jcarlin Jeffrey Carlin added a comment - Sorry, Arun - I forgot to mention that there are two PRs for this ticket. The ci_hsc_gen3 PR is just to exclude faro from being run during that CI job.

            Looks good. Left a comment about a minor reorganization at a later point.

            kannawad Arun Kannawadi added a comment - Looks good. Left a comment about a minor reorganization at a later point.

            Did a little reorganization of the subsets. I was going to simply call "faro_step2" in "step2" of the pipeline, but Nate pointed out that subsets of subsets don't work. Nonetheless, at least now there are faro_step2, faro_step3, and faro_step5 subsets.

            Jenkins is green: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35188/pipeline

            jcarlin Jeffrey Carlin added a comment - Did a little reorganization of the subsets. I was going to simply call "faro_step2" in "step2" of the pipeline, but Nate pointed out that subsets of subsets don't work. Nonetheless, at least now there are faro_step2, faro_step3, and faro_step5 subsets. Jenkins is green:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35188/pipeline

            People

              jcarlin Jeffrey Carlin
              jcarlin Jeffrey Carlin
              Arun Kannawadi
              Arun Kannawadi, Dan Taranu, Jeffrey Carlin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.