Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34853

Make a drp_pipe/pipelines/HSC/DRP-Prod.yaml

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      https://github.com/lsst/drp_pipe/blob/main/pipelines/HSC/DRP-RC2.yaml is currently our official HSC pipeline, but its step2 subset cheats in that it is a bag of tasks that are run globally (without dataquery constraints) on our 3-tract RC2 validation runs. It contains tasks that need to be global (FGCM), tract-level (jointcal/isolatedStarAssociation) and visit-level (skyCorr, consolidateVisitSummary, finalizeCharacterization). I say that it cheats because this is only doable because it's 3-tracts.

      DP0.2 processing showed us that at 100's of sq deg, we can't submit 150-tracts all at once. The execution team's solution was to submit groups of visits and groups of tracts. As a courtesy. I propose splitting step2 for large production runs into it's visit-level and tract-level components:

      step2a: # visit-Level
      		 
            - consolidatePreSourceTable
      		 
            - consolidateVisitSummary
      		 
            - skyCorr
      		 
      step2b: # tract-Level
      		 
            - isolatedStarAssociation
      		 
            - jointcal
      		 
      step2c: # global (can be run any time after 2a)
      		 
            - fgcmBuildStarsTable
      		 
            - fgcmFitCycle
      		 
            - fgcmOutputProducts
      		 
      step2d:  # visit-Level
      		 
            - finalizeCharacterization
      		 
            - writeRecalibratedSourceTable
      		 
            - transformSourceTable
      		 
            - consolidateSourceTable

      this has the added benefit of communicating with NAOJ the order of imageCharacterization tasks that need to be run before coaddition.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35269 Fix new-line parsing issue in DRP pipelines

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35426 Fix isolated star association task connection type for DECam

          • Done

            Activity

            Ascending order - Click to sort in descending order
            4 older comments
            Hide
            Permalink
            yusra Yusra AlSayyad added a comment - - edited

            Interesting fun fact about subsets:

            DRP-RC2.yaml --> DRP-RC2+fakes.yaml --> DRP-ci_hsc+fakes.yaml
            		 
                          excl TE1, TE2, wPerp.       excl faro_all (new)

            * TE1, TE2, wPerp are members of faro_all, which made excluding faro_all from DRP-ci_hsc+fakes.yaml a no-op.  To demo: the PR that I Jenkins'd gave me: https://lsst.ncsa.illinois.edu/~yusra/download/fakes.pdf for DRP-ci_hsc+fakes.yaml  and Removing the exclusion of TE1, TE2, wPerp from DRP-RC2+fakes.yaml gives https://lsst.ncsa.illinois.edu/~yusra/download/fakes2.pdf

            In conclusion, because w22 gets tagged in 66 minutes, there isn't enough time for another ci_hsc on Jenkins, and I lose either way. Tomorrow I'll work on learning more about DRP-RC2+fakes.yaml and what's special about TE1,TE2,wPerp.

            Show
            yusra Yusra AlSayyad added a comment - - edited Interesting fun fact about subsets: DRP-RC2.yaml --> DRP-RC2+fakes.yaml --> DRP-ci_hsc+fakes.yaml excl TE1, TE2, wPerp. excl faro_all (new) * TE1, TE2, wPerp are members of faro_all, which made excluding faro_all from DRP-ci_hsc+fakes.yaml a no-op.  To demo: the PR that I Jenkins'd gave me: https://lsst.ncsa.illinois.edu/~yusra/download/fakes.pdf  for DRP-ci_hsc+fakes.yaml  and Removing the exclusion of TE1, TE2, wPerp from DRP-RC2+fakes.yaml gives https://lsst.ncsa.illinois.edu/~yusra/download/fakes2.pdf In conclusion, because w22 gets tagged in 66 minutes, there isn't enough time for another ci_hsc on Jenkins, and I lose either way. Tomorrow I'll work on learning more about DRP-RC2+fakes.yaml and what's special about TE1,TE2,wPerp.
            Hide
            Permalink
            yusra Yusra AlSayyad added a comment -

            OK Lee Kelvin and Sophie Reed gave me their blessing to skip the faro tasks in the fakes pipeline. 

            New PR on ci_hsc_gen3. Jenkins running now. Ken, once you're happy, hit the reviewed button.

            Show
            yusra Yusra AlSayyad added a comment - OK Lee Kelvin  and Sophie Reed  gave me their blessing to skip the faro tasks in the fakes pipeline.  New PR on ci_hsc_gen3. Jenkins running now. Ken, once you're happy, hit the reviewed button.
            Hide
            Permalink
            kherner Kenneth Herner added a comment -

            I'm happy once you've got a successful Jenkins run.

            Show
            kherner Kenneth Herner added a comment - I'm happy once you've got a successful Jenkins run.
            Hide
            Permalink
            kherner Kenneth Herner added a comment -

            Looks good modulo the pending Jenkins run.

            Show
            kherner Kenneth Herner added a comment - Looks good modulo the pending Jenkins run.
            Hide
            Permalink
            yusra Yusra AlSayyad added a comment -

            I'm calling https://ci.lsst.codes/job/stack-os-matrix/36761/display/redirect a success because both builds passed.

            It was something transient the stack build:  "Channel "unknown": Remote call on high_sierra-2 failed. The channel is closing down or has closed down"

            Show
            yusra Yusra AlSayyad added a comment - I'm calling https://ci.lsst.codes/job/stack-os-matrix/36761/display/redirect  a success because both builds passed. It was something transient the stack build:  "Channel "unknown": Remote call on high_sierra-2 failed. The channel is closing down or has closed down"

              People

              Assignee:
              yusra Yusra AlSayyad
              Reporter:
              yusra Yusra AlSayyad
              Reviewers:
              Kenneth Herner
              Watchers:
              Jim Bosch, Kenneth Herner, Lee Kelvin, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.