Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34853

Make a drp_pipe/pipelines/HSC/DRP-Prod.yaml

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      https://github.com/lsst/drp_pipe/blob/main/pipelines/HSC/DRP-RC2.yaml is currently our official HSC pipeline, but its step2 subset cheats in that it is a bag of tasks that are run globally (without dataquery constraints) on our 3-tract RC2 validation runs. It contains tasks that need to be global (FGCM), tract-level (jointcal/isolatedStarAssociation) and visit-level (skyCorr, consolidateVisitSummary, finalizeCharacterization). I say that it cheats because this is only doable because it's 3-tracts.

      DP0.2 processing showed us that at 100's of sq deg, we can't submit 150-tracts all at once. The execution team's solution was to submit groups of visits and groups of tracts. As a courtesy. I propose splitting step2 for large production runs into it's visit-level and tract-level components:

      step2a: # visit-Level
            - consolidatePreSourceTable
            - consolidateVisitSummary
            - skyCorr
      step2b: # tract-Level
            - isolatedStarAssociation
            - jointcal
      step2c: # global (can be run any time after 2a)
            - fgcmBuildStarsTable
            - fgcmFitCycle
            - fgcmOutputProducts
      step2d:  # visit-Level
            - finalizeCharacterization
            - writeRecalibratedSourceTable
            - transformSourceTable
            - consolidateSourceTable
      

      this has the added benefit of communicating with NAOJ the order of imageCharacterization tasks that need to be run before coaddition.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              yusra Yusra AlSayyad
              Reporter:
              yusra Yusra AlSayyad
              Reviewers:
              Kenneth Herner
              Watchers:
              Jim Bosch, Kenneth Herner, Lee Kelvin, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.