Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34853

Make a drp_pipe/pipelines/HSC/DRP-Prod.yaml

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      https://github.com/lsst/drp_pipe/blob/main/pipelines/HSC/DRP-RC2.yaml is currently our official HSC pipeline, but its step2 subset cheats in that it is a bag of tasks that are run globally (without dataquery constraints) on our 3-tract RC2 validation runs. It contains tasks that need to be global (FGCM), tract-level (jointcal/isolatedStarAssociation) and visit-level (skyCorr, consolidateVisitSummary, finalizeCharacterization). I say that it cheats because this is only doable because it's 3-tracts.

      DP0.2 processing showed us that at 100's of sq deg, we can't submit 150-tracts all at once. The execution team's solution was to submit groups of visits and groups of tracts. As a courtesy. I propose splitting step2 for large production runs into it's visit-level and tract-level components:

      step2a: # visit-Level
            - consolidatePreSourceTable
            - consolidateVisitSummary
            - skyCorr
      step2b: # tract-Level
            - isolatedStarAssociation
            - jointcal
      step2c: # global (can be run any time after 2a)
            - fgcmBuildStarsTable
            - fgcmFitCycle
            - fgcmOutputProducts
      step2d:  # visit-Level
            - finalizeCharacterization
            - writeRecalibratedSourceTable
            - transformSourceTable
            - consolidateSourceTable
      

      this has the added benefit of communicating with NAOJ the order of imageCharacterization tasks that need to be run before coaddition.

        Attachments

          Issue Links

            Activity

            yusra Yusra AlSayyad created issue -
            yusra Yusra AlSayyad made changes -
            Field Original Value New Value
            Labels hsc-pdr4
            yusra Yusra AlSayyad made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            yusra Yusra AlSayyad made changes -
            Reviewers Kenneth Herner [ kherner ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            kherner Kenneth Herner made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            yusra Yusra AlSayyad made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-30551 [ 511217 ]
            Story Points 1 3
            lskelvin Lee Kelvin made changes -
            Link This issue relates to DM-35269 [ DM-35269 ]
            lskelvin Lee Kelvin made changes -
            Link This issue relates to DM-35426 [ DM-35426 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-30551 [ 511217 ] DM-32153 [ 779823 ]

              People

              Assignee:
              yusra Yusra AlSayyad
              Reporter:
              yusra Yusra AlSayyad
              Reviewers:
              Kenneth Herner
              Watchers:
              Jim Bosch, Kenneth Herner, Lee Kelvin, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.