Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34623 AP Performance sprint
  3. DM-34828

Process large-scale processing run and persist timing metrics

    XMLWordPrintable

    Details

    • Type: Technical task
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Sprint:
      AP S22-6 (May)
    • Team:
      Alert Production

      Description

      Perform a large-scale processing run of the dataset identified in DM-34827. Use the current weekly pipeline.

      Import appropriate ap_verify timing tasks into the pipeline run so that we can persist timing metrics to the Butler for later aggregation.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34896 Collate timing metrics from run on 4-patch >200 visit DC2 dataset

          • Done
          is triggering

          Technical task - Created by JIRA Agile - do not edit or delete. Issue type for a technical task. DM-34947 Investigate imageDifference failures

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-34924 Execution butler can't handle new StorageClass conversion

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            mrawls Meredith Rawls added a comment -

            My pipeline is /project/mrawls/ap_profile/ApPipe_DM-34828.yaml and my bps submit yaml is bps_ApPipe_DM-34828.yaml. I already ran make_apdb on my usual postgres APDB schema.

            Unfortunately, my first attempt to kick off a run failed to create the execution butler due to a problem with the task metadata storage class. The problem was previously addressed on DM-33220, but only for "regular" butlers, not the BPS-specific execution butler. I created DM-34924 in the hopes of a fix.

            Show
            mrawls Meredith Rawls added a comment - My pipeline is /project/mrawls/ap_profile/ApPipe_ DM-34828 .yaml and my bps submit yaml is bps_ApPipe_ DM-34828 .yaml . I already ran make_apdb on my usual postgres APDB schema. Unfortunately, my first attempt to kick off a run failed to create the execution butler due to a problem with the task metadata storage class. The problem was previously addressed on DM-33220 , but only for "regular" butlers, not the BPS-specific execution butler. I created DM-34924 in the hopes of a fix.
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            I have pushed a revert of DM-33220 to `u/parejkoj/DM-34828-nomerge` of `verify`, and have a clone of it ready to test at `lsst-devl:/project/parejkoj/ap_profile/verify`. Please give it a try.

            Show
            Parejkoj John Parejko added a comment - I have pushed a revert of DM-33220 to `u/parejkoj/ DM-34828 -nomerge` of `verify`, and have a clone of it ready to test at `lsst-devl:/project/parejkoj/ap_profile/verify`. Please give it a try.
            Hide
            Permalink
            mrawls Meredith Rawls added a comment - - edited

            John's verify package hack did the trick, and I managed to process the dataset today. However, I did it without the ap_verify hack that was intended to force diaPipe to process visits one at a time in sequential order, because after trying some things with Nate's help, I don't think it is really compatible with bps.

            RUN output collection in /repo/dc2: u/mrawls/DM-34828/20220524T214644Z

            APDB schema in postgres: mrawls_DM34828

            Submit directory for interrogating logs etc.: /project/mrawls/ap_profile/submit/u/mrawls/DM-34828/20220524T214644Z

            13 datasets failed imageDifference due to not being able to find a template, which was surprising. The exact error is, e.g., InvalidParameterError: Cannot compute CoaddPsf at point (x, y); no input images at that point.

             

            Show
            mrawls Meredith Rawls added a comment - - edited John's verify package hack did the trick, and I managed to process the dataset today. However, I did it without the ap_verify hack that was intended to force diaPipe to process visits one at a time in sequential order, because after trying some things with Nate's help, I don't think it is really compatible with bps. RUN output collection in /repo/dc2: u/mrawls/ DM-34828 /20220524T214644Z APDB schema in postgres: mrawls_DM34828 Submit directory for interrogating logs etc.: /project/mrawls/ap_profile/submit/u/mrawls/ DM-34828 /20220524T214644Z 13 datasets failed imageDifference due to not being able to find a template, which was surprising. The exact error is, e.g., InvalidParameterError: Cannot compute CoaddPsf at point (x, y); no input images at that point.  
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Thanks for the notes on what was run and what failed.

            Show
            Parejkoj John Parejko added a comment - Thanks for the notes on what was run and what failed.
            Hide
            Permalink
            mrawls Meredith Rawls added a comment - - edited

            Important update -

            Redid this now with all the proper template patches existing  - see DM-34947

            Pipeline: /project/mrawls/ap_profile/ApPipe_DM-34828.yaml
            bps submit yaml: /project/mrawls/ap_profile/bps_ApPipe_DM-34828.yaml

            "Before" collection (w2022.21): u/mrawls/DM-34828-w_2022_21
            Postgres schema: mrawls_DM34828_w202221
            Submit directory: /project/mrawls/ap_profile/submit/u/mrawls/DM-34828-w_2022_21/20220526T191849Z/

            "After" collection (w2022.22): u/mrawls/DM-34828-w_2022_22
            Postgres schema: mrawls_DM34828_w202222
            Submit directory: /project/mrawls/ap_profile/submit/u/mrawls/DM-34828-w_2022_22/20220526T202214Z/ 

            Show
            mrawls Meredith Rawls added a comment - - edited Important update - Redid this now with all the proper template patches existing  - see DM-34947 Pipeline: /project/mrawls/ap_profile/ApPipe_ DM-34828 .yaml bps submit yaml: /project/mrawls/ap_profile/bps_ApPipe_ DM-34828 .yaml "Before" collection (w2022.21): u/mrawls/ DM-34828 -w_2022_21 Postgres schema: mrawls_DM34828_w202221 Submit directory: /project/mrawls/ap_profile/submit/u/mrawls/ DM-34828 -w_2022_21/20220526T191849Z/ "After" collection (w2022.22): u/mrawls/ DM-34828 -w_2022_22 Postgres schema: mrawls_DM34828_w202222 Submit directory: /project/mrawls/ap_profile/submit/u/mrawls/ DM-34828 -w_2022_22/20220526T202214Z/  

              People

              Assignee:
              mrawls Meredith Rawls
              Reporter:
              ebellm Eric Bellm
              Reviewers:
              John Parejko
              Watchers:
              Eric Bellm, John Parejko, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.