Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35269

Fix new-line parsing issue in DRP pipelines

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: drp_pipe
    • Labels:
      None

      Description

      The recently merged DM-34853 ticket added production DRP piplines for a number of cameras, splitting step2 for large production runs into it's visit-level and tract-level components.

      It appears that new line parsing in the description block requires a double new line in YAML in order to appear as a single new line in the terminal when using pipetask build.

      For example, in the file $DRP_PIPE_DIR/ingredients/DECam/DRP.yaml, Step 2a looks like this in YAML:

        step2a:
          subset:
            - consolidatePreSourceTable
            - consolidateVisitSummary
          description: >
            Visit-level tasks
            Allowed data query constraints: visit
       
            Tasks aggregate all detectors for a given visit.
      

      but gets parsed like this when running pipetask build -p $DRP_PIPE_DIR/ingredients/DECam/DRP.yaml --show pipeline:

        step2a:
          subset:
          - consolidatePreSourceTable
          - consolidateVisitSummary
          description: |
            Visit-level tasks Allowed data query constraints: visit
            Tasks aggregate all detectors for a given visit.
      

      This ticket resolves this issue, either by adding new lines to each relevant YAML file from DM-34853, or switching from 'description: >' to 'description: |'.

        Attachments

          Issue Links

            Activity

            Hide
            lskelvin Lee Kelvin added a comment - - edited

            I made the decision to change all instances of 'description: >' to 'description: |' on this ticket. Not only does this preserve new lines as typed by the original pipeline author, but it also ensures that description text blocks keep to within an 80-character line width limit. This helps improve readability within a terminal, maintaining the layout intended when the description was initially written (WYSIWYG), and ensuring that mid-word line breaks do not occur.

            For the above example in the description, DECam Step 2a now looks like this on the command line:

              step2a:
                subset:
                - consolidatePreSourceTable
                - consolidateVisitSummary
                description: |
                  Visit-level tasks
                  Allowed data query constraints: visit
             
                  Tasks aggregate all detectors for a given visit.
            

            Pull request for this ticket. As this is a cosmetic change and doesn't touch any code, I don't believe a Jenkins run is necessary. However, I am happy to kick one off if recommended.

            Show
            lskelvin Lee Kelvin added a comment - - edited I made the decision to change all instances of ' description: > ' to ' description: | ' on this ticket. Not only does this preserve new lines as typed by the original pipeline author, but it also ensures that description text blocks keep to within an 80-character line width limit. This helps improve readability within a terminal, maintaining the layout intended when the description was initially written (WYSIWYG), and ensuring that mid-word line breaks do not occur. For the above example in the description, DECam Step 2a now looks like this on the command line: step2a: subset: - consolidatePreSourceTable - consolidateVisitSummary description: | Visit-level tasks Allowed data query constraints: visit   Tasks aggregate all detectors for a given visit. Pull request for this ticket . As this is a cosmetic change and doesn't touch any code, I don't believe a Jenkins run is necessary. However, I am happy to kick one off if recommended.
            Hide
            lskelvin Lee Kelvin added a comment -

            Thanks Yusra! Ticket merged and branch deleted.

            Show
            lskelvin Lee Kelvin added a comment - Thanks Yusra! Ticket merged and branch deleted.

              People

              Assignee:
              lskelvin Lee Kelvin
              Reporter:
              lskelvin Lee Kelvin
              Reviewers:
              Yusra AlSayyad
              Watchers:
              Lee Kelvin, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.